Re: [Zope-dev] Warning! TransparentFolders and ZClasses
Shane Hathaway:
Jason Spisak wrote:
Thank you for the heads up! I was going to mail you this morning anyway. What are the potential drawbacks to having all forlders be TransparentFolders?
A reasonable question, but other than the fact that every object lookup would look through every folder on the entire system, making Zope entirely useless once you have more than a few objects, there are no drawbacks at all. ;^)
Nasty pointy teeth. How is this different from having the 30,000 objects that I have in one of my instace folders? All my best, Jason Spisak CIO HireTechs.com 6151 West Century Boulevard Suite 900 Los Angeles, CA 90045 P. 310.665.3444 F. 310.665.3544 Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
On Thu, 4 May 2000, Jason Spisak wrote:
How is this different from having the 30,000 objects that I have in one of my instace folders?
How do you manage to get Zope to even respond with 30K objects in one Folder? You must be using boatloads of memory. This does bring up an interesting question though. A suggestion was made earlier that it would be nice to have a BTree based folder type to remove the limitations inherent in the current Folder data structure, allowing the efficient placement of many objects in such a BTree folder. So the question is, if you do that, what performance impact does acquisition have on such a large collection of objects in one folder? --RDM
R. David Murray:
On Thu, 4 May 2000, Jason Spisak wrote:
How is this different from having the 30,000 objects that I have in one of my instace folders?
How do you manage to get Zope to even respond with 30K objects in one Folder? You must be using boatloads of memory.
That's a yes. 1 gig of memory. Actually I don't use the management interface for these folders as you probably guessed :) Well I have a folder called Candidates which contains 15,000 candidates, and each candidate contains a resume. Isn't that 30,000 objects in the Candidates namespace?
This does bring up an interesting question though. A suggestion was made earlier that it would be nice to have a BTree based folder type to remove the limitations inherent in the current Folder data structure, allowing the efficient placement of many objects in such a BTree folder. So the question is, if you do that, what performance impact does acquisition have on such a large collection of objects in one folder?
Goo question. Michel P. and I talked about that before I brought this product online. He mentioned the BTree based folder. All my best, Jason Spisak CIO HireTechs.com 6151 West Century Boulevard Suite 900 Los Angeles, CA 90045 P. 310.665.3444 F. 310.665.3544 Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
At 06:57 PM 5/4/00 +0000, Jason Spisak wrote:
R. David Murray:
On Thu, 4 May 2000, Jason Spisak wrote:
How is this different from having the 30,000 objects that I have in one of my instace folders?
How do you manage to get Zope to even respond with 30K objects in one Folder? You must be using boatloads of memory.
That's a yes. 1 gig of memory.
Actually I don't use the management interface for these folders as you probably guessed :)
Well I have a folder called Candidates which contains 15,000 candidates, and each candidate contains a resume. Isn't that 30,000 objects in the Candidates namespace?
I would suggest you move them to a Rack/Specialist combination, which is designed for this sort of thing. You'll still have the data in the ZODB, but it won't be as attributes and you'll avoid the boatloads of memory issue. Plus, you won't have the enormous disk space hits that occur whenever you add another candidate to the folder. :) To do this with ZPatterns 0.3.0, subclass Candidate from Products.ZPatterns.Rack.RackMountable (or use the ZClass base provided). Create a Specialist called Candidates (or whatever), with a Rack to store the instances. On the Rack's Storage tab, select your candidate ZClass as the kind of object to store. You can then proceed to load up your Rack with objects, and add methods to the Specialist for any search operations you might need. The default Specialist implementation is traversable like a folder, but will search all contained racks for the specified id. Acquisition will be in effect (each candidate acquiring from the Specialist, but not the Rack) but will not include the other candidates as acquirable attributes. Less memory, less fuss, less disk space. To get an object from DTML, just say Candidates.getItem(id), and to create one, use Candidates.newItem(id), then populate its attributes. You know, I probably should stop doing all this posting today so I can finish 0.3.0 for release tomorrow. There are still a few bits of cleanup and testing not done yet. Ty's also working on the necessary updates to LoginManager for compatibility with 0.3.0.
----- Original Message ----- From: "Phillip J. Eby" <pje@telecommunity.com> To: "Jason Spisak" <444@hiretechs.com> Cc: <zope-dev@zope.org> Sent: Thursday, May 04, 2000 3:12 PM Subject: Re: [Zope-dev] Warning! TransparentFolders and ZClasses
At 06:57 PM 5/4/00 +0000, Jason Spisak wrote:
Well I have a folder called Candidates which contains 15,000 candidates, and each candidate contains a resume. Isn't that 30,000 objects in the Candidates namespace?
I would suggest you move them to a Rack/Specialist combination, which is designed for this sort of thing. You'll still have the data in the ZODB, but it won't be as attributes and you'll avoid the boatloads of memory issue. Plus, you won't have the enormous disk space hits that occur whenever you add another candidate to the folder. :)
How does the Rack alleviate this problem? As I understand it, an ObjectManager has its contents as attributes... so when the object is activated, you need to bring in the whole object. I'm just wondering how the Rack works around this (and if there are still upper bounds to be considering...) Thanks, Kevin
Phillip J. Eby:
At 06:57 PM 5/4/00 +0000, Jason Spisak wrote:
R. David Murray:
On Thu, 4 May 2000, Jason Spisak wrote:
How is this different from having the 30,000 objects that I have in one of my instace folders?
How do you manage to get Zope to even respond with 30K objects in one Folder? You must be using boatloads of memory.
That's a yes. 1 gig of memory.
Actually I don't use the management interface for these folders as you probably guessed :)
Well I have a folder called Candidates which contains 15,000 candidates, and each candidate contains a resume. Isn't that 30,000 objects in the Candidates namespace?
I would suggest you move them to a Rack/Specialist combination, which is designed for this sort of thing. You'll still have the data in the ZODB, but it won't be as attributes and you'll avoid the boatloads of memory issue. Plus, you won't have the enormous disk space hits that occur whenever you add another candidate to the folder. :)
Which I am begining to notice.
To do this with ZPatterns 0.3.0, subclass Candidate from Products.ZPatterns.Rack.RackMountable (or use the ZClass base provided). Create a Specialist called Candidates (or whatever), with a Rack to store the instances. On the Rack's Storage tab, select your candidate ZClass as the kind of object to store. You can then proceed to load up your Rack with objects, and add methods to the Specialist for any search operations you might need. The default Specialist implementation is traversable like a folder, but will search all contained racks for the specified id. Acquisition will be in effect (each candidate acquiring from the Specialist, but not the Rack) but will not include the other candidates as acquirable attributes.
It sounds easy enough.
Less memory, less fuss, less disk space. To get an object from DTML, just say Candidates.getItem(id), and to create one, use Candidates.newItem(id), then populate its attributes.
You know, I probably should stop doing all this posting today so I can finish 0.3.0 for release tomorrow. There are still a few bits of cleanup and testing not done yet. Ty's also working on the necessary updates to LoginManager for compatibility with 0.3.0.
This is a production systems that is in *heavy* use. I don't feel comfortable with the fact that the RIPP will probably change quite a bit and break a few things. Also, it's all ZClass based. I am really in a bind as to using ZClasses for web access to development, or Python Products for mutability. All my best, Jason Spisak CIO HireTechs.com 6151 West Century Boulevard Suite 900 Los Angeles, CA 90045 P. 310.665.3444 F. 310.665.3544 Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
----- Original Message ----- From: "Jason Spisak" <444@hiretechs.com> To: "Phillip J. Eby" <pje@telecommunity.com> Cc: <zope-dev@zope.org> Sent: Thursday, May 04, 2000 3:46 PM Subject: Re: [Zope-dev] Warning! TransparentFolders and ZClasses
This is a production systems that is in *heavy* use. I don't feel comfortable with the fact that the RIPP will probably change quite a bit and break a few things. Also, it's all ZClass based. I am really in a bind as to using ZClasses for web access to development, or Python Products for mutability.
I'm very much a fan of ZClasses (as I think I've said before around here :) The biggest trouble I see with them is that the most "supported" method of changing the base classes of a ZClass seems to be to create a Python base class for each ZClass. While this is certainly doable, it's also somewhat annoying to have this non-web-based dependency for a product that is otherwise completely thru-the-web. Does anyone know what exactly the potential issue is with the hold-on-to-the-seat-of-your-pants method of base class changing? I know lots of people have done it without trouble... but, I bet a lot of people would prefer Zope had a more "supported" method of doing this through the web. Kevin
Kevin Dangoor writes:
----- Original Message ----- From: "Jason Spisak" <444@hiretechs.com> To: "Phillip J. Eby" <pje@telecommunity.com> Cc: <zope-dev@zope.org> Sent: Thursday, May 04, 2000 3:46 PM Subject: Re: [Zope-dev] Warning! TransparentFolders and ZClasses
This is a production systems that is in *heavy* use. I don't feel comfortable with the fact that the RIPP will probably change quite a bit and break a few things. Also, it's all ZClass based. I am really in a bind as to using ZClasses for web access to development, or Python Products for mutability.
I'm very much a fan of ZClasses (as I think I've said before around here :)
The biggest trouble I see with them is that the most "supported" method of changing the base classes of a ZClass seems to be to create a Python base class for each ZClass. While this is certainly doable, it's also somewhat annoying to have this non-web-based dependency for a product that is otherwise completely thru-the-web.
Does anyone know what exactly the potential issue is with the hold-on-to-the-seat-of-your-pants method of base class changing? I know lots of people have done it without trouble... but, I bet a lot of people would prefer Zope had a more "supported" method of doing this through the web.
Kevin
Just a thought about the "through the web" portion of things. If you use Johnathan's LocalFS products, then you have complete access with permissions to the python files through the web interface. Just a small adjust ment would make those files editable throught a text box like you have to do now with ZClass development. All my best, Jason Spisak CIO HireTechs.com 6151 West Century Boulevard Suite 900 Los Angeles, CA 90045 P. 310.665.3444 F. 310.665.3544 Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
From: "Jason Spisak":
Just a thought about the "through the web" portion of things. If you use Johnathan's LocalFS products, then you have complete access with permissions to the python files through the web interface. Just a small adjust ment would make those files editable throught a text box like you have to do now with ZClass development.
One big reason I'm not keen on developing Products in Python is the fact that you have to restart whenever you want to test a change. Also, I think I like the security controls placed on through-the-web products. Kevin
Kevin Dangoor writes:
From: "Jason Spisak":
Just a thought about the "through the web" portion of things. If you use Johnathan's LocalFS products, then you have complete access with permissions to the python files through the web interface. Just a small adjust ment would make those files editable throught a text box like you have to do now with ZClass development.
One big reason I'm not keen on developing Products in Python is the fact that you have to restart whenever you want to test a change.
Also, I think I like the security controls placed on through-the-web products.
Kevin
having never done anything but ZClass products, I have to agree. I am just wondering if I'll be able to change my existing Products to support the latest and greatest RIPP and others. The hold-on-to-your-butts method has me doubtful too. All my best, Jason Spisak CIO HireTechs.com 6151 West Century Boulevard Suite 900 Los Angeles, CA 90045 P. 310.665.3444 F. 310.665.3544 Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
At 03:44 PM 5/4/00 -0400, Kevin Dangoor wrote:
Does anyone know what exactly the potential issue is with the hold-on-to-the-seat-of-your-pants method of base class changing? I know lots of people have done it without trouble... but, I bet a lot of people would prefer Zope had a more "supported" method of doing this through the web.
The ZClass machinery uses several levels of metaclasses which are created at the time of setting the base classes. Unlike a Python class which is recreated when you edit the code, you are dealing with changing a class which has instance data (so to speak). And not just one, but at least *three* such classes, namely the class itself, its runtime propertysheets class, and its editing class. Each of these has dependencies on the original base class, not to mention data strucure issues for permissions and other Zope framework items. This is why the _setBasesHoldOnToYourButts() method creates a brand new ZClass to replace itself, then copies everything into it that was in the original class. Now, I'm not aware of any problems that *will* be caused by changing bases... but I have studied some bits of the ZClass code (and have even written code that "implements the notional subclassable propertysheets class interface") and let's just say if it makes Jim worried, it should make the rest of us even more so. :) Side note: Ty and I used to joke about this crazy ZClass variable called "_implements_the_notional_subclassable_propertysheets_class_interface"... until we found out what it actually *does*, and why we needed to use it in the ZPatterns framework to give objects pluggable propertysheets. And upon understanding it (and the other ZClass wizardry we discovered but have not exploited yet), our level of respect for the Zen of Master Fulton went up yet another notch. :) Anyway, so like I said, in this context, if Jim's worried about it, I'm *really* worried about it!
Kevin Dangoor wrote:
I'm very much a fan of ZClasses (as I think I've said before around here :)
The biggest trouble I see with them is that the most "supported" method of changing the base classes of a ZClass seems to be to create a Python base class for each ZClass. While this is certainly doable, it's also somewhat annoying to have this non-web-based dependency for a product that is otherwise completely thru-the-web.
Does anyone know what exactly the potential issue is with the hold-on-to-the-seat-of-your-pants method of base class changing? I know lots of people have done it without trouble... but, I bet a lot of people would prefer Zope had a more "supported" method of doing this through the web.
And this brings us right back again to TransparentFolders. One thing they provide is a form of inheritance (as opposed to acquisition). If there was a ZClassReference that was much like a TransparentFolder, but in fact refers to a ZClass somewhere else or a Python Base class, or anything, really. (ZClasses just shouldn't be confined to the product screen at all), you could easily change base classes by just dropping in new ones, deleting old ones, and so on. Is there any deep technical reason why something like this isn't feasible? It would just be a much nicer interface than now, when you can only easily change the base classes upon creation. Regards, Martijn
participants (5)
-
Jason Spisak -
Kevin Dangoor -
Martijn Faassen -
Phillip J. Eby -
R. David Murray