Toby is right about using the Catalog, I think (or some other form of indexing). The ObjectManager methods are not going to scale well because: 1 - An ObjectManager/Folder has no way of batching small sets of the large store. a. BTreeFolder 2 can batch, but only in its default sort (see 2) 2 - An ObjectManager/Folder has no way of sorting, unless you do it after the fact in presentation logic (slow/bloated). You should likely be using the catalog, and create a string-based FieldIndex containing the absolute physical path of the folder, and indexes for all the fields you might sort on. Batching with a Catalog query should just be a matter of getting a slice of the returned list of Brain objects. You can write a wrapper around this that will just get the objects those Brains refer to anyway, since it is likely your application will need them. What I might suggest is writing a simple product that creates a Catalog-backed folder: it effeciently returns sorted slices of all or a subset of objects in the folder, encapsulating access to the Catalog, and creating your own API for a slightly smarter folder. So you would be adding some smart methods on top of the existing dumb methods in the ObjectManager interface that the folder already exposes. Then, each folder can have its own Catalog specifically to act as a sorting engine, and its own batching code that uses the sorting engine. I'm not sure what the performance implications of this are for your application on write (yet another Catalog to write to), but the implications on read performance should improve, given what your application does. Note: supposedly, there are eventually going to be improvements to the ZCatalog sorting code (in Zope 2.7) that will improve its performance in general by some great degree. Sean -----Original Message----- From: Wankyu Choi [mailto:wankyu@neoqst.com] Sent: Friday, January 31, 2003 3:48 AM To: 'Toby Dickenson'; 'Mario Valente'; zope@zope.org Subject: RE: [Zope] Hardware for Zope + ZEO
Is there a way in Zope to limit the returned results as a limit clause would do in an SQL statement?
The best approach is to use ZCatalog. ZCatalog means you only need to load the articles into memory when you need them.
A threaded board needs to maintain its threaded structure. It's rather cumbersome with SQL but efficient. In Zope, it's a snap but gives poor performance. A folder (parent article) and its subfolders (its replies) strucutre directly turns into a thread hierarchy. I do use ZCatalog for all views of NeoBoard except for its default threaded look. Maybe I was stupid in not using ZCatalog for the threaded look. I'll give it a try. But I just wanted to know if there was a way to pull a limited set of objects from a **folderish object** not using ZCatalog. Not sure if your suggestion would work with NeoBoard yet, but I ran into a number of situations where I should pull all objects from a folder. I posted a simliar query months ago when I was trying to write my own user folder. No definate answer yet. The stock userfolder, for example, returns every user object no matter how many there are in response to a getUsers method call. I wouldn't use it with my 300,000 users. Nobody would, I suspect. There was no info on how to pull a limited set of users from a user folder. And guess we can't use ZCatalog to do the job. Simply put, are Zope folderish objects including BTreeFolderish objects **not** designed to hold this many objects? Well, if so, I'll just rewrite NeoBoard to use MySQL again. That would be a lot easier :( Thanks again for your time. Regards, Wankyu Choi --------------------------------------------------------------- To the dedicated staff at NeoQuest, language is not a problem to be dealt with, but an art waiting to be performed. --------------------------------------------------------------- Wankyou Choi CEO/President NeoQuest Communications, Inc. 3rd Floor, HMC Bldg., 730-14, Yoksam-dong, Kangnam-gu Seoul, Korea Tel: 82-2 - 501 - 7124 Fax: 82-2-501-7058 Corporate Home: http://www.neoqst.com Personal Home: http://www.zoper.net, http://www.neoboard.net e-mail: wankyu@neoqst.com --------------------------------------------------------------- _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Well, that seems to be the answer I've been looking for ;-) Thanks a lot for your comments and ideas. Guess I'll be writing a "smarter" folder then :-) Best Regards, Wankyu Choi --------------------------------------------------------------- To the dedicated staff at NeoQuest, language is not a problem to be dealt with, but an art waiting to be performed. --------------------------------------------------------------- Wankyou Choi CEO/President NeoQuest Communications, Inc. 3rd Floor, HMC Bldg., 730-14, Yoksam-dong, Kangnam-gu Seoul, Korea Tel: 82-2 - 501 - 7124 Fax: 82-2-501-7058 Corporate Home: http://www.neoqst.com Personal Home: http://www.zoper.net, http://www.neoboard.net e-mail: wankyu@neoqst.com --------------------------------------------------------------- -----Original Message----- From: sean.upton@uniontrib.com [mailto:sean.upton@uniontrib.com] Sent: Saturday, February 01, 2003 2:40 AM To: wankyu@neoqst.com; tdickenson@geminidataloggers.com; mvalente@ruido-visual.pt; zope@zope.org Subject: RE: [Zope] Hardware for Zope + ZEO Toby is right about using the Catalog, I think (or some other form of indexing). The ObjectManager methods are not going to scale well because: 1 - An ObjectManager/Folder has no way of batching small sets of the large store. a. BTreeFolder 2 can batch, but only in its default sort (see 2) 2 - An ObjectManager/Folder has no way of sorting, unless you do it after the fact in presentation logic (slow/bloated). You should likely be using the catalog, and create a string-based FieldIndex containing the absolute physical path of the folder, and indexes for all the fields you might sort on. Batching with a Catalog query should just be a matter of getting a slice of the returned list of Brain objects. You can write a wrapper around this that will just get the objects those Brains refer to anyway, since it is likely your application will need them. What I might suggest is writing a simple product that creates a Catalog-backed folder: it effeciently returns sorted slices of all or a subset of objects in the folder, encapsulating access to the Catalog, and creating your own API for a slightly smarter folder. So you would be adding some smart methods on top of the existing dumb methods in the ObjectManager interface that the folder already exposes. Then, each folder can have its own Catalog specifically to act as a sorting engine, and its own batching code that uses the sorting engine. I'm not sure what the performance implications of this are for your application on write (yet another Catalog to write to), but the implications on read performance should improve, given what your application does. Note: supposedly, there are eventually going to be improvements to the ZCatalog sorting code (in Zope 2.7) that will improve its performance in general by some great degree. Sean -----Original Message----- From: Wankyu Choi [mailto:wankyu@neoqst.com] Sent: Friday, January 31, 2003 3:48 AM To: 'Toby Dickenson'; 'Mario Valente'; zope@zope.org Subject: RE: [Zope] Hardware for Zope + ZEO
Is there a way in Zope to limit the returned results as a limit clause would do in an SQL statement?
The best approach is to use ZCatalog. ZCatalog means you only need to load the articles into memory when you need them.
A threaded board needs to maintain its threaded structure. It's rather cumbersome with SQL but efficient. In Zope, it's a snap but gives poor performance. A folder (parent article) and its subfolders (its replies) strucutre directly turns into a thread hierarchy. I do use ZCatalog for all views of NeoBoard except for its default threaded look. Maybe I was stupid in not using ZCatalog for the threaded look. I'll give it a try. But I just wanted to know if there was a way to pull a limited set of objects from a **folderish object** not using ZCatalog. Not sure if your suggestion would work with NeoBoard yet, but I ran into a number of situations where I should pull all objects from a folder. I posted a simliar query months ago when I was trying to write my own user folder. No definate answer yet. The stock userfolder, for example, returns every user object no matter how many there are in response to a getUsers method call. I wouldn't use it with my 300,000 users. Nobody would, I suspect. There was no info on how to pull a limited set of users from a user folder. And guess we can't use ZCatalog to do the job. Simply put, are Zope folderish objects including BTreeFolderish objects **not** designed to hold this many objects? Well, if so, I'll just rewrite NeoBoard to use MySQL again. That would be a lot easier :( Thanks again for your time. Regards, Wankyu Choi --------------------------------------------------------------- To the dedicated staff at NeoQuest, language is not a problem to be dealt with, but an art waiting to be performed. --------------------------------------------------------------- Wankyou Choi CEO/President NeoQuest Communications, Inc. 3rd Floor, HMC Bldg., 730-14, Yoksam-dong, Kangnam-gu Seoul, Korea Tel: 82-2 - 501 - 7124 Fax: 82-2-501-7058 Corporate Home: http://www.neoqst.com Personal Home: http://www.zoper.net, http://www.neoboard.net e-mail: wankyu@neoqst.com --------------------------------------------------------------- _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Hi all, Not sure if anyone is still interested in this thread ;-) I tried what Toby suggested, and indeed he was right: - Installed Zope 2.6.1 with the new Catalog code. ( impressive performance boost ^^ ) + Python 2.2 - Created a NeoBoard with 5,000 heavy articles containing heavy attachments ( simply creating this many articles took me more than a day ). - Changed the internals of NeoBoard to use ZCatalog when creating the threaded article list. The major problem with this scheme was how to sort the result set in a such a way that NeoBoard would still retain its threaded look. For a couple of reasons, I need two sort keys: one to sort the threads (parents) and the other to sort their replies (childs). ZCatalog supports a single sort index, so I had to come up with my own solution: slicing the result set. Since we always need only a part of the whole result set that needs to be viewed in the current request, NeoBoard only needs to sort that portion. Admittedly, the previous versions of NeoBoard couldn't handle that many articles without hogging the resources. After changing the internals, it pulls and sorts 5,000 articles ( brains to be exact ) and displays part of them ( a pageful of them ) in less than a second even on my P4 desktop. ( still not so fast as the SQL version, though ;-( Regarding the hardware... I bought two servers: a 2-way Xeon server with a raid device (MegaRAID dual-channel controller/raid level 5 ) and a 2-way tualatin P3 server. The Xeon server runs ZEO with Toby's DirectoryStorage ( ReiserFS on 290+G raid 5 -- 72G SCSI HDD * 6 --) with the tualatin server being a ZEO client. Will add two more ZEO clients to this setup. Guess it is a viable setup for the needs I mentioned in this thread. So far, so good. Pretty fast and pretty stable. Thanks for the great product and tips, Toby. I also want to thank the others who shared their ideas in this thread. All the best, Wankyu Choi -----Original Message----- From: sean.upton@uniontrib.com [mailto:sean.upton@uniontrib.com] Sent: Saturday, February 01, 2003 2:40 AM To: wankyu@neoqst.com; tdickenson@geminidataloggers.com; mvalente@ruido-visual.pt; zope@zope.org Subject: RE: [Zope] Hardware for Zope + ZEO Toby is right about using the Catalog, I think (or some other form of indexing). The ObjectManager methods are not going to scale well because: 1 - An ObjectManager/Folder has no way of batching small sets of the large store. a. BTreeFolder 2 can batch, but only in its default sort (see 2) 2 - An ObjectManager/Folder has no way of sorting, unless you do it after the fact in presentation logic (slow/bloated). You should likely be using the catalog, and create a string-based FieldIndex containing the absolute physical path of the folder, and indexes for all the fields you might sort on. Batching with a Catalog query should just be a matter of getting a slice of the returned list of Brain objects. You can write a wrapper around this that will just get the objects those Brains refer to anyway, since it is likely your application will need them. What I might suggest is writing a simple product that creates a Catalog-backed folder: it effeciently returns sorted slices of all or a subset of objects in the folder, encapsulating access to the Catalog, and creating your own API for a slightly smarter folder. So you would be adding some smart methods on top of the existing dumb methods in the ObjectManager interface that the folder already exposes. Then, each folder can have its own Catalog specifically to act as a sorting engine, and its own batching code that uses the sorting engine. I'm not sure what the performance implications of this are for your application on write (yet another Catalog to write to), but the implications on read performance should improve, given what your application does. Note: supposedly, there are eventually going to be improvements to the ZCatalog sorting code (in Zope 2.7) that will improve its performance in general by some great degree. Sean -----Original Message----- From: Wankyu Choi [mailto:wankyu@neoqst.com] Sent: Friday, January 31, 2003 3:48 AM To: 'Toby Dickenson'; 'Mario Valente'; zope@zope.org Subject: RE: [Zope] Hardware for Zope + ZEO
Is there a way in Zope to limit the returned results as a limit clause would do in an SQL statement?
The best approach is to use ZCatalog. ZCatalog means you only need to load the articles into memory when you need them.
A threaded board needs to maintain its threaded structure. It's rather cumbersome with SQL but efficient. In Zope, it's a snap but gives poor performance. A folder (parent article) and its subfolders (its replies) strucutre directly turns into a thread hierarchy. I do use ZCatalog for all views of NeoBoard except for its default threaded look. Maybe I was stupid in not using ZCatalog for the threaded look. I'll give it a try. But I just wanted to know if there was a way to pull a limited set of objects from a **folderish object** not using ZCatalog. Not sure if your suggestion would work with NeoBoard yet, but I ran into a number of situations where I should pull all objects from a folder. I posted a simliar query months ago when I was trying to write my own user folder. No definate answer yet. The stock userfolder, for example, returns every user object no matter how many there are in response to a getUsers method call. I wouldn't use it with my 300,000 users. Nobody would, I suspect. There was no info on how to pull a limited set of users from a user folder. And guess we can't use ZCatalog to do the job. Simply put, are Zope folderish objects including BTreeFolderish objects **not** designed to hold this many objects? Well, if so, I'll just rewrite NeoBoard to use MySQL again. That would be a lot easier :( Thanks again for your time. Regards, Wankyu Choi --------------------------------------------------------------- To the dedicated staff at NeoQuest, language is not a problem to be dealt with, but an art waiting to be performed. --------------------------------------------------------------- Wankyou Choi CEO/President NeoQuest Communications, Inc. 3rd Floor, HMC Bldg., 730-14, Yoksam-dong, Kangnam-gu Seoul, Korea Tel: 82-2 - 501 - 7124 Fax: 82-2-501-7058 Corporate Home: http://www.neoqst.com Personal Home: http://www.zoper.net, http://www.neoboard.net e-mail: wankyu@neoqst.com --------------------------------------------------------------- _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Wankyu Choi wrote at 2003-3-27 05:24 +0900:
.... I need two sort keys: one to sort the threads (parents) and the other to sort their replies (childs). ZCatalog supports a single sort index,
I have code to extend ZCatalog's sorting capability to arbitrary many sort keys. Maybe, we want something like this in the Zope core? Dieter
This might be something for 2.7 if is compatible with older versions. -aj --On Samstag, 29. März 2003 0:26 Uhr +0100 Dieter Maurer <dieter@handshake.de> wrote:
Wankyu Choi wrote at 2003-3-27 05:24 +0900:
.... I need two sort keys: one to sort the threads (parents) and the other to sort their replies (childs). ZCatalog supports a single sort index,
I have code to extend ZCatalog's sorting capability to arbitrary many sort keys.
Maybe, we want something like this in the Zope core?
Dieter
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Dieter Maurer wrote:
Wankyu Choi wrote at 2003-3-27 05:24 +0900:
.... I need two sort keys: one to sort the threads (parents) and the other to sort their replies (childs). ZCatalog supports a single sort index,
I have code to extend ZCatalog's sorting capability to arbitrary many sort keys.
Maybe, we want something like this in the Zope core?
Yes PLEASE! What does the code look like? Anything I can do to help? cheers, Chris
Chris Withers wrote at 2003-3-31 10:24 +0100:
Dieter Maurer wrote:
I have code to extend ZCatalog's sorting capability to arbitrary many sort keys.
Maybe, we want something like this in the Zope core?
Yes PLEASE!
What does the code look like?
Anything I can do to help?
Attached. Dieter
I plan to work some more on the sort functionality for 2.7. This functionality was a consideration so the patch is welcome. Feel free to stick it in the collector. -Casey On Friday 28 March 2003 06:26 pm, Dieter Maurer wrote:
Wankyu Choi wrote at 2003-3-27 05:24 +0900:
.... I need two sort keys: one to sort the threads (parents) and the other to sort their replies (childs). ZCatalog supports a single sort index,
I have code to extend ZCatalog's sorting capability to arbitrary many sort keys.
Maybe, we want something like this in the Zope core?
Dieter
participants (6)
-
Andreas Jung -
Casey Duncan -
Chris Withers -
Dieter Maurer -
sean.upton＠uniontrib.com -
Wankyu Choi