We have some BTreeFolder2's containing hundreds of thousands of objects, even one containing over 2 million instances of Folder. That is really no problem in itself. Under our setup, beyond 300-400,000 contained objects, the batching provided by getBatchObjectListing becomes unusable. I've seen this problem mentioned somewhere but never any hint of a solution. We have a new application that will require pagination, or ideally, the ability to go forward and back through the items (any order). Is there a way to refer to the contained items by something like offset? It might be easy if the objects had sequential, consecutive numeric ids, but sadly this cannot be the case. We are using the BTreeFolder2-1.0.2 version that came with Zope 2.10. Thanks. Ken
What kind of practical sense does it make to batch 400k objects? I can no imagine single usecase where one would be interested in walking through such an amount of objects manually. Better organize your data in a more handy way or implement some search logic for bringing the batch size to a number of relevant object.s -aj On Mon, Nov 2, 2009 at 00:04, Ken Ara <feedreader@yahoo.com> wrote:
Under our setup, beyond 300-400,000 contained objects, the batching provided by getBatchObjectListing becomes unusable. I've seen this problem mentioned somewhere but never any hint of a solution.
I agree it is hard to imagine, but I am just the web guy... If I dare to guess, I would say that what they want to do is improve access to the information for the users. The two million objects are strongly interlinked, so this is the navigation system used up to now. But why not open it up to browsing? For the content team, it might also be nice to use the ZMI or similar when working with this content. We have not been successful to create a Catalog of this many objects. The process seemed to time out after many hours, probably hardware-bound. Also, the size of the resulting ZODB is of concern, but we may try again with the Catalog on a mounted database. My general question remains: Is there a way to address the objects contained in a BTreeFolder2 using something like an 'offset' or other identifier? Has anyone found a strategy that scales better than getBatchObjectListing? Thanks Ken --- On Mon, 11/2/09, Andreas Jung <lists@zopyx.com> wrote: From: Andreas Jung <lists@zopyx.com> Subject: Re: [Zope] Large BTreeFolder2 batching/pagination To: "Ken Ara" <feedreader@yahoo.com> Cc: zope@zope.org Date: Monday, November 2, 2009, 6:23 AM What kind of practical sense does it make to batch 400k objects? I can no imagine single usecase where one would be interested in walking through such an amount of objects manually. Better organize your data in a more handy way or implement some search logic for bringing the batch size to a number of relevant object.s -aj On Mon, Nov 2, 2009 at 00:04, Ken Ara <feedreader@yahoo.com> wrote: Under our setup, beyond 300-400,000 contained objects, the batching provided by getBatchObjectListing becomes unusable. I've seen this problem mentioned somewhere but never any hint of a solution.
Am 02.11.09 11:28, schrieb Ken Ara:
I agree it is hard to imagine, but I am just the web guy...
If I dare to guess, I would say that what they want to do is improve access to the information for the users. The two million objects are strongly interlinked, so this is the navigation system used up to now. But why not open it up to browsing?
Because no human can deal in a reasonable way with 500k object..build your *own* custom and working navigation throughout the data records...presenting the ZMI view for BTreefolders to a human is just sick.
For the content team, it might also be nice to use the ZMI or similar when working with this content.
We have not been successful to create a Catalog of this many objects. The process seemed to time out after many hours, probably hardware-bound. Also, the size of the resulting ZODB is of concern, but we may try again with the Catalog on a mounted database.
we have ZCatalogs with millions of objects..likely you are indexing everything within one big transaction without using savepoints or subtransactions..
-aj
I should also mention that one of these large collections contains objects with sequential, but not consecutive, numeric ids (timestrings) like some blogs. Users of this database can benefit from the ability to access items added around the same time. With or without a catalog, how would one get the next or previous id in such a case? --- On Mon, 11/2/09, Andreas Jung <lists@zopyx.com> wrote: From: Andreas Jung <lists@zopyx.com> Subject: Re: [Zope] Large BTreeFolder2 batching/pagination To: "Ken Ara" <feedreader@yahoo.com> Cc: zope@zope.org Date: Monday, November 2, 2009, 11:36 AM Am 02.11.09 11:28, schrieb Ken Ara:
I agree it is hard to imagine, but I am just the web guy...
If I dare to guess, I would say that what they want to do is improve access to the information for the users. The two million objects are strongly interlinked, so this is the navigation system used up to now. But why not open it up to browsing?
Because no human can deal in a reasonable way with 500k object..build your *own* custom and working navigation throughout the data records...presenting the ZMI view for BTreefolders to a human is just sick.
For the content team, it might also be nice to use the ZMI or similar when working with this content.
We have not been successful to create a Catalog of this many objects. The process seemed to time out after many hours, probably hardware-bound. Also, the size of the resulting ZODB is of concern, but we may try again with the Catalog on a mounted database.
we have ZCatalogs with millions of objects..likely you are indexing everything within one big transaction without using savepoints or subtransactions..
-aj
participants (2)
-
Andreas Jung -
Ken Ara