[Zope] BTreeFolders + Catalog + lots of objects?

[Chris Withers]

sean.upton@uniontrib.com wrote:
> 
> Has anyone used the BTreeFolder product to store hundreds-of-thousands or
> millions of objects?

Nope.

> I'm developing an internal CRM system that will contain somewhere between
> 300k-500k records, stored in a back-end relational datastore, and exposed
> via metadata proxy objects (1-per-record) sitting in a container subclassed
> from BTreeFolder; 

Is there any particular reason the proxy objects need to live in the ZODB?

> I am wondering if anybody has done anything similar to this, in terms of
> number of objects stored in a BTreeFolder, and the type of storage that they
> used.  I'm also wondering about anyone using ZCatalog for such a large
> number of indexed objects.

I'm having infinite ammounts of fun (not!) attempting to index 40,000 word
documents. It largely depends on what types of indexes you will be using on
these objects and what size the objects themselves are.

If the objects are anything more than extremely simple and small proxies, I
reckon you'll run into problems with BTreeFolder. Likewise, if you're indexing
is anything other than simple Field indexing (and with that number of objects,
even that may be enough) you won't successfully index that many objects.

It's go for sticking the lot in your relational backend.

cheers,

Chris (who's had his faith in ZODB scalability systematically destroyed over the
last couple of months :-( )
> 
> I'm also thinking of overriding BTreeFolder.manage_main_listing() with a
> user interface that allows users to simply type in an object id into a text
> box, instead of listing them (the string object ids correspond to a long
> integer from 1..n); it wouldn't work very well to list half-a-million
> objects in a html-form select control...  Ideally, I'd do batching of some
> sort, but I'm having trouble figuring out how I would do that.
> 
> Anyone have any thoughts?
> 
> Sean
> 
> =========================
> Sean Upton
> Senior Programmer/Analyst
> SignOnSanDiego.com
> The San Diego Union-Tribune
> 619.718.5241
> sean.upton@uniontrib.com
> =========================
> 
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )