[Zope] BTreeFolders + Catalog + lots of objects?

[sean.upton@uniontrib.com]

Has anyone used the BTreeFolder product to store hundreds-of-thousands or
millions of objects?

I'm developing an internal CRM system that will contain somewhere between
300k-500k records, stored in a back-end relational datastore, and exposed
via metadata proxy objects (1-per-record) sitting in a container subclassed
from BTreeFolder; these objects will have meta-data cataloged in a ZCatalog
for a data management interface.  All this will sit in a non-undo storage
(for now, Ty Sarna's Berkeley DB 2 storage, since that is what I am using
now on my production ZEO storage servers, but eventually it will be migrated
to a bsddb3 packless storage).  These metadata proxy objects would likely be
fully recreated from the underlying relational database on a weekly basis.  

I am wondering if anybody has done anything similar to this, in terms of
number of objects stored in a BTreeFolder, and the type of storage that they
used.  I'm also wondering about anyone using ZCatalog for such a large
number of indexed objects.  

I'm also thinking of overriding BTreeFolder.manage_main_listing() with a
user interface that allows users to simply type in an object id into a text
box, instead of listing them (the string object ids correspond to a long
integer from 1..n); it wouldn't work very well to list half-a-million
objects in a html-form select control...  Ideally, I'd do batching of some
sort, but I'm having trouble figuring out how I would do that.

Anyone have any thoughts?

Sean

=========================
Sean Upton
Senior Programmer/Analyst
SignOnSanDiego.com
The San Diego Union-Tribune
619.718.5241
sean.upton@uniontrib.com
=========================