thanks to all the responses. I will test with BTreeFolder2, and subfoldering with first character of id as another subfolder2, so that working with ZMI will also be possible without further scripting or customized views. i will report the results back to the list.
just an additional info: I run into this problem recently...
I've got something around 40k objects to insert into ZODB.
I started doing some tests with normal Folders, but I discarded it completely in favor of BTreeFolders. I got information that BTreeFolder was already tested with more than 400k objects... and that made me happy! :-)
But (there's always a 'but'!)...
Even using BTreeFolder to store all objects I was getting >300s delay to show a single object (ok, it's an archetypes-based one, containing >50 fields, splitted into 7 schematas, with lots of fancy stuff...). So I made a directory hash structure based on UID from each object (an AT UID is md5, so we have a hex base).
Using this hash structure, with 16 divisions, I got a better performance (but still far from acceptable): 'only' ~100s to show a single object... :-(
So I decided to create another level for the hash structure: now each folder has another 16 folder inside it. This time I was getting a delay of ~20s...
As you're already thinking... it's time for another hash level. Now, with 16^3 additional BTreeFolders to split all my objects I got an acceptable performance: ~3s...
The path is uggly as hell, f/f0/f0a/f0a5aac38aeff101b3168f2592dd879b, but at least the system is usable...
Resuming what I've learned: don't abuse of BTreeFolder, hash your content and live happy forever... ;-)
I very much doubt you solved the problem you think you solved. Access time to a single object in the hundred of seconds is not a BTreeFolder problem. BTreeFolder is designed to not be a bottleneck for concurrent access and large number of objects. You should have benched (using ZopeProfiler for instance) to find out where time really is spent. Maybe some of your/AT's code does a stupid loop on folder.objectIds() or something. Florent -- Florent Guillaume, Nuxeo (Paris, France) +33 1 40 33 79 87 http://nuxeo.com mailto:fg@nuxeo.com