[Zope-dev] Re: any limits on object number?

[Dorneles Treméa]

Hello all,

> thanks to all the responses.  I will test with BTreeFolder2, and
> subfoldering with first character of id as another subfolder2, so that
> working with ZMI will also be possible without further scripting or
> customized views.  i will report the results back to the list.

just an additional info: I run into this problem recently...

I've got something around 40k objects to insert into ZODB.

I started doing some tests with normal Folders, but I discarded
it completely in favor of BTreeFolders. I got information that
BTreeFolder was already tested with more than 400k objects...
and that made me happy! :-)

But (there's always a 'but'!)...

Even using BTreeFolder to store all objects I was getting >300s
delay to show a single object (ok, it's an archetypes-based one,
containing >50 fields, splitted into 7 schematas, with lots of
fancy stuff...). So I made a directory hash structure based on
UID from each object (an AT UID is md5, so we have a hex base).

Using this hash structure, with 16 divisions, I got a better
performance (but still far from acceptable): 'only' ~100s to
show a single object... :-(

So I decided to create another level for the hash structure:
now each folder has another 16 folder inside it. This time I
was getting a delay of ~20s...

As you're already thinking... it's time for another hash level.
Now, with 16^3 additional BTreeFolders to split all my objects
I got an acceptable performance: ~3s...

The path is uggly as hell, f/f0/f0a/f0a5aac38aeff101b3168f2592dd879b,
but at least the system is usable...

Resuming what I've learned: don't abuse of BTreeFolder, hash
your content and live happy forever... ;-)

PS: The server is a modest PIII 1.2GHz, 1GB RAM with a 160MB/s
SCSI controller running only one instance of Zope 2.7.1 without
ZEO.

HTH,

-- 

Dorneles Treméa
Caxias do Sul - RS - Brasil
+55 54 9114 9312 - UIN: 2413568
X3ng Web Technology <http://www.x3ng.com.br>

-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GCS/IT d- s:->: a25 C+++ UBL++++$ P--- L++ E-- W+++
N++ o? K? w+ O M+ V-- PS+ PE- Y-- PGP++ t+ 5 X++ R+
tv+ b(++) DI+ D++ G+>+++ e++>++++ h---- r+++ y+++**
------END GEEK CODE BLOCK------