[Zope-dev] any limits on object number?

[Jens Vagelpohl]

> When we tried to upload objects from wikipedia's sqldump of after about
> 40,000 objects in a folder the script went for a toss. roughly each 
> time
> we ran the script the same problem.  even to visit the folder of even
> 1000 objects through ZMI is a pain.  I think that is where BTreeFolder2
> will be very useful.

Your observations make sense in the context of the standard Zope 
folder. BTreeFolder2 will help. I am assuming whatever script you use 
to populate the folder(s) commits the transaction ever N objects, 
right?


> I am also thinking to use a squid cache style of storing objects in
> zodb, where the objects go into many subfolders (i dont know what is
> this method called technically) with  several sub folders with names
> alphabetized/numbered.  this way i can store large number within zodb
> though not in the same folder/BTreefolder2.
>
> Should I use both methods?

I would definitely use the subfolder method.


> Not related to Zope, but what about filesystems? there must be a limit
> on the number of files within one folder depending on which filesystem
> one uses.

The ZODB, in case you use the standard FileStorage, is one monolithic 
file. So there is no large number of files on the server file system 
regardless of how many objects you store.

For files on the file systems your choice of file system depends not 
just on the number of objects but also its size. To re-use your earlier 
example of Squid cache files, which are numerous and mostly small, 
ReiserFS is normally recommended. It deals much better with large 
quantities of files in one folder.

jens