(cc Shane too) On Wednesday 10 Jul 2002 7:01 pm, Dieter Maurer wrote:
How many items can I have in a folder?
Normal folders store their content in a tuple. If you access the folder, the complete tuple is fetched into memory.
Hmm, not quite. it stores the sub-objects in the folder's __dict__. It does have a tuple which stores the objects "id"s (so it can know which attributes are ObjectManager-managed, and which are not) and a cached copy of the meta-type.
You do not want this for large numbers of items. Use a BTreeFolder in this case. As the name tells, it uses a tree structure to store the content. Access is far more fine grained than with standard Folders.
BTreeFolder was definitely a huge advantage before Zope 2.6, because the old ZODB cache did not cope well with the fact that the __dict__ loaded all 40,000 sub-objects into memory as ghosts. Zope 2.6 has a different cache manager that does not panic when it is given huge numbers of ghosts. As a rough guess each ghost adds 100 bytes, so BTreeFolder is saving you 4M of ram (per worker thread). Not bad, but maybe not enough to justify installing a seperate product. BTreeFolder does give you are more scalable user interface as standard, but with that many you still might want to think about replacing it with something customised to your data.
I'm looking to be able to handle at least 40,000 user objects and a similar number of other larger data objects of various sizes (2k - 20k+). Users could be pushed off to an ldap server or some other environment via the LoginManager Mod etc, however I would still have 40,000 user folders in a single folder.
With a BTreeFolder, I would not be worried with this number.
BTreeFolder has a problem that it doesnt store *all* of its data in the BTree. It still has the tuple caching id and meta-type, thanks to it ObjectManager base class. In this case it is a 40,000 element tuple. That would be enough to get me worried. (ps; I worry easily)