[Zope-DB] Re: BTreeFolder

[Toby Dickenson]

(cc Shane too)

On Wednesday 10 Jul 2002 7:01 pm, Dieter Maurer wrote:

>  > How
>  > many items can I have in a folder?
>
> Normal folders store their content in a tuple.
> If you access the folder, the complete tuple is fetched into memory.

Hmm, not quite. it stores the sub-objects in the folder's __dict__. It do=
es=20
have a tuple which stores the objects "id"s (so it can know which attribu=
tes=20
are ObjectManager-managed, and which are not) and a cached copy of the=20
meta-type.

> You do not want this for large numbers of items.
> Use a BTreeFolder in this case. As the name tells, it uses
> a tree structure to store the content. Access is far more fine
> grained than with standard Folders.

BTreeFolder was definitely a huge advantage before Zope 2.6, because the =
old=20
ZODB cache did not cope well with the fact that the __dict__ loaded all=20
40,000 sub-objects into memory as ghosts.

Zope 2.6 has a different cache manager that does not panic when it is giv=
en=20
huge numbers of ghosts. As a rough guess each ghost adds 100 bytes, so=20
BTreeFolder is saving you 4M of ram (per worker thread). Not bad, but may=
be=20
not enough to justify installing a seperate product. BTreeFolder does giv=
e=20
you are more scalable user interface as standard, but with that many you=20
still might want to think about replacing it with something customised to=
=20
your data.

>  > I'm looking to be able to handle at least 40,000 user objects and a
>  > similar number of other larger data objects of various sizes (2k -
>  > 20k+).  Users could be pushed off to an ldap server or some other
>  > environment via the LoginManager Mod etc, however I would still have
>  > 40,000 user folders in a single folder.
>
> With a BTreeFolder, I would not be worried with this number.

BTreeFolder has a problem that it doesnt store *all* of its data in the B=
Tree.=20
It still has the tuple caching id and meta-type, thanks to it ObjectManag=
er=20
base class. In this case it is a 40,000 element tuple. That would be enou=
gh=20
to get me worried.

(ps; I worry easily)