[ZODB-Dev] BTree memory bomb

[Tim Peters]

[Jeremy Hylton]
...
> In your example, the keys 10,000,000 through 10,009,999 each store a
> 128KB string as the value.  If a single bucket holds 50 items (I'm making
> that number up), it will use roughly 40MB of memory.  The program is
> going to make thousands of buckets that require this much memory or more,
> so it's not going to be effective at limit memory consumption.

Concretely, an OOBTree generally holds at most 30 entries per bucket, and
when entries are inserted in sorted order to begin with (as the test driver
does) it averages closer to 15 entries per bucket (if your actual app has
integer keys and values, an IIBTree is a better choice, where the
entries/bucket max changes from 30 to 120).

So if this program ran to completion, it would build about

>>> 10000**2/15
6666666

OOBuckets.  If FileStorage is used, that also means we'll have an
approximately 7-million entry in-memory dict mapping the ~7 million bucket
object ids to their data records' current offsets in the .fs file.  Dicts
can grow that large, but FileStorage really isn't suited to many millions of
distinct objects.  Perhaps DirectoryStorage would work better for that:

    http://dirstorage.sourceforge.net/

Simon, the repeated "lesson" here is that to the extent your test program
doesn't model actual application behavior, it's going to have you staring at
problems you won't have in real life (multi-gigabyte transactions, many
millions of distinct persistent objects, and so on).