[ZODB-Dev] BTree memory bomb

[Tim Peters]

[Tim Peters]
>> ...
>> So if this program ran to completion, it would build about
>>
>> >>> 10000**2/15
>> 6666666
>>
>> OOBuckets.  If FileStorage is used, that also means we'll have an
>> approximately 7-million entry in-memory dict mapping the ~7 million
>> bucket object ids to their data records' current offsets in the .fs
>> file.

[Jim Fulton]
> This isn't a dict. It is a fsBTree, which should use around 10 bytes per
> object, so around 70 megabytes.

Oops!  I even knew that <wink>.  Thanks for the correction.

> That's not a huge amount of memory by today's standards.
>
> Of course, the serial cache could, in a sample program like this, use a
> lot more memory, because it does use a dictionary.

So same point in the end.

Here's the test driver again, using a fixed amount of new data per
iteration, and avoiding len(BTree) calls in the loops.  It does grow
steadily over time (probably due to various in-memory indexing structures),
to about 20MB RAM on my box after 300 outer-loop iterations (at which point
it was still running fine, I just got tired of tying up the machine with
it):

from ZODB import FileStorage, DB
from BTrees.OOBTree import OOBTree

from time import sleep

def main():
    storage=FileStorage.FileStorage('test.fs')
    db = DB(storage, cache_size=400)
    connection = db.open()
    root = connection.root()

    data = OOBTree()
  
    root[0] = data
    print "data:", len(data)

    for i in xrange(10000):
        for j in xrange(10000):
            data[i*10000+j] = "abc"
        get_transaction().commit()
        print "i:", i

    print "sleep"
    sleep(100)

main()