[ZODB-Dev] ZODB memory problems (was: processing a Very Large file)

[Jeremy Hylton]

On 5/21/05, DJTB <usenet at terabytemusic.cjb.net> wrote:
> [posted to comp.lang.python, mailed to zodb-dev at zope.org]

[Following up to both places.]

> I'm having problems storing large amounts of objects in a ZODB.
> After committing changes to the database, elements are not cleared from
> memory. Since the number of objects I'd like to store in the ZODB is too
> large to fit in RAM, my program gets killed with signal 11 or signal 9...

The problem here is a common one with a first attempt at using ZODB. 
The problem is that ZODB manages memory at the granularity of
first-class persistent objects --  that is, instances of classes that
inherit from Persistent.  ZODB can move such objects in and out of
memory at transaction boundaries, which allows your application to use
many more objects than it has physical memory for.

It looks like your application has a single persistent instance -- the
root ExtendedTupleTable -- so there's no way for ZODB to manage the
memory.  That object and everything reachable from it must be in
memory at all times.

You need to re-structure the program so that is has more first-class
persistent objects.  If, for example, the ExtendedTuple objects
inherited from Persistent, then they could reside on disk except when
you are manipulating them.

The ObjectInterning instance is another source of problem, because
it's a dictionary that has an entry for every object you touch.  The
various other dictionaries in your program will also be memory hogs in
they have very many entries.  The typical way to structure a ZODB
program is to use one of the BTrees implementation types instead of a
dictionary, because the BTree does not keep all its keys and values in
memory at one time.  (Its internal organization is a large collection
of first-class persistent objects representing the BTree buckets and
internal tree nodes.)

You must use some care with BTrees, because the data structure
maintains a total ordering on the keys.  (And a dictionary does not.) 
  The ZODB/ZEO programming guide has a good section on BTrees here:
http://www.zope.org/Wikis/ZODB/guide/node6.html

Jeremy