[ZODB-Dev] Space used by IOBTrees

[Jeremy Hylton]

On Thu, 2003-02-27 at 15:12, Andreas Jung wrote:
> A XML document with about 26.000 nodes (1.3 MB data) is represented
> in an application by a nested structure of nodes where the childs
> are stored in IOBTrees (17.000 nodes out of the 26.000 nodes are leafs).
> The complete object allocates about 20MB inside the ZODB. analyze.py
> shows that there are about 9600 IOBTree objects inside the ZODB with an
> average size of 1850 bytes. The amount of data stored as attributes
> of a single treenode instance is very small (not more than 100-200 bytes).
> So why is the pickle of an IOBTree nearly 2KB large instead of several
> hundred bytes?

It's hard to follow the details here.  Concrete examples might help a
lot to convey what it is you are doing.

If you want to know what takes up the space in the BTrees pickle, you
should probably load an individual database record and look at it.  I
think that will give you a better answer than anyone on this list :-).
We can probably help you read the pickle if it doesn't make sense.  The
pickletools module that's new in Python 2.3a2 would also help.

My first guess would be that the pickled representation of whatever
objects you're storing as the values of the BTree uses more space than
you think.  Also recall that each BTree node is stored as a separate
database record, so you don't get to share a memo across records.  For
example, =the fully qualified name of the class for the value object is
stored in every database record.

Jeremy