[ZODB-Dev] Pack using loads of RAM on FileStorage

[Jeremy Hylton]

>>>>> "CW" == Chris Withers <chrisw@nipltd.com> writes:

  CW> Hi, While on the subject of excessive RAM usage, I have
  CW> mentioned this before, but has anyone else noticed packing
  CW> FileStorage's taking up loads of RAM?

I'm not too familiar with the pack() implementation in FileStorage,
but this seems consistent with its known memory problems.  FileStorage
uses a Python dictionary to map oids to file offsets.  Since it's a
Python dictionary, it is always in memory -- and each entry takes a
fair amount of space, perhaps 100 bytes per object.  

When you do a pack90, you end up with at least *two* such
dictionaries: one for the old locations, one for the new locations.
There might be other temporary dictionaries.  A good rule of thumb may
be that pack() doubles the memory used by FileStorage.

Given the memory problems you're having, it might make sense to try to
the latest CVS version of bsddb3Storage.Full.  You need to be sure to
set the BerkeleyDB cache size to a reasonable value.  Once you do, the
storage shouldn't use more than that amount of memory.

Jeremy