[Zope3-Users] MemoryError Evolving a ZODB

[Marius Gedminas]

On Wed, Dec 12, 2012 at 12:39:18AM -0800, Jeroen Michiel wrote:
> Thanks for the reply!
> 
> I already tried
> transaction.savepoint()
> every minute, but that didn't help: I only saw the memory usage dropping the
> first time, but never after.
> 
> I changed the code to what you suggested, but it still doesn't seem to help.
> Something must be wrong somewhere along the line, but I don't have a clue
> where to begin looking.
> Would using something like guppy (or heapy, or what it's called) reveal
> something?

(I tried to figure out guppy/heapy once, gave up.)

> Could it be something about objects with circular references not being able
> to be garbage-collected?
> The objects in my DB are quite complex, so something like that might
> actually be happening.

Usually when a Python process eats too much memory, your server ends up
in swappy death land.  My gut feeling is that MemoryError means a
corrupted pickle that tries to allocate a large amount of memory (e.g.
multiple gigabytes) all in one go.

Can you add a try:/except MemoryError: import pdb; pdb.set_trace() in
there?  See if the process memory usage is really big.  Write down the
object OID (ZODB.utils.u64(obj._p_oid)), try to load it from a separate
Python script process (with the same sys.path, so custom classes are
unplickleable):

  db = ZODB.DB.DB(ZODB.FileStorage.FileStorage('Data.fs', read_only=True))
  conn = db.open()
  obj = conn.get(ZODB.utils.p64(0xXXXXX))    # creates a ghost
  try:
      obj._p_activate()                      # tries to load it
  except MemoryError:
      import pdb; pdb.set_trace()

See if you get a memory error there.  If so, try do disassemble the
pickle (pickletools.dis) maybe -- if you've got pdb, you can find the
pickle itself one stack frame up.

Marius Gedminas
-- 
I want patience, and I WANT IT NOW!
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: Digital signature
URL: <http://mail.zope.org/pipermail/zope3-users/attachments/20121212/8273ab41/attachment.sig>