[ZODB-Dev] Old memory leak related to the ObjectWriter?

[Tim Peters]

[Jeremy Hylton]
> What version of Python?  I believe we added cyclic GC support to
> picklers to address this problem,

That sounds right.

> but that was probably for Python 2.4.

But that doesn't.  Cyclic gc wasn't new in 2.4, and for ZODB 3.3 the first
release that even had a 2.4 Windows installer was 3.3 final.  We developed
with then-current Python 2.3 all along.

Pierre, why do you suspect there's a memory leak?  A cycle isn't evidence of
a leak in Python 2.3+, and when you do (as you did)

    gc.set_debug(gc.DEBUG_LEAK)

then _all_ objects in cyclic trash get put into gc.garbage, regardless of
whether they're collectable (DEBUG_LEAK implies DEBUG_SAVEALL -- see the
docs, although I'm not sure it's possible to understand all implications
fully without studying Python's gcmodule.c; I happen to be an expert on
that, so I have an unfair advantage;-)):

>>> import gc
>>> gc.set_debug(gc.DEBUG_LEAK)
>>> x = []
>>> x.append(x) # create a cycle
>>> del x       # make it unreachable
>>> gc.collect()
gc: collectable <list 008CAA50>
1
>>> gc.garbage  # because of DEBUG_LEAK, it ends up in gc.garbage
[[[...]]]

When I run your program, under Python 2.3.5 and current ZODB 3.4 branch, I
get this output on the console:

gc: collectable <dict 00A0A390>
gc: collectable <ObjectWriter instance at 00A5A850>
gc: collectable <dict 00A0A4B0>
gc: collectable <cPickle.Pickler 009DAE10>
gc: collectable <instancemethod 00A5A828>
gc: collectable <list 00C3FE30>
gc: collectable <dict 00A0A660>
gc: collectable <tuple 008CA890>
gc: collectable <tuple 00A13378>
gc: collectable <tuple 00C410A8>
gc: collectable <tuple 00C328F0>
gc: collectable <tuple 00A139E0>
gc: collectable <tuple 00C32F30>
gc: collectable <tuple 00C32E90>
gc: collectable <tuple 00C383F0>
gc: collectable <tuple 00A20030>
gc: collectable <tuple 009E7288>

Note that all those say "collectable":  gc _would_ reclaim them, except you
told it not to (by setting DEBUG_LEAK).

You get different output if you create something gc couldn't clean up:

>>> import gc
>>> gc.set_debug(gc.DEBUG_LEAK)
>>> class C:
...     def __del__(self):
...         print "stopping gc"
...
>>> c = C()
>>> c.c = c  # create a cycle
>>> del c    # make it unreachable
>>> gc.collect()
gc: uncollectable <C instance at 008D0FA8>
gc: uncollectable <dict 008D1540>
2

That cycle can't be reclaimed, because it contains an object with a __del__
method.  Note that the console output says "uncollectable" instead of
"collectable".

Finally, if I remove the gc.set_debug(gc.DEBUG_LEAK) from your program,
there's no output at all, and resultConnectionLeak.txt remains empty:  no
trash is left behind.

In short, there certainly are cycles here, but there's no evidence of a
leak.