[Zope-dev] debugging memory leaks

[Tim Peters]

[Seb Bacon]
> ...
> I know from the refcounts in Zope that items of class Foo are
> definitely leaking, yet when I do a sys.getobjects(0, Foo) I get
> nothing back.

If Foo is an old-style class, then every instance of Foo has type
InstanceType (and so does every instance of every other old-style class):

>>> class Foo: pass

>>> type(Foo())
<type 'instance'>
>>> import types
>>> types.InstanceType
<type 'instance'>
>>> types.InstanceType is type(Foo())
True
>>>

getobjects() filters on type, so nothing will ever match Foo as a type.  If
you can change Foo to a new-style class (most easily by inheriting from
object, in a recent-enough Python), life gets easier:

>>> class Foo(object): pass

>>> type(Foo())
<class '__main__.Foo'>
>>>

Then getobjects() can filter on Foo as a type.  Classes and types before
Python 2.2 are distinct concepts (barring Zope ExtensionClass
complications).

> Navigating all 100000-ish references one by one seems a
> bit daunting.

Na, with list comprehension syntax (for brevity -- you can do the same with
a for-loop, of course),

    foos = [x for x in sys.getobjects(0)
              if isinstance(x, types.InstanceType) and
                 x.__class__ is Foo]

will extract just the Foo instances (if Foo is an old-style class).

> So I'm not sure where to go from here.

Debugging memory leaks can be hard, in any language.  Another place to look
for ideas is in the top-level test.py from a current Zope HEAD checkout (or
2.7 branch, or Zope3).  If you're able to run your code in a loop, the
TrackRefs class in test.py automates some measure of identifying what (if
anything) is leaking.  We've changed many internal ZODB and ZEO classes to
new-style classes primarily just so this test.py's -r option is more useful
in identifying the source of leaks.  Some yielded easily to analysis, others
slobbered on for part-time weeks.

There are two common culprits:

1. Some class keeps a list, or dict, of all instances ever created.
   These are obvious once found, but can be surprisingly hard to
   locate.  Of course the instances never go away then until the
   class goes away.  Sometimes it's due to leftover debugging code
   someone forgot to delete again.

2. "Reference cycles".  Big topic <wink>.