[Seb Bacon]
... I know from the refcounts in Zope that items of class Foo are definitely leaking, yet when I do a sys.getobjects(0, Foo) I get nothing back.
If Foo is an old-style class, then every instance of Foo has type InstanceType (and so does every instance of every other old-style class):
class Foo: pass
type(Foo()) <type 'instance'> import types types.InstanceType <type 'instance'> types.InstanceType is type(Foo()) True
getobjects() filters on type, so nothing will ever match Foo as a type. If you can change Foo to a new-style class (most easily by inheriting from object, in a recent-enough Python), life gets easier:
class Foo(object): pass
type(Foo()) <class '__main__.Foo'>
Then getobjects() can filter on Foo as a type. Classes and types before Python 2.2 are distinct concepts (barring Zope ExtensionClass complications).
Navigating all 100000-ish references one by one seems a bit daunting.
Na, with list comprehension syntax (for brevity -- you can do the same with a for-loop, of course), foos = [x for x in sys.getobjects(0) if isinstance(x, types.InstanceType) and x.__class__ is Foo] will extract just the Foo instances (if Foo is an old-style class).
So I'm not sure where to go from here.
Debugging memory leaks can be hard, in any language. Another place to look for ideas is in the top-level test.py from a current Zope HEAD checkout (or 2.7 branch, or Zope3). If you're able to run your code in a loop, the TrackRefs class in test.py automates some measure of identifying what (if anything) is leaking. We've changed many internal ZODB and ZEO classes to new-style classes primarily just so this test.py's -r option is more useful in identifying the source of leaks. Some yielded easily to analysis, others slobbered on for part-time weeks. There are two common culprits: 1. Some class keeps a list, or dict, of all instances ever created. These are obvious once found, but can be surprisingly hard to locate. Of course the instances never go away then until the class goes away. Sometimes it's due to leftover debugging code someone forgot to delete again. 2. "Reference cycles". Big topic <wink>.