[ZODB-Dev] Investigating a Zope reference leak... trackingobjectcreation

[Tim Peters]

[Ben Last]
> I think I expressed myself badly...

Oh, I don't know about that.  It's at least equally likely that I'm
misunderstanding you -- you're (and naturally so) phrasing things in terms
of how they look from Zope, while I'm looking at it from the lowest level of
ZODB's C code <0.5 wink>.

> I do understand how references work in Python,

Good!

> I guess what I'm concerned about is why the Image class has so
> many references (which may not be a problem)

I can't guess.  Where are you getting your idea about how many references
there are to the Image class?  And if you minimize the caches (I think the
Zope interface calls that "flushing" caches), do these counts go down?

> and (more of a concern) try to see if those references relate to
> actual instances.

And you can't figure that out from looking at the "extreme cache detail"
page (note:  not the same as "cache detail"!)?  For each object in cache,
the "extreme cache detail" page lists things like its oid, class, and
whether it's a ghost.  That should tell you how many persisted instances of
Image exist (you'll have to count them, one by one -- e.g., paste the page
into a file and use egrep+wc).
 
> I used the term "reference leak" to distinguish this from a simple
> memory leak.

Yup.

> Very little actual referencing of Images happens in our code (a=b, etc),
> so I'm still concerned as to why the reference count always increases.

Ah, that's new information.  You gave one pair of static numbers at first
(17761 references, 4345 Images) but no indication of the growth rate (if
any).  I'm not sure how you obtained either number.  However you did that,
do the # of references continue to increase while the # of Images remains
steady?

> All the references that are created (I've checked the damn things every
> one) are in either:
> a) local variables which *presumably* go out of scope when the external
> method they're in is exited

Have you tried a controlled experiment, doing nothing with the test site
except invoking such a method repeatedly, and noting the effect on the
counts you're looking at?

> b) temporary objects that are returned from ExternalMethods to
> PageTemplates so that they can be used in ZPT expressions

Ditto.

> As such, I think I'd restate the question as follows: How do I track the
> creation (and destruction) of *instances* of a given Persistent class
> (such as OFS.Image.Image)?

Sorry, same answer:  there isn't a way, and "destruction" is a fuzzy concept
for persistent objects.  For example, a populated persistent object may
become a ghost again, but that doesn't release any references to it:  the
Python object is still there, it simply loses (most of) its state.  In the
other direction, __setstate__ is called when an object is unghostified, but
a single instance can be ghostified and unghostified any number of times,
and neither operation changes its reference count.

> If I can do that, I can at least see how many instances are being created
> for every actual image to see if there's duplication.

Across all the caches, each copy of a given persistent object has the same
oid.  The "cache extreme detail" page shows which oids are in which caches.
It's not possible that more than one copy of a given persistent object can
appear in a single cache (the cache essentially maps an oid to an in-memory
Python object, and it's impossible for a single oid to map to more than one
in-memory object in a single cach).