Investigating a reference leak...
Dear Zopistas I've been trying to track down what's either a memory or a reference leak in a Zope 2.7.3 (Python 2.3.4) system. The symptoms are that after several days of normal running on the agressively proxy-cached production site or a few hours on the development site under simulated load, the number of OFS.Image.Image references reaches very high levels, such as 17761. In the cache at that time are just 4345 images. There are only 1332 images on the whole site, so assuming (as I understand it) that each of the 5 Zope thread has its own cache, there should be a maximum of (1332 * threads) = 6660 Images. The site never creates new Images, it just loads existing ones. In a bid to see where these references are being created, I installed LeakFinder, but that doesn't help me because Images are persistent objects, and thus their __init__ methods are not called (LeakFinder appears to patch just the __init__ and __del__ methods for tracking). Thus I've been on a quest to find a place to stick a patch that will show me, for a persistent object, from where a reference to it is being created. I've tried __setstate__, and overridden it for OFS.Image.Image to print a traceback to stdout. Then I run the development Zope site with bin/runzope and watch. This shows me Images being created in the expected places, but not in quantities that would explain those huge refcounts. So; to questions: 1) Is there a better place than __setcache__ to identify where references to Images are being created? I don't fully understand the way in which persistent objects are actually created and populated. 2) Our site is very dependent on ExternalMethods (Zope provides the visual layout, business logic is in ExternalMethods, all from one module). In some of those methods, the code loads Images in order to look up information about them - this is because we have some particular standards for things like Image tags which the built-in tag() method can't support, so we have to build our own. Thus there's code like: img = getattr(self.Images,'image.gif') #all images live in /Images, got by acquisition from self buildTag(img.title,img.width,img.height) #use image attributes to build fuller tag del img #avoid leaving references around (this is paranoid!) Is there any reason that this is bad practise? I see __setstate__ for images being called from this code and then again shortly afterwards when the Image is served up to the browser. I'd be very grateful for any help on this! Regards to all Ben Ben Last Technical Director SleepyDog
Ben Last (Zope) wrote:
1) Is there a better place than __setcache__ to identify where references to Images are being created? I don't fully understand the way in which persistent objects are actually created and populated.
Not sure you mean __setcache__, but this is the kind of question the good folks on zodb-dev@zope.org maybe be able to help with...
2) Our site is very dependent on ExternalMethods (Zope provides the visual layout, business logic is in ExternalMethods, all from one module). In some of those methods, the code loads Images in order to look up information about them - this is because we have some particular standards for things like Image tags which the built-in tag() method can't support,
you sure about that? tag() is pretty flexible...
img = getattr(self.Images,'image.gif') #all images live in /Images, got by acquisition from self buildTag(img.title,img.width,img.height) #use image attributes to build fuller tag del img #avoid leaving references around (this is paranoid!)
what does buildTag look like?
I'd be very grateful for any help on this!
Looks like something somewhere is hanging on to references. You could try the old fashioned way: keep commenting out bits of your code in small chunks until the leak goes away, when it does, analyse the bit you've just take out to death ;-) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
From: Chris Withers [mailto:chris@simplistix.co.uk]
Ben Last (Zope) wrote:
1) Is there a better place than __setcache__... Not sure you mean __setcache__, but this is the kind of question the good folks on zodb-dev@zope.org maybe be able to help with... Ouch - I mean "__setstate__" . I believe the appropriate phrase is "D'oh!" I'll join zope-dev and see what they have to say.
some particular standards for things like Image tags which the built-in tag() method can't support, you sure about that? tag() is pretty flexible... Yeah, we're sure. Our tag handling does stuff like override a GIF with a SWF for the flash version, forces the images to be served via a dedicated images hostname, etc. These are external requirements set on us by others...
img = getattr(self.Images,'image.gif') #all images live in /Images, got by acquisition from self buildTag(img.title,img.width,img.height) #use image attributes to build fuller tag del img #avoid leaving references around (this is paranoid!) what does buildTag look like? It's just code that builds up a full <img...> tag using the passed in parameters. Something I did notice is that, according to the traceback, __setstate__ is invoked on a line that reads: if img: ...which seems odd to me. Is there any reason that tracebacks through ExternalMethods would dump the wrong line numbers/lines?
Looks like something somewhere is hanging on to references. You could try the old fashioned way: keep commenting out bits of your code in small chunks until the leak goes away, when it does, analyse the bit you've just take out to death ;-) Yeah... am sort of doing that by trying single URLs and watching the refcounts. However, they don't seem to follow any strict relationship between page templates and increases...
Thanks anyway :) Ben
participants (2)
-
Ben Last (Zope) -
Chris Withers