RE: Leaking HTTP requests (was: RE: [Zope] Leaking Acquisition.Im plic itAcquirerWrapper)
Dieter, Argh, I don't even see the problem in your example :( (I'm having a bad day!) I'll leave the iste running throught the week-end and we'll see what shows up. "high" refcounts is somewhat arbitrary I guess ... When you say high you mean high and stable or high and growing? Right now, the only suspect class as far as refcounts (Other than the HTTP requests) is the DateTime one. I have some content ones that look high, but then they're highly used (CMFImage for example). Apart from using aq_explicit in a few places, I do also use __of__ in one case ... To do things like: def getAbstract(self): try: pt = ZopePageTemplate( '', self.abstract, 'text/html').__of__(self) cookedbody = pt() return cookedbody except: return '' Could such use cause problems? I don't see why ... It's quite straightforward. That's about as fancy as anything I do gets I'm afraid, so I'm still having a hard time imagining anything I wrote messing with Wrappers ... I make it a point to avoid them! Oh wait a minute ... How about this? pt = ZopePageTemplate( '', self.aq_explicit.intropart1, 'text/html' ).__of__(self) Could THAT be bad? I'l look into it, it'll be easy to detect. TGIF, I'm loosing morale here ... Thanks for the help! The hunt continues ... J.F. -----Original Message----- From: Dieter Maurer [mailto:dieter@handshake.de] Sent: May 14, 2004 2:28 PM To: Jean-Francois.Doyon@CCRS.NRCan.gc.ca Cc: stefan@epy.co.at; zope@zope.org Subject: Re: Leaking HTTP requests (was: RE: [Zope] Leaking Acquisition.Implic itAcquirerWrapper) Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote at 2004-5-13 15:31 -0400:
Looks like I'm actually leaking HTTP requests!
I fear I brought in the "leaking HTTP requests" story... Meanwhile, I understand that whenever you leak AcquisitionWrappers you are leaking HTTP requests, too. That's because at the bottom of the acquisition chain is a REQUEST (usually an HTTPRequest). While a cycle created by storing an acquisition wrapper into REQUEST may cause this leakage, it is by far not the only potential cause. Yesterday, I found a leak in my DOM wrapper code caused by code like this: class Element(Implicit,...) __children = None def getChildren(self): '''the elements children.''' if self.__children is not None: return self.__children children = self.__children \ = NodeList([c.__of__(self) for c in self._proxy.children]) This creates a cycle between acquisition wrappers. And leaks acquisition wrappers as well as Request objects.... The major clue to solve my problem has been to look for further high reference counts for classes that are supposed to have small reference counts. In my case, I found "DOM.Document", created at a single place. That allowed me to quickly verify that I indeed leaked "DOM.Document" references (and everything below) and after I few hours, I found the code above... -- Dieter
On Fri, 2004-05-14 at 17:23, Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
Dieter,
Argh, I don't even see the problem in your example :( (I'm having a bad day!)
I'll leave the iste running throught the week-end and we'll see what shows up. "high" refcounts is somewhat arbitrary I guess ... When you say high you mean high and stable or high and growing?
Right now, the only suspect class as far as refcounts (Other than the HTTP requests) is the DateTime one. I have some content ones that look high, but then they're highly used (CMFImage for example).
Every persistent Zope object holds a reference to a datetime (bobobase_modification_time), so if you leak any persistent objects, the datetime refcount will also increase. So this isn't unexpected if an acquisition wrapper leaks. - C
Chris McDonough wrote at 2004-5-14 17:43 -0400:
... Every persistent Zope object holds a reference to a datetime (bobobase_modification_time)
Are you sure? "bobobase_modification_time" is a method, not an attribute. I expect it to reformat "_p_serial" as a "DateTime" object (when called). "DateTime" objects usually come from the catalog. -- Dieter
On Sun, 2004-05-16 at 16:48, Dieter Maurer wrote:
Chris McDonough wrote at 2004-5-14 17:43 -0400:
... Every persistent Zope object holds a reference to a datetime (bobobase_modification_time)
Are you sure?
"bobobase_modification_time" is a method, not an attribute.
I expect it to reformat "_p_serial" as a "DateTime" object (when called).
Sorry, I meant "_p_mtime". - C
Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote at 2004-5-14 17:23 -0400:
... I'll leave the iste running throught the week-end and we'll see what shows up. "high" refcounts is somewhat arbitrary I guess ... When you say high you mean high and stable or high and growing?
In fact, I mean unexpectedly high. The relevant references counts are not necessarily the highest ones. For example, when you make catalog searches, it is completely normal that you see lots of "DateTime" objects. Nevertheless, you can be sure that they are not the cause of memory leaks. The same applies for acquisition wrappers. You must look for usually rare objects that suddenly show up in large numbers. It may help very much to clear the ZODB cache ("Control_Panel" --> "Database management" --> <databases> --> "Flush Cache" --> "Minimize"). This should delete all objects from memory that are "normal" artefacts of Zope processing. Apart from a few objects only the leaked ones should remain. Be warned, there was a bug in "cPickleCache.c" that led "Minimize" enter an infinite loop. I am not sure whether this bug is fixed in the current Zope 2.7 release.
Right now, the only suspect class as far as refcounts (Other than the HTTP requests) is the DateTime one. I have some content ones that look high, but then they're highly used (CMFImage for example).
"DateTime" are normal, "HTTP requests" is strange....
Apart from using aq_explicit in a few places, I do also use __of__ in one case ... To do things like:
def getAbstract(self): try: pt = ZopePageTemplate( '', self.abstract, 'text/html').__of__(self) cookedbody = pt()
Harmless. The problem in my code has been that I stored an acquisition wrapped object in my object.
... How about this?
pt = ZopePageTemplate( '', self.aq_explicit.intropart1, 'text/html' ).__of__(self)
Could THAT be bad?
Unlikely (unless you store it as attribute of "self"). -- Dieter
participants (3)
-
Chris McDonough -
Dieter Maurer -
Jean-Francois.Doyon＠CCRS.NRCan.gc.ca