RE: Leaking HTTP requests (was: RE: [Zope] LeakingAcquisition.Imp licitAcquirerWrapper)
Chris, No problem, thanks for the effort! I've been told to devote all of my resources to this myself, so that's what I'm doing ... Unfortunately, I'm pushing my skillset as far as deep debugging of Zope/Python, so it means a learning curve for me, which takes time ... The one great thing I'd like to be able to do is "attach" myself to the instances left around so I can inspect them ... Is that at all doable? I have the monitor port open, maybe I can go in that way ... There's also gdb, but I've never used it at all :( Oh and there's pdb ... can I run Zope within the debugger somehow? Maybe from within the monitor? Anyways, I guess I'll find out the hard way! Cheers, J.F. -----Original Message----- From: Chris McDonough [mailto:chrism@plope.com] Sent: May 14, 2004 1:47 PM To: Jean-Francois.Doyon@CCRS.NRCan.gc.ca Cc: brian@zope.com; stefan@epy.co.at; zope@zope.org Subject: RE: Leaking HTTP requests (was: RE: [Zope] LeakingAcquisition.Imp licitAcquirerWrapper) FWIW, I've been trying to fix the leak Tres found (which appears to be unrelated to your leak), but I'm pretty close to giving up after ~5 hours, as I'm pretty much out of time. - C On Fri, 2004-05-14 at 13:23, Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
G'Day,
Preliminary information seems to suggest the patch isn't helping either.
So right now I've both removed the standard_error_message AND applied the LeakBGone monkey patch.
I'll leave my production site in this state through the week-end and see what the numbers look like on monday.
In the mean time, I'll try to dig deeper ...
Thanks, J.F.
-----Original Message----- From: Chris McDonough [mailto:chrism@plope.com] Sent: May 13, 2004 5:50 PM To: Brian Lloyd Cc: Stefan H. Holek; Jean-Francois.Doyon@CCRS.NRCan.gc.ca; zope@zope.org Subject: RE: Leaking HTTP requests (was: RE: [Zope] LeakingAcquisition.ImplicitAcquirerWrapper)
Tres and me were playing around with this a little and the easiest way to provoke a REQUEST leak is to do an anonymous request to a resource inside a subfolder where the subfolder has all permission acquisition turned off (and only "authenticated" granted view access). It will turn around and try to run "standard_error_message", which will fail due to security (and leak in the process). The code responsible for this is SimpleItem.raise_standardErrorMessage, which does some funky passing around/munging of traceback objects. However, "raise_standardErrorMessage" never gets called if "standard_error_message" doesn't exist in the acquisition path. If it never gets called, the leak does not occur.
The simplest way to rid yourself of the leak temporarily is to remove "standard_error_message".
On Thu, 2004-05-13 at 17:41, Brian Lloyd wrote:
Zope.org doesn't use Localizer (or Archetypes - another thing that has come up in this thread).
In our experience, this sort of thing has almost always turned out to be a wrapped object ending up in the REQUEST or a wrapped object holding onto a request for some reason.
I've attached a quick product you can drop in to test that theory on your own instance (just make a directory Products/LeakBGone and drop this __init__.py into it and restart).
If it fixes the leak, we can extend it to do some logging and try to figure out the root cause. I'll try it out on zope.org as soon as I'm able.
Brian Lloyd brian@zope.com V.P. Engineering 540.361.1716 Zope Corporation http://www.zope.com
-----Original Message----- From: zope-bounces+brian=zope.com@zope.org [mailto:zope-bounces+brian=zope.com@zope.org]On Behalf Of Stefan H. Holek Sent: Thursday, May 13, 2004 3:14 PM To: Brian Lloyd Cc: Jean-Francois.Doyon@CCRS.NRCan.gc.ca; zope@zope.org Subject: Re: Leaking HTTP requests (was: RE: [Zope] LeakingAcquisition.ImplicitAcquirerWrapper)
Does zope.org use Localizer or some type of "global request" patch?
Stefan
On Donnerstag, Mai 13, 2004, at 21:42 Europe/Vienna, Brian Lloyd wrote:
FWIW - zope.org is suffering hugely from this as well, so I'm following this thread eagerly ;) -- The time has come to start talking about whether the emperor is as well dressed as we are supposed to think he is. /Pete McBreen/
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
______________________________________________________________________ _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
The monitor port might be the best way for this particular bug, as you are "looking into" the running process, so you can inspect refcounts of various things as the state of the system changes. That hasn't proved very useful for me so far however. On Fri, 2004-05-14 at 14:10, Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
Chris,
No problem, thanks for the effort! I've been told to devote all of my resources to this myself, so that's what I'm doing ...
Unfortunately, I'm pushing my skillset as far as deep debugging of Zope/Python, so it means a learning curve for me, which takes time ...
The one great thing I'd like to be able to do is "attach" myself to the instances left around so I can inspect them ... Is that at all doable? I have the monitor port open, maybe I can go in that way ...
There's also gdb, but I've never used it at all :( Oh and there's pdb ... can I run Zope within the debugger somehow? Maybe from within the monitor?
Anyways, I guess I'll find out the hard way!
Cheers, J.F.
-----Original Message----- From: Chris McDonough [mailto:chrism@plope.com] Sent: May 14, 2004 1:47 PM To: Jean-Francois.Doyon@CCRS.NRCan.gc.ca Cc: brian@zope.com; stefan@epy.co.at; zope@zope.org Subject: RE: Leaking HTTP requests (was: RE: [Zope] LeakingAcquisition.Imp licitAcquirerWrapper)
FWIW, I've been trying to fix the leak Tres found (which appears to be unrelated to your leak), but I'm pretty close to giving up after ~5 hours, as I'm pretty much out of time.
- C
On Fri, 2004-05-14 at 13:23, Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote:
G'Day,
Preliminary information seems to suggest the patch isn't helping either.
So right now I've both removed the standard_error_message AND applied the LeakBGone monkey patch.
I'll leave my production site in this state through the week-end and see what the numbers look like on monday.
In the mean time, I'll try to dig deeper ...
Thanks, J.F.
-----Original Message----- From: Chris McDonough [mailto:chrism@plope.com] Sent: May 13, 2004 5:50 PM To: Brian Lloyd Cc: Stefan H. Holek; Jean-Francois.Doyon@CCRS.NRCan.gc.ca; zope@zope.org Subject: RE: Leaking HTTP requests (was: RE: [Zope] LeakingAcquisition.ImplicitAcquirerWrapper)
Tres and me were playing around with this a little and the easiest way to provoke a REQUEST leak is to do an anonymous request to a resource inside a subfolder where the subfolder has all permission acquisition turned off (and only "authenticated" granted view access). It will turn around and try to run "standard_error_message", which will fail due to security (and leak in the process). The code responsible for this is SimpleItem.raise_standardErrorMessage, which does some funky passing around/munging of traceback objects. However, "raise_standardErrorMessage" never gets called if "standard_error_message" doesn't exist in the acquisition path. If it never gets called, the leak does not occur.
The simplest way to rid yourself of the leak temporarily is to remove "standard_error_message".
On Thu, 2004-05-13 at 17:41, Brian Lloyd wrote:
Zope.org doesn't use Localizer (or Archetypes - another thing that has come up in this thread).
In our experience, this sort of thing has almost always turned out to be a wrapped object ending up in the REQUEST or a wrapped object holding onto a request for some reason.
I've attached a quick product you can drop in to test that theory on your own instance (just make a directory Products/LeakBGone and drop this __init__.py into it and restart).
If it fixes the leak, we can extend it to do some logging and try to figure out the root cause. I'll try it out on zope.org as soon as I'm able.
Brian Lloyd brian@zope.com V.P. Engineering 540.361.1716 Zope Corporation http://www.zope.com
-----Original Message----- From: zope-bounces+brian=zope.com@zope.org [mailto:zope-bounces+brian=zope.com@zope.org]On Behalf Of Stefan H. Holek Sent: Thursday, May 13, 2004 3:14 PM To: Brian Lloyd Cc: Jean-Francois.Doyon@CCRS.NRCan.gc.ca; zope@zope.org Subject: Re: Leaking HTTP requests (was: RE: [Zope] LeakingAcquisition.ImplicitAcquirerWrapper)
Does zope.org use Localizer or some type of "global request" patch?
Stefan
On Donnerstag, Mai 13, 2004, at 21:42 Europe/Vienna, Brian Lloyd wrote:
FWIW - zope.org is suffering hugely from this as well, so I'm following this thread eagerly ;) -- The time has come to start talking about whether the emperor is as well dressed as we are supposed to think he is. /Pete McBreen/
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
______________________________________________________________________ _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Jean-Francois.Doyon@CCRS.NRCan.gc.ca wrote at 2004-5-14 14:10 -0400:
... The one great thing I'd like to be able to do is "attach" myself to the instances left around so I can inspect them ... Is that at all doable? I have the monitor port open, maybe I can go in that way ...
I fear it is at least very difficult... Normal Python builds just maintain the numbers of objects. You do not get a handle to them. You can build Python in a way that all objects are linked together. With such a build, you can locate and analyse the objects. However, I doubt that this will be the most efficient way to solve the problem. I am still convinced that the following approach gives the fasted success: * let your system run for a while * minimize the ZODB caches * look at the reference counts in "Control_Panel --> Debug management" * identify the class with should be rarely instantiated but nevertheless shows a high reference count. * test the assumption that this class is responsible by mass operations on the class (e.g. with the "ab" utilitiy (part of Apache)).
There's also gdb, but I've never used it at all :(
It will not help you.
Oh and there's pdb
It will only help you, once you know the primary class responsible for the cycle (to pin the cycle building down to a specific location). -- Dieter
I have the monitor port open, maybe I can go in that way ...
Just in case you haven't come across this very useful howto yet: http://www.zope.org/Members/klm/ZopeDebugging/ConversingWithZope -- Jean Jordaan http://www.upfrontsystems.co.za
participants (4)
-
Chris McDonough -
Dieter Maurer -
Jean Jordaan -
Jean-Francois.Doyon＠CCRS.NRCan.gc.ca