[Zope-dev] Re: Preliminary findings: Zope 2.7 leakage caused by errors

[Tres Seaver]

Jean-Francois.Doyon at CCRS.NRCan.gc.ca wrote:
> Gents,
> 
> In following to the discussions regarding the leakage zope.org and my site
> suffer from, I've done some testing, and here's what I've found.
> 
> First of all, the leak does seem to occur when errors occur.  Unlike what
> Chris suggested, I still suffer from this leak even when I completely
> remove standard_error_message.
> 
> How I tested:
> 
> I use JMeter, and a 99% identical box, dedicated for testing:
> 
> - RH 7.3
> - Python 2.3.3 compiled with ./configure and no options
> - Zope 2.7 + a variety of products, both custom and well known (CMF,
> Localizer, etc ...)
> 
> I look at my error log on my main site to see the types of errors coming up
> and note the URL/parameters.
> Such errors include KeyErrors, a RecursionDepth error, a
> Method-something-or-other error, and so on.
> 
> I setup JMeter with multiple concurrent threads and run it.  Within minutes
> I can see refcounts growing abnormally on the test server.
> 
> I then shut everything down and minimize the caches.  The following are left
> around that shouldn't be:
> 
> Class  May 19, 2004 1:52 pm  May 19, 2004 2:26 pm  Delta  
> Acquisition.ImplicitAcquirerWrapper  1025  1309  +284  
> ZServer.HTTPResponse.ZServerHTTPResponse  150  247  +97  
> ZPublisher.BaseRequest.RequestContainer  146  240  +94  
> ZPublisher.HTTPRequest.HTTPRequest  172  262  +90
> 
> Right now I've focused on finding out why the Requests are still around.
> 
> Using the gc module, I've found that all leaked requests are being held by a
> dictionary.  If I look at said dictionaries, this is what they might look
> like:
> 
> {'REQUEST': <HTTPRequest,
> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
> _site/francais/mapsatoz>},
>  {'REQUEST': <HTTPRequest,
> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
> _site/francais/mapsatoz>},
>  {'REQUEST': <HTTPRequest,
> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
> _site/english/search/political>},
>  {'REQUEST': <HTTPRequest,
> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
> _site/english/search/political>},
>  {'REQUEST': <HTTPRequest,
> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
> _site/english/search/political>},
>  {'REQUEST': <HTTPRequest,
> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
> _site/english/search/political>},
>  {'REQUEST': <HTTPRequest,
> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
> _site/english/search/political>},
>  {3075: <HTTPRequest,
> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
> _site/english/maps/peopleandsociety/age/youth/HEAD>,
>   4100: <HTTPRequest,
> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
> _site/debug>,
>   5125: <HTTPRequest,
> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
> _site/debug>,
>   6150: <HTTPRequest,
> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
> _site/english/search/political>,
>   7175: <HTTPRequest,
> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/vh_
> site/english/maps/archives/5thedition/other/referencemaps>},
>  {'REQUEST': <HTTPRequest,
> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
> _site/debug>},
>  {'REQUEST': <HTTPRequest,
> URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
> _site/misc_/WFSAdapter/sqlmethod.gif>}

I know that the Localizer product does (or used to do) some kind of 
magic to make the REQUEST available to code as a "global".  Can you 
reproduce this data without that product present?

> 
> NOTE: I had to hack the __repr__ of HTTPRequest not to use self['URL'] since
> it seems that attribute is unavailable in a closed request.  I use PATH_INFO
> instead.
> 
> Most of the time I get the dictionary with the 'REQUEST' key.  Those
> dictionaries with int's as keys are ones generated by the ExternalMethod I
> use to run the code to debug ... So even errors raised while trying to debug
> this leaks a dictionary that holds on to the request(s).
> 
> Said dictionary does not seem to be referred to by anything, at least not
> anything the gc module can find (gc.get_referrers()).  One possible
> situation is that said dictionary is actually part of the HTTPRequest, and
> we have a cyclic reference problem.  I don't think so however, I've looked
> at the HTTPRequest instances, and they have nothing in them other than
> strings, lists, basic dict's and an HTTPResponse instance.  I'll take
> another look to be sure.

Again, I would be looking at the Localizer's module-level globals for 
this one.

> Next step is too look at the Wrappers and see what they're all about and
> what's holding them there.

The wrappers are almost certainly there because the request is holding 
on to objects, which hold on to others, etc.

> I'm hoping this information might give a clue to someone who knows Zope
> error handling better than I as to what might be the problem here ... Also I
> understand zope.org suffers from this? :)
> 
> Based on this information also, I have no reason to believe this is caused
> by product code ... I have to guess it's a Zope bug?
> 
> In some ways this is motivating me to reduce the number of errors that could
> occur, but because ALL errors seems to cause the problem, I ultimately have
> no control over it and under certain circumstances I suffer heavily from
> this.  404's alone probably cause this (I'm plannign to test that
> explicitely as well).
> 
> Thoughts, suggestions, comments welcome ...

I am CC'ing Juan-David, in case he may be able to offer any insights 
from the perspective of the Localizer product.

Tres.
-- 
===============================================================
Tres Seaver                                tseaver at zope.com
Zope Corporation      "Zope Dealers"       http://www.zope.com