[Zope-dev] Preliminary findings: Zope 2.7 leakage caused by errors

Wed May 19 14:44:18 EDT 2004

Gents,

In following to the discussions regarding the leakage zope.org and my site
suffer from, I've done some testing, and here's what I've found.

First of all, the leak does seem to occur when errors occur.  Unlike what
Chris suggested, I still suffer from this leak even when I completely
remove standard_error_message.

How I tested:

I use JMeter, and a 99% identical box, dedicated for testing:

- RH 7.3
- Python 2.3.3 compiled with ./configure and no options
- Zope 2.7 + a variety of products, both custom and well known (CMF,
Localizer, etc ...)

I look at my error log on my main site to see the types of errors coming up
and note the URL/parameters.
Such errors include KeyErrors, a RecursionDepth error, a
Method-something-or-other error, and so on.

I setup JMeter with multiple concurrent threads and run it.  Within minutes
I can see refcounts growing abnormally on the test server.

I then shut everything down and minimize the caches.  The following are left
around that shouldn't be:

Class  May 19, 2004 1:52 pm  May 19, 2004 2:26 pm  Delta  
Acquisition.ImplicitAcquirerWrapper  1025  1309  +284  
ZServer.HTTPResponse.ZServerHTTPResponse  150  247  +97  
ZPublisher.BaseRequest.RequestContainer  146  240  +94  
ZPublisher.HTTPRequest.HTTPRequest  172  262  +90

Right now I've focused on finding out why the Requests are still around.

Using the gc module, I've found that all leaked requests are being held by a
dictionary.  If I look at said dictionaries, this is what they might look
like:

{'REQUEST': <HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
_site/francais/mapsatoz>},
 {'REQUEST': <HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
_site/francais/mapsatoz>},
 {'REQUEST': <HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
_site/english/search/political>},
 {'REQUEST': <HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
_site/english/search/political>},
 {'REQUEST': <HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
_site/english/search/political>},
 {'REQUEST': <HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
_site/english/search/political>},
 {'REQUEST': <HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
_site/english/search/political>},
 {3075: <HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
_site/english/maps/peopleandsociety/age/youth/HEAD>,
  4100: <HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
_site/debug>,
  5125: <HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
_site/debug>,
  6150: <HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
_site/english/search/political>,
  7175: <HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/vh_
site/english/maps/archives/5thedition/other/referencemaps>},
 {'REQUEST': <HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
_site/debug>},
 {'REQUEST': <HTTPRequest,
URL=/VirtualHostBase/http/atlastest2.ccrs.nrcan.gc.ca:80/VirtualHostRoot/_vh
_site/misc_/WFSAdapter/sqlmethod.gif>}

NOTE: I had to hack the __repr__ of HTTPRequest not to use self['URL'] since
it seems that attribute is unavailable in a closed request.  I use PATH_INFO
instead.

Most of the time I get the dictionary with the 'REQUEST' key.  Those
dictionaries with int's as keys are ones generated by the ExternalMethod I
use to run the code to debug ... So even errors raised while trying to debug
this leaks a dictionary that holds on to the request(s).

Said dictionary does not seem to be referred to by anything, at least not
anything the gc module can find (gc.get_referrers()).  One possible
situation is that said dictionary is actually part of the HTTPRequest, and
we have a cyclic reference problem.  I don't think so however, I've looked
at the HTTPRequest instances, and they have nothing in them other than
strings, lists, basic dict's and an HTTPResponse instance.  I'll take
another look to be sure.

Next step is too look at the Wrappers and see what they're all about and
what's holding them there.

I'm hoping this information might give a clue to someone who knows Zope
error handling better than I as to what might be the problem here ... Also I
understand zope.org suffers from this? :)

Based on this information also, I have no reason to believe this is caused
by product code ... I have to guess it's a Zope bug?

In some ways this is motivating me to reduce the number of errors that could
occur, but because ALL errors seems to cause the problem, I ultimately have
no control over it and under certain circumstances I suffer heavily from
this.  404's alone probably cause this (I'm plannign to test that
explicitely as well).

Thoughts, suggestions, comments welcome ...

Jean-François Doyon
Internet Service Development and Systems Support / Spécialiste de
dèveloppements internet et soutien technique
Canada Centre for Remote Sensing / Centre Canadien de télédétection
Natural Resources Canada / Ressources naturelles Canada
http://atlas.gc.ca
Tel./Tél. : (613) 992-4902
Fax: (613) 947-2410