[Zope] My Zope is leaking memory ...

[Charlie Reiman]

> -----Original Message-----
> From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of
> Jean-Francois.Doyon@CCRS.NRCan.gc.ca
> Sent: Monday, November 25, 2002 10:57 AM
> To: zope@zope.org
> Subject: [Zope] My Zope is leaking memory ...
>
>
> Hello,
>
> Yup, that's right, my Zope is leaking memory, and so far, I've had no luck
> figuring out where it might come from. I run a Zope 2.5.0 on RedHat 7.3.
>
> I started by looking into the Python/C type modules, which I
> fugred are most
> likely to be causing this.  I upgraded Psycopg to the latest version, but
> it's still occuring.  I also use the PIL, but it is seldom used, and could
> not be the cause of the kind of leakage I'm seeing. Then there's
> a much more
> obscure thing, which is a SWIG interface to PROJ.4 (A geographical
> projection library) ... I had the author look at it, and no leaks were
> detected in the C  library, or using the SWIG interface (Using Valgrind I
> beleive).  My Z SQL Methods do NOT cache results (Maximum time set 0).
>
> I'm using a custom built (with default configuration) Python 2.1.3 ...
>
> The symptom: I run Zope with -t 6 due to the site being very busy. There's
> 1GB of RAM. When I first start Zope, it the memory of the process running
> running the threads is around 10% (as shown in "ps"). Here, now
> after 4 days
> 19 hours up:
>
> root       943  0.0  0.1  5024 1456 ?        S    Nov20   0:00
> /usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 -t 6
> -w 8080 -p - -f - -D
> nobody     946  0.8 40.4 470040 417420 ?     S    Nov20  60:26
> /usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 -t 6
> -w 8080 -p - -f - -D
> nobody     950  0.0 40.4 470040 417420 ?     S    Nov20   0:03
> /usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 -t 6
> -w 8080 -p - -f - -D
> nobody     951  3.8 40.4 470040 417420 ?     S    Nov20 266:02
> /usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 -t 6
> -w 8080 -p - -f - -D
> nobody     952  3.8 40.4 470040 417420 ?     S    Nov20 264:41
> /usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 -t 6
> -w 8080 -p - -f - -D
> nobody     953  3.7 40.4 470040 417420 ?     S    Nov20 262:58
> /usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 -t 6
> -w 8080 -p - -f - -D
> nobody     954  3.8 40.4 470040 417420 ?     S    Nov20 267:13
> /usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 -t 6
> -w 8080 -p - -f - -D
> nobody     955  4.3 40.4 470040 417420 ?     S    Nov20 299:51
> /usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 -t 6
> -w 8080 -p - -f - -D
> nobody     956  3.8 40.4 470040 417420 ?     S    Nov20 265:31
> /usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 -t 6
> -w 8080 -p - -f - -D
>
> Memory usage is at 40.4 percent ...
>
> In another 4 days, it will have reached almost 60%, and I will restart it
> ... Basically, after a week, it gets to be too much.
>
> As far as ZODB Caching, here are my numbers right now:
>
> Total objects in DB: 44786
> Total objects in all caches: 30290 (This is peak actually, it
> varies between
> 26K and 30K depending on time of day).
> Target size: 5000
> Max time: 300
>
> When packed my Data.fs is under 100Mb ...
>
> The memory keeps growing regardless of the caching going on with the
> database, so that's not the issue.
>
> The site takes millions of this ever week ...
>
> So now I'm running out of ideas to try and track this down, I was
> hoping it
> would be the PROJ interface/library ... But apparently not (Even then the
> leakage would probably be much smaller than what we're seeing here).
>
> Does anyone know how I might go about tracking this down further?
> I know of
> the Medusa monitor thing, but I wouldn't know the first thing on how to
> start using that to track memory usage, if it is even possible.
>
> Right now I'm wondering about the PostgreSQL usage ... It is the
> only thing
> that has intensive enough usage to leak this much memory I think ...
>
> Any ideas, comments, insights, tips would be MOST appreciated!
>
> Thanks in advance,
>
> Jean-François Doyon
> Internet Service Development and Systems Support
> GeoAccess Division
> Canada Center for Remote Sensing
> Natural Resources Canada
> http://atlas.gc.ca
> Phone: (613) 992-4902
> Fax: (613) 947-2410

Look under Control Panel:Debug. There is a display of current object
allocations. You can refresh to see deltas.

There a good chance you are exercising a known Zope bug:

http://collector.zope.org/Zope/421

This is fixed in 2.6 and there a patch for 2.5.1 on the collector issue.