[Zope] My Zope is leaking memory ...

Mon, 25 Nov 2002 13:56:58 -0500

Hello,

Yup, that's right, my Zope is leaking memory, and so far, I've had no =
luck
figuring out where it might come from. I run a Zope 2.5.0 on RedHat =
7.3.

I started by looking into the Python/C type modules, which I fugred are =
most
likely to be causing this.  I upgraded Psycopg to the latest version, =
but
it's still occuring.  I also use the PIL, but it is seldom used, and =
could
not be the cause of the kind of leakage I'm seeing. Then there's a much =
more
obscure thing, which is a SWIG interface to PROJ.4 (A geographical
projection library) ... I had the author look at it, and no leaks were
detected in the C  library, or using the SWIG interface (Using Valgrind =
I
beleive).  My Z SQL Methods do NOT cache results (Maximum time set 0).

I'm using a custom built (with default configuration) Python 2.1.3 ...

The symptom: I run Zope with -t 6 due to the site being very busy. =
There's
1GB of RAM. When I first start Zope, it the memory of the process =
running
running the threads is around 10% (as shown in "ps"). Here, now after 4 =
days
19 hours up:

root       943  0.0  0.1  5024 1456 ?        S    Nov20   0:00
/usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 =
-t 6
-w 8080 -p - -f - -D
nobody     946  0.8 40.4 470040 417420 ?     S    Nov20  60:26
/usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 =
-t 6
-w 8080 -p - -f - -D
nobody     950  0.0 40.4 470040 417420 ?     S    Nov20   0:03
/usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 =
-t 6
-w 8080 -p - -f - -D
nobody     951  3.8 40.4 470040 417420 ?     S    Nov20 266:02
/usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 =
-t 6
-w 8080 -p - -f - -D
nobody     952  3.8 40.4 470040 417420 ?     S    Nov20 264:41
/usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 =
-t 6
-w 8080 -p - -f - -D
nobody     953  3.7 40.4 470040 417420 ?     S    Nov20 262:58
/usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 =
-t 6
-w 8080 -p - -f - -D
nobody     954  3.8 40.4 470040 417420 ?     S    Nov20 267:13
/usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 =
-t 6
-w 8080 -p - -f - -D
nobody     955  4.3 40.4 470040 417420 ?     S    Nov20 299:51
/usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 =
-t 6
-w 8080 -p - -f - -D
nobody     956  3.8 40.4 470040 417420 ?     S    Nov20 265:31
/usr/local/bin/python2.1 /usr/local/Zope-2.5.0/z2.py -a 132.156.10.87 =
-t 6
-w 8080 -p - -f - -D

Memory usage is at 40.4 percent ...

In another 4 days, it will have reached almost 60%, and I will restart =
it
... Basically, after a week, it gets to be too much.

As far as ZODB Caching, here are my numbers right now:

Total objects in DB: 44786
Total objects in all caches: 30290 (This is peak actually, it varies =
between
26K and 30K depending on time of day).
Target size: 5000
Max time: 300

When packed my Data.fs is under 100Mb ...

The memory keeps growing regardless of the caching going on with the
database, so that's not the issue.

The site takes millions of this ever week ...

So now I'm running out of ideas to try and track this down, I was =
hoping it
would be the PROJ interface/library ... But apparently not (Even then =
the
leakage would probably be much smaller than what we're seeing here).

Does anyone know how I might go about tracking this down further? I =
know of
the Medusa monitor thing, but I wouldn't know the first thing on how to
start using that to track memory usage, if it is even possible.

Right now I'm wondering about the PostgreSQL usage ... It is the only =
thing
that has intensive enough usage to leak this much memory I think ...

Any ideas, comments, insights, tips would be MOST appreciated!

Thanks in advance,

Jean-Fran=E7ois Doyon
Internet Service Development and Systems Support
GeoAccess Division
Canada Center for Remote Sensing
Natural Resources Canada
http://atlas.gc.ca
Phone: (613) 992-4902
Fax: (613) 947-2410