[Zope] Zope 2.7.3 Memory Leaks

[Andy Yates]

> -----Original Message-----
> From: Chris McDonough [mailto:chrism at plope.com]
> Sent: Thursday, December 02, 2004 5:17 PM
> To: Andy Yates
> Cc: zope at zope.org
> Subject: Re: [Zope] Zope 2.7.3 Memory Leaks
> 
> On Thu, 2004-12-02 at 17:24, Andy Yates wrote:
> > Now it seems to leak memory (400+ Mb) to the point I have to restart
> > Zope every 3 days.
> >
> > I've read that memory leaks are almost always caused by the
programmer,
> > so I set up some very simple tests that did not involve existing
code.
> 
> Great idea.

Note that I'm not trying to find a problem with Zope.  My plan was to
start with something very simple and then add more and more of the
things we do on our production site and see when it starts leaking.
However, even the simplest tests seem to leak memory.

> > Next I created an 8Mb page template.  This caused the memory usage
to
> > climb much faster.  After the test the memory never goes back down.
> 
> Do these templates access the sessioning machinery?
>
No.  It is just a page template file with 8Mb of random text.
 
> > I installed LeakFinder and started running more tests.
> >
> > I used ab to get a python script that stored a 1k string in the
session
> > object.  The transient object container timeout was set to 1 minute.
> > This also caused Zope to consume memory.  LeakFinder said the
> > Products.Transience.TransientObject.TransientObject ref count grew
with
> > each request and never went down.  Everything else seemed to level
off.
> > The debug output of Transience.py showed that the "buckets" seemed
to be
> > getting deleted as expected, but the memory usage never goes down.
1
> > minute after the test stopped the transient object container showed
that
> > there were no more items in the container.  It seems like when Zope
> > deletes expired sessions but contents of the sessions are not
deleted.
> 
> This is actually probably normal.
> 
> TransientObject buckets hang around for some period of time before
they
> are "garbage collected" (deleted from their container).  A particular
> TransientObject isn't garbage collected immediately when it expires,
but
> gets gc'ed much later along with other TransientObjects that were
> created around the same time as a side effect of otherwise exercising
> the sessioning machinery.  

Thanks for the explaination of how the transient gc works.  That is
exactly what I found when I looked in the source code and I can see all
of this happening when I turn on the debug output.  Sessions are created
and placed in buckets.  The active sessions move to the current bucket
and expired session and old empty buckets get removed.  Then there is a
1 in 6 chance they will be gc'ed on each request.  I see all this
happening but when the buckets are gc'ed the memory usage according to
top does not go down.  

I ran the test again that puts a 1k string in the session objects.  I
used ab to run the python script 3000 times with a concurrency of 10.
Python's memory went from 22Mb to 54Mb.  Transient objects went from 0
to about 900 and stayed steady during the test.  After the test the
trans obs dropped back down to 0 as they expired.

I continued to make occasional Zope requests to plain pages to make sure
gc was getting tickled.  After 8+ hours of this the memory was never
released.  Python was still using 54Mb.  If I continue to create session
objects the 54Mb is not reused by python it just continues to grow.

> 
> Note that Python will usually not release memory back to the OS once
it
> has been allocated for its own use, so a large concrete amount of
memory
> consumed by Python may not even be an indicator of a leak!  See
> http://mail.python.org/pipermail/python-dev/2004-October/049480.html
for
> more info.  The only solid basis for determining leaks is refcounts,
and
> it's tricky to figure out whether growing refcounts are leaks or if
> they're just side effects of the ZODB cache holding on to old objects.

I tested that too and python 2.3.4 on RedHat 9 does not seem to behave
this way.  I wrote a script that creates 100Mb of random strings and
places them in a list.  Then it waits a few seconds and deletes the
list.  When I look at the process with top I see memory (RSS) go up by
about 100Mb and then a few seconds later it goes back down to just above
the level when the script started.  Then I put this in a loop.  I can
watch the memory usage just cycle up and down.  This is what I would
expect.

Thanks!
Andy