-----Original Message----- From: Chris McDonough [mailto:chrism@plope.com] Sent: Thursday, December 02, 2004 5:17 PM To: Andy Yates Cc: zope@zope.org Subject: Re: [Zope] Zope 2.7.3 Memory Leaks
On Thu, 2004-12-02 at 17:24, Andy Yates wrote:
Now it seems to leak memory (400+ Mb) to the point I have to restart Zope every 3 days.
I've read that memory leaks are almost always caused by the programmer, so I set up some very simple tests that did not involve existing code.
Great idea.
Note that I'm not trying to find a problem with Zope. My plan was to start with something very simple and then add more and more of the things we do on our production site and see when it starts leaking. However, even the simplest tests seem to leak memory.
Next I created an 8Mb page template. This caused the memory usage to climb much faster. After the test the memory never goes back down.
Do these templates access the sessioning machinery?
No. It is just a page template file with 8Mb of random text.
I installed LeakFinder and started running more tests.
I used ab to get a python script that stored a 1k string in the session object. The transient object container timeout was set to 1 minute. This also caused Zope to consume memory. LeakFinder said the Products.Transience.TransientObject.TransientObject ref count grew with each request and never went down. Everything else seemed to level off. The debug output of Transience.py showed that the "buckets" seemed to be getting deleted as expected, but the memory usage never goes down. 1 minute after the test stopped the transient object container showed that there were no more items in the container. It seems like when Zope deletes expired sessions but contents of the sessions are not deleted.
This is actually probably normal.
TransientObject buckets hang around for some period of time before they are "garbage collected" (deleted from their container). A particular TransientObject isn't garbage collected immediately when it expires, but gets gc'ed much later along with other TransientObjects that were created around the same time as a side effect of otherwise exercising the sessioning machinery.
Thanks for the explaination of how the transient gc works. That is exactly what I found when I looked in the source code and I can see all of this happening when I turn on the debug output. Sessions are created and placed in buckets. The active sessions move to the current bucket and expired session and old empty buckets get removed. Then there is a 1 in 6 chance they will be gc'ed on each request. I see all this happening but when the buckets are gc'ed the memory usage according to top does not go down. I ran the test again that puts a 1k string in the session objects. I used ab to run the python script 3000 times with a concurrency of 10. Python's memory went from 22Mb to 54Mb. Transient objects went from 0 to about 900 and stayed steady during the test. After the test the trans obs dropped back down to 0 as they expired. I continued to make occasional Zope requests to plain pages to make sure gc was getting tickled. After 8+ hours of this the memory was never released. Python was still using 54Mb. If I continue to create session objects the 54Mb is not reused by python it just continues to grow.
Note that Python will usually not release memory back to the OS once
it
has been allocated for its own use, so a large concrete amount of memory consumed by Python may not even be an indicator of a leak! See http://mail.python.org/pipermail/python-dev/2004-October/049480.html for more info. The only solid basis for determining leaks is refcounts, and it's tricky to figure out whether growing refcounts are leaks or if they're just side effects of the ZODB cache holding on to old objects.
I tested that too and python 2.3.4 on RedHat 9 does not seem to behave this way. I wrote a script that creates 100Mb of random strings and places them in a list. Then it waits a few seconds and deletes the list. When I look at the process with top I see memory (RSS) go up by about 100Mb and then a few seconds later it goes back down to just above the level when the script started. Then I put this in a loop. I can watch the memory usage just cycle up and down. This is what I would expect. Thanks! Andy