Memory Leak in Session Data Container
Good day everyone! I hope your weekend was as good as mine. I didn't have to work and I got to watch my son's first soccer (football) game! For the past couple of years (I think) the Session Data Container has been leaking memory. During this time several leaks have been patch but at least for our servers these fixes have only slowed down the process. As our use of Zope grows and traffic to our site increases so does the memory usage of Zope. Unless Zope is restarted it will eventually use all physical memory and swap space and crash the server. Over the past month the problem has once again grown to the point where we must restarted Zope daily, sometimes even every 8 hours. I don't know why this problem does not seem more wide spread. My guess is because our site has a large load and we make extensive use of session variables. Fortunately, I have an easy way to reproduce the problem and even better I've found a work around. My hope is that the work around will lead somebody more familiar with the innards of Zope to where the problem is located. TEST CASE To reproduce the problem I used the following: VA Linux Server w/ 1Gig Fedora Core 3 Zope 2.7.8-final, python 2.3.4, linux2 Python 2.3.4 (#1, Feb 2 2005, 12:11:53) [GCC 3.4.2 20041017 (Red Hat 3.4.2-6.fc3)] top and/or snmp to watch memory usage Step 1: In Zope I created the following script: session = context.REQUEST.SESSION s= 'Andy' * 102400; session['sessiontest1']=s The size of 's' depends purely on how fast you would like Zope to eat memory. On my server this will consume all physical memory in about 1000 calls. Step 2: Changed the session_data object timeout to 1 minute. This is just to cause the session objects to expire in 1 minute to hurry the test along. Changed the Max subobjects to 0 (unlimited). I don't know if this is needed, but it is how we have our production servers set. Step 3: Use ab (apache bench) to call the script 1000 times using 10 concurrent threads. This causes Zope to consume over 90% of the memory as one would expect. The transient object container (/temp_folder/session_data) will now show 1009 objects. I don't know where the 6-9 extra come from.? Step 4: Wait 1-2 minutes for the objects to expire. Note: Use a browser or ab to access some other resource in Zope and or the script a number of times. This is needed to "tickle" the transient object container's expiration/garbage collection system. (You've got to keep those buckets moving) As expected the transient object container now shows 0 objects. However, Zope never releases the memory. You can wait minutes, hours, days what ever all the while tickling Zope. Zope will NEVER release the memory until it is restarted. Oh, and new requests just use more memory. Zope is NOT just hanging on to it for future use. THE WORK ARROUND I created a script called onDelete Parameter List sdo, toc sdo.clear() Then in /temp_folder/session_data I set the Script to call when objects are deleted to /onDelete Now when the tests are run the same amount of memory is consumed, but it is released back to the system when the session objects expire. CONCLUSION To me it looks like data placed in the session object are not getting deleted when the session object expires. Anything placed in a session object is essentially leaked. When the onDelete script explicitly removes the contents of the session data object the memory leak stops. Comment? Questions? Fixes? ;-) Thanks! Andy Yates
At Monday 10/4/2006 14:43, Andy Yates wrote:
Fortunately, I have an easy way to reproduce the problem and even better I've found a work around. My hope is that the work around will lead somebody more familiar with the innards of Zope to where the problem is located.
Quoting Andreas Jung on this list a few weeks ago: "Please put such _bug_ into the bugtracker. Reporting bugs to the mailinglist hoping that someone picks it up is insane, sorry." That is, go to http://www.zope.org/Collectors/Zope and file a bug report. Gabriel Genellina Softlab SRL
Andy Yates wrote at 2006-4-10 12:43 -0500:
... Fortunately, I have an easy way to reproduce the problem and even better I've found a work around. My hope is that the work around will lead somebody more familiar with the innards of Zope to where the problem is located.
Congratulations! I will look into this within the next days (in Zope 2.8.1, however).
... CONCLUSION
To me it looks like data placed in the session object are not getting deleted when the session object expires. Anything placed in a session object is essentially leaked. When the onDelete script explicitly removes the contents of the session data object the memory leak stops.
Comment? Questions? Fixes? ;-)
The ZODB cache may keep your session objects (and thereby its content) alive. As you probably know, each ZODB connection has an associated cache containing the recently accessed objects. Objects in the cache can be ghosts (they contain no data) or real objects. Ghosts disappear from the cache as soon as the last reference in the application is deleted. For real objects, however, the cache holds an extra reference and thereby prevents the object release. For the storage, an object held by the cache looks exactly equal to an object held be the application. It cannot get rid of it. You can try to flush (minimize) the caches associated with the temporary storage and see whether this releases the memory (the release may be postponed, as the "TempStorage" performs garbage collection only at certain times). If this should be your problem, you can reduce the size of the ZODB caches. -- Dieter
Dieter Maurer wrote at 2006-4-11 22:14 +0200:
Andy Yates wrote at 2006-4-10 12:43 -0500:
... Fortunately, I have an easy way to reproduce the problem and even better I've found a work around. My hope is that the work around will lead somebody more familiar with the innards of Zope to where the problem is located.
Congratulations!
I will look into this within the next days (in Zope 2.8.1, however).
I checked this: There is no memory leak related to session handling in Zope 2.8.1!
... The ZODB cache may keep your session objects (and thereby its content) alive.
Almost surely, this is what happened in Andy's test. Here is a description how I checked for the existence of a memory leak: I instrumented the pickles in "tempstorage.TemporaryStorage.TemporaryStorage" such that I can find out the number of pickles stored (similar to the reference counts in "Control_Panel/DebugInfo"). I performed a mass test, creating thousands of sessions (similar to Andy's test). The pickle number stored by "TemporaryStorage" grew as expected (as it stores the sessions). I forced all session objects maintained by "/temp_folder/session_data" to be discarded. Creating a single new session caused almost all "TemporaryStorage" pickles to be released. However, the ZODB caches still contained about 20.000 session objects. It was very surprising (and contrary to my previous message) that the storage flushed most of its pickles while references to them were still in the ZODB caches. This looks like a bug (but not a memory leak). I flushed the ZODB caches. I repeated the test and verified that the number of pickles remaining in "TemporaryStorage" does not increase. Results: * no memory leak! * maybe something that looks like a memory leak caused by the ZODB caches which may hold up to 20.000 (with standard configuration) no longer used objects A more senseful configuration can work around this. -- Dieter
participants (3)
-
Andy Yates -
Dieter Maurer -
Gabriel Genellina