[Zope] Memory Leak in Session Data Container

[Andy Yates]

Good day everyone!  I hope your weekend was as good as mine.  I didn't
have to work and I got to watch my son's first soccer (football) game!

For the past couple of years (I think) the Session Data Container has
been leaking memory.  During this time several leaks have been patch but
at least for our servers these fixes have only slowed down the process.
As our use of Zope grows and traffic to our site increases so does the
memory usage of Zope.  Unless Zope is restarted it will eventually use
all physical memory and swap space and crash the server.  Over the past
month the problem has once again grown to the point where we must
restarted Zope daily, sometimes even every 8 hours.

I don't know why this problem does not seem more wide spread.  My guess
is because our site has a large load and we make extensive use of
session variables. 

Fortunately, I have an easy way to reproduce the problem and even better
I've found a work around.  My hope is that the work around will lead
somebody more familiar with the innards of Zope to where the problem is
located.

TEST CASE

To reproduce the problem I used the following:
VA Linux Server w/ 1Gig
Fedora Core 3
Zope 2.7.8-final, python 2.3.4, linux2
Python 2.3.4 (#1, Feb 2 2005, 12:11:53) [GCC 3.4.2 20041017 (Red Hat
3.4.2-6.fc3)]
top and/or snmp to watch memory usage

Step 1:
In Zope I created the following script:

session = context.REQUEST.SESSION
s= 'Andy' * 102400;
session['sessiontest1']=s

The size of 's' depends purely on how fast you would like Zope to eat
memory.  On my server this will consume all physical memory in about
1000 calls.

Step 2:
Changed the session_data object timeout to 1 minute.  This is just to
cause the session objects to expire in 1 minute to hurry the test along.
Changed the Max subobjects to 0 (unlimited).  I don't know if this is
needed, but it is how we have our production servers set.

Step 3:
Use ab (apache bench) to call the script 1000 times using 10 concurrent
threads.
This causes Zope to consume over 90% of the memory as one would expect.
The transient object container (/temp_folder/session_data) will now show
1009 objects.  I don't know where the 6-9 extra come from.?

Step 4:
Wait 1-2 minutes for the objects to expire.
Note: Use a browser or ab to access some other resource in Zope and or
the script a number of times.  This is needed to "tickle" the transient
object container's expiration/garbage collection system.  (You've got to
keep those buckets moving)
As expected the transient object container now shows 0 objects.
However, Zope never releases the memory.  You can wait minutes, hours,
days what ever all the while tickling Zope.  Zope will NEVER release the
memory until it is restarted.  Oh, and new requests just use more
memory.  Zope is NOT just hanging on to it for future use.


THE WORK ARROUND

I created a script called onDelete
Parameter List sdo, toc
sdo.clear()


Then in /temp_folder/session_data I set the Script to call when objects
are deleted to /onDelete

Now when the tests are run the same amount of memory is consumed, but it
is released back to the system when the session objects expire.

CONCLUSION

To me it looks like data placed in the session object are not getting
deleted when the session object expires.  Anything placed in a session
object is essentially leaked.  When the onDelete script explicitly
removes the contents of the session data object the memory leak stops.


Comment?  Questions?  Fixes? ;-)

Thanks!
Andy Yates