[Zope] Zope 2.7.3 Memory Leaks

[Andy Yates]

> 
> Andy Yates wrote at 2005-1-5 15:30 -0600:
> > ...
> >In fact we can reproduce this behavior
> >on a fresh install.  On a fresh system create a python script that
puts
> >data in a session object.
> >
> >s = context.REQUEST.SESSION
> >t = ' ' * 1024
> >s['data'] = t
> >print "foo"
> >return printed
> >
> >Then call this script with your favorite bench marking program.  We
used
> >apache bench (ab).  This will cause python2.3 to consume all
available
> >memory and crash or lockup.
> 

Can anybody else reproduce these results?

> And I am not surprised at all:
> 
>   You allow an unlimited number of sessions.
> 
>   Your session timeout is 45 minutes
> 
>   "ab" does not honour the "set cookie" requests for the sessioning.
>   Thus, each request creates a new session.
>
Yes, I understand that ab will not set the cookie and each request will
look like a new connection to Zope.  That is not the point.  The point
is that whatever memory is used by these sessions should be released
once the session has expired.  This does not seem to happen.  Memory use
just grows and grows until the server dies.  We turned on the debug
output in Transient.py and the buckets seem to get expired as expected
but like I said memory used just grows without bound.

For example if I have ab make X requests, X new sessions are created and
memory use goes up by 100MB.  I also see about X objects in the
transient object container.  Now I wait for the sessions to expire based
on the "Data object timeout value".  Then I make a few calls to tickle
the Transient system's garbage collection.  The objects in the toc goes
down but memory use does not.  Some have posted that python does not
release memory it just reuses it.  Well, when I make X more requests the
memory is not reused.  Now it is using 200MB.

 
>   This way, it should not be difficult to use
>   a large amount of RAM.
>   My Zope serves about 500 request/s, 30.000 request/min
>   and 1.350.000 requests in 45 min.
>   You must expect a number of sessions in this order
 
> > ...
> >In a very basic sense this is all our production server does.  When a
> >user first comes to our site we query a mysql database for several
> >hundred values.  These values are then stored in the session object
as a
> >map.  All other pages are built dynamically based on the values
stored
> >in the session object.
> >
> >Is the session object system just not supposed to be used like this?
> 
> We use the session system in quite a similar way -- without problems.
> But, we have a maximal session count of 5.000.
> 
> We, too, observe some increase in memory requirements which we do
> not yet understand: memory seems to increase from about 300 MB
> to about 600 MB within a week. Therefore, we restart
> once every Sunday night. But, the problem has not yet been pressing
> and we did not yet seriously tried to understand it.
> 

Consider the problem pressed.  ;-)  

Well, I guess I'll just restart Zope every day for now.  

Could it be possible that the BTrees code which is written in C is
leaking?  Memory leaked here would not show up as refcounts in Zope
right?  It is FULL of malloc and free calls.  Isn't this a fairly new
addition to Zope?

Thanks again for your help and insight!
Andy