[Zope] Zope 2.7.3 Memory Leaks

[Andy Yates]

> > The fact remains that if given enough time Zope will use so much
memory
> > that it crashes the server.  I've reproduced this on at least 3
Linux servers
> > with a clean install of Zope.  The pattern SEEMS to be the more data
I
> > put in a session object the faster the server dies.  This can not be
> > normal behavior.  I'm used to servers running for months.  I have to
> > restart Zope every 3 days at least.  I need to at least have a long
> > weekend away form this thing!  ;-)
> >
> > Can anybody please verify my results.  I've given a very simple test
> > case.  If I'm only one in the world that can cause this to happen
then
> > fine.  At least that narrows my search to something internal to our
> > system.
> 
> I think people are trying to help, and you need to follow up more.

Oh I know and I'm grateful.  Please don't misunderstand.  My frustration
is with the situation not with the members of the list.  I have a high
respect for the Zope community and especially those that take to time to
help with people's bizarro problems.

> For example, I didn't see a reply to Tres's questions in this thread,
> and I didn't see what looked like a particularly relevant reply to
> Dieter's observation that the test setup appears to provoke an
> unbounded number of sessions each with a 45-minute timeout (although
> Dieter appeared to miss this part of the message he was replying to:
> 
>     The session timeout does not seem to affect the memory use.  We
even
>     tried setting the session timeout to 1 minute.
> 
> your reply to him didn't have any indication of how many
> requests/sessions were getting created; and you didn't respond to his
> point that he limits max sessions to 5000 -- Tres also asked about max
> # of sessions).
> ...
>

The upper bound on the sessions is not relevant in my opinion.  We have
set it to unlimited because no matter what we have set it to in the past
we always seem to spike up to it when nobody is around to delete the
session objects by hand.  This happens because of something very
specific to the wi-fi hotspot market that I'd rather not go into.  In
most cases we avoid these spikes but it can happen so we do not set a
limit.  99.9% of the time we have 500-2000 objects in the transient
object container.  

Dieter is right.  My test could provoke an unbound number of sessions
but it doesn't because of the way the test is run.  I'm not trying to
simulate one user making 1000 requests.  I'm trying to simulate 1000
users making one request which should produce 1000 sessions.  These
sessions are allowed to timeout and then we create 1000 more.  If this
cycle is repeated enough times all memory is consumed and the server
crashes.  This happens even if there are hours between cycles.  The
memory use never goes down it just grows even though the number of
sessions never goes over 1000.  

> How many simultaneous sessions are we talking about?  A few hundred 
> simultaneous users could chew up RAM pretty fast.  What is your "max 
> number of sessions" value?
>
> Tres.

Sorry, Tres's.  Let me explain.  The max sessions or the amount of data
in each session should not matter.  The server should reach some steady
state value of memory usage.  This is because you always have new users
coming to the site which creates new session objects, but on the other
end you have users that have gone away and so their sessions expire.
When one restarts the server I would expect the steady state to happen
within 3 or 4 times the session timeout value.  My problem is that this
steady state never seems to happen.  Memory use just goes up and up.


I have found something today!  We store some images for our site in a
MySQL database.  I noticed today that these images (some could be large)
are copied to the session object.  This was probably done to reduce
queries to the database.  There are so many caches along the way (
browser, Zope, zsql...) that this is probably not needed.  I turned this
feature off.  At least this should push the horizon off a bit.  Time
will tell.

 
> > Please don't take offence I'm sure the BTree code is really cool.
> 
> Oh, no offense at all!  C code usually leaks like a sieve when it's
> new <wink>.  I was just trying to steer you away from the BTree code
> in this case, because it's not new, has benefited from a lot of work
> to plug leaks, and just isn't a high-probability suspect anymore.
> BTW, you *do* see BTree building blocks from the debug control panel;
> they're in the lines like:
> 
> BTrees.OOBTree.OOBTree  2019    11997   +9978
> BTrees.OOBTree.OOBucket         1297    5512    +4215
> 
> > I'm just grasping at straws.  When this problem started I was told
the
> > problem must be in MY code.  So I removed ALL my code and made a
very
> > simple test case that reproduces the problem.  Then I was told well
you
> > just don't understand how memory REALLY works.
> 
> I didn't see anyone say that in this thread.  Maybe I missed it.

Perhaps stated too strong that's just my impression.  I was referring to
the long posts last month about how Zope, python, and Linux manage
memory.  That's all voodoo to me.  I just don't understand how this
simple test can cause so much memory to be used.
 
> >  At this point I don't really care how memory really works.  I just
want to be
> > able to put data in session objects without my server crashing.
> 
> Sure.
> 
> > Run my test case and see what happens.  I double dog dare you!  ;-)
> 
> Sorry, but I don't have time to dig into this at all.  My replies have
> just regurgitated info I knew off the top of my head.  That doesn't
> mean I think your problem is trivial, it just means I can't volunteer
> substantive help.

Can't blame you there.  I don't have time for this either.  

I'm going to restart my Zope server just for good measure and go to the
gym.  Y'all have to great weekend!

Andy