[Zope] Zope 2.7.3 Memory Leaks

[Tim Peters]

[Andy Yates]
> ...
> For example if I have ab make X requests, X new sessions are created and
> memory use goes up by 100MB.  I also see about X objects in the
> transient object container.  Now I wait for the sessions to expire based
> on the "Data object timeout value".  Then I make a few calls to tickle
> the Transient system's garbage collection.  The objects in the toc goes
> down but memory use does not.  Some have posted that python does not
> release memory it just reuses it.  Well, when I make X more requests the
> memory is not reused.  Now it is using 200MB.

I know as much about the gory details of Python's memory-management as
anyone on Earth, so you can believe this:  if anyone gives you a
catchy one-line summary, don't believe them.  Python has many distinct
memory-management subsystems, and details are crucial.  In addition,
Python builds on top of the platform C library's malloc/free/realloc
system, and that builds in turn on the platform operating system
policies.  If you pursue it, you'll discover that it's a long
discussion just trying to figure out what "release memory" could
possibly mean.

That said, it's not generally expected that the virtual memory
highwater mark (which I'm guessing you're really talking about when
you say "now it is using 200MB") will grow rapidly and consistently.

> ...
> Could it be possible that the BTrees code which is written in C is
> leaking?

Yes -- and if you use an old enough version of ZODB, it's guaranteed
to leak <wink>.

> Memory leaked here would not show up as refcounts in Zope
> right?

For the most part, it should:  BTrees are built out of Python objects,
each with its own refcount (as is true of all Python objects).  To
Zope, BTree building blocks look like all other kinds of persistent
Python objects.

I haven't done this for a while, but when leaks in the BTree code were
still suspected (they truly aren't any more), I used to run the ZODB
test suite under a debug-build ZODB and a debug-build Python, for days
at a time in a loop.  There are special gimmicks you can use in a
debug-build Python to detect leaks.  This did find leaks, too.  I
stopped doing this after the last leak it pointed at took days to
track down, and turned out to be in a BTree endcase soooooo rare that
it may never have occurred in real life (but did occur once in one of
the test cases for "pathological" BTrees).

That doesn't mean there can't be more leaks, but it would be pretty
amazing if there were still a leak on any common path thru the BTree
code.

> It is FULL of malloc and free calls.  Isn't this a fairly new addition to Zope?

No.  The BTree code is really part of ZODB, and has been there (as far
as I know) forever; Jeremy Hylton and I did a lot concentrated work to
squash memory leaks in ZODB (including its BTree code) a couple years
ago.  The BTree code has been very stable (== has been changed very
little) since then.