[Zope] Zope Memory Problem

[Dieter Maurer]

Douwe Osinga wrote at 2003-11-18 18:01 +0100:
 > ...
 > I installed the product and found out that when the memory consumption
 > grows, I also get a *lot* of instances of IOBucket and OOBucket and
 > similar named objects,

These are data types from the "BTrees" package.
They are extensively used for indexes and the ZCatalog (among others).

 > but mostly DateTime objects.

They, too, probably come from the catalog.

 > ...
 >    zobj = context.Catalog.getobject( catitem.data_record_id_ )
 > 
 > where zobj has properties of DateTime type, so that might be the problem.
 > The logs actually suggest that this statement is then executed for 300
 > times a second, though the surrounding code does not seem to do that.

Wow! Quite fast....

 > Now, the ZClass is question is also derived from a class I created myself
 > in Python, so that is probably where the memory leak is to be found. The
 > thing is that I don't know what I'm looking for: how do I create a
 > memory leak in Python? Doesn't the garbage collector clear these things?
 > 
 > Any help is much appreciated.

It is quite difficult to track down memory leaks.

First of all: Python's main garbage collector uses reference counting 
(and therefore is not a standard garbage collector).
It can collect only non-cyclic garbage.
Cyclic garbage contains an unreachable object with a reference
(usually indirect) to itself.

Modern Python version have a collecting garbage collector as
well which is able to recycle also cyclic garbage.
However, external C extensions must play with it
and "ExtensionClass" (base class of almost all Zope classes)
does not yet do that. Thus, Zope objects containing cyclic
references are (usually) not collected.
Furthermore, the cyclic garbage collector will not release
and recycle garbage that contains objects with destructors ("__del__"
methods). This is because reclaiming such cycles may lead to
non-deterministic behaviour depending on the order of destructor
calls.

That's for background information...


Now to some techniques how to find the leak:

  *  Go to "Control_Panel" -> "Database Management" --> "Flush cache"
     and press "Minimize". This will clear the ZODB caches (which contribute
     to your reference counts)

  *  Reexamine the reference counts.

     Almost all of the "DateTime" and "*Bucket", "*Set" objects
     should have gone.

     If you still have lots of objects of some class, then
     such objects may be part of cycles. Examine there code.

If these steps do not reveal something strange -- let your system
run for quite a while and then redo the steps above.
Memory leaks are persistent -- they never go away, this means, when
you wait long enough, it gets easier to recognize the leaking object
class.

-- 
Dieter