Douwe Osinga wrote at 2003-11-18 18:01 +0100:
... I installed the product and found out that when the memory consumption grows, I also get a *lot* of instances of IOBucket and OOBucket and similar named objects,
These are data types from the "BTrees" package. They are extensively used for indexes and the ZCatalog (among others).
but mostly DateTime objects.
They, too, probably come from the catalog.
... zobj = context.Catalog.getobject( catitem.data_record_id_ )
where zobj has properties of DateTime type, so that might be the problem. The logs actually suggest that this statement is then executed for 300 times a second, though the surrounding code does not seem to do that.
Wow! Quite fast....
Now, the ZClass is question is also derived from a class I created myself in Python, so that is probably where the memory leak is to be found. The thing is that I don't know what I'm looking for: how do I create a memory leak in Python? Doesn't the garbage collector clear these things?
Any help is much appreciated.
It is quite difficult to track down memory leaks. First of all: Python's main garbage collector uses reference counting (and therefore is not a standard garbage collector). It can collect only non-cyclic garbage. Cyclic garbage contains an unreachable object with a reference (usually indirect) to itself. Modern Python version have a collecting garbage collector as well which is able to recycle also cyclic garbage. However, external C extensions must play with it and "ExtensionClass" (base class of almost all Zope classes) does not yet do that. Thus, Zope objects containing cyclic references are (usually) not collected. Furthermore, the cyclic garbage collector will not release and recycle garbage that contains objects with destructors ("__del__" methods). This is because reclaiming such cycles may lead to non-deterministic behaviour depending on the order of destructor calls. That's for background information... Now to some techniques how to find the leak: * Go to "Control_Panel" -> "Database Management" --> "Flush cache" and press "Minimize". This will clear the ZODB caches (which contribute to your reference counts) * Reexamine the reference counts. Almost all of the "DateTime" and "*Bucket", "*Set" objects should have gone. If you still have lots of objects of some class, then such objects may be part of cycles. Examine there code. If these steps do not reveal something strange -- let your system run for quite a while and then redo the steps above. Memory leaks are persistent -- they never go away, this means, when you wait long enough, it gets easier to recognize the leaking object class. -- Dieter