Please stay on the list! Readded... Dylan Jay wrote at 2005-5-29 23:31 +1000:

...

...

- Memory grows and mostly doesn't shrink.

That's normal behavior.

Normal as in desirable or normal as in known bug?

Normal as in "extremely difficult to avoid". When you read a book about memory management and look at the operating system provided API for memory management you will understand why.

...

...

- Its not due to objects that can't be collected by the GC - Its not due to the ZODB cache

That leaves the following causes I think - RAMCache etc and module/class variables. Either mine, plone's or zope's.

If the reference counts as shown by "Control_Panel --> Debug information" do not show them, then the leaked objects are neither class nor "ExtensionClass" instances and are probably elementary Python types. Some moths ago, I found such a leak in the "TempStorage" implementation. However, this is fixed in recent Zope versions (I think since 2.7.5).

...

...

Python has no big chance. The memory management options of the underlaying operating systems are quite limited and Python uses the C memory management at its lowest level. Thus, it must live with what "C" provides.

I don't really understand the implications of what your saying. Are you saying that oses don't allow python to free memory?

Something like that: "posix" provides two ways to add address space for a process: "brk" and "mmap". "brk" allows to add and release space at the *end* of the address space. You cannot release such space unless a complete range is free at the very end! "mmap" can add (and release) space anywhere but only in large chunks and only at corresponding boundaries. You can use "mmap" only for large memory blocks managed atomically (i.e. allocated and released as a whole). The "C" "malloc" usually uses "brk" to allocate large chunks of memory and subdivides them as needed (for efficiency reasons). While it would be possible to check for the possibility to release such blocks again, it would be expensive (and not so often successful). Therefore, it is rarely done. Read more in a book about memory management!

... am I right in saying that using the catalog involves lots of btree objects which are objects and would not be affected by the memory issue above?

"BTree" objects are objects and would be counted in the "Reference Counts" display of "Control_Panel --> Debug Information". You should be able to detect leaks involving them by analysing this display (do not forget to flush the caches before you trust the reference counts!).

...

...

Did I not tell you about a special Python build to analyse memory leaks?

No.

You can build Python to account for any allocated object. However, all extensions must then be build with this same option (otherwise, the object structures do not agree and chaos results). You will need many hours to analyse the thousands of objects to distinguish between the leaked objects and the one rightfully used by the application... -- Dieter