[Zope] zope vs. memory [was Re: Refcounts, ZODB Cache and memory leaks?]

[Dieter Maurer]

Please stay on the list!
Readded...

Dylan Jay wrote at 2005-5-29 23:31 +1000:
>> >- Memory grows and mostly doesn't shrink.
>> 
>> That's normal behavior.
>
>Normal as in desirable or normal as in known bug? 

Normal as in "extremely difficult to avoid".

When you read a book about memory management and
look at the operating system provided API for
memory management you will understand why.

>> >- Its not due to objects that can't be collected by the GC
>> >- Its not due to the ZODB cache
>> >
>> >That leaves the following causes I think
>> >- RAMCache etc and module/class variables. Either mine, plone's or
>> zope's.

If the reference counts as shown by "Control_Panel --> Debug information"
do not show them, then the leaked objects are neither class
nor "ExtensionClass" instances and are probably elementary
Python types.

Some moths ago, I found such a leak in the "TempStorage"
implementation. However, this is fixed in recent Zope versions
(I think since 2.7.5).

> ...
>> Python has no big chance. The memory management options of the
>> underlaying operating systems are quite limited and Python
>> uses the C memory management at its lowest level.
>> Thus, it must live with what "C" provides.
>
>I don't really understand the implications of what your saying. Are you
>saying that oses don't allow python to free memory?

Something like that:

  "posix" provides two ways to add address space for a process:
  "brk" and "mmap".
  
  "brk" allows to add and release space at the *end*
  of the address space. You cannot release such space unless
  a complete range is free at the very end!

  "mmap" can add (and release) space anywhere but only
  in large chunks and only at corresponding boundaries.
  You can use "mmap" only for large memory blocks managed
  atomically (i.e. allocated and released as a whole).

  The "C" "malloc" usually uses "brk" to allocate large chunks
  of memory and subdivides them as needed (for efficiency reasons).
  While it would be possible to check for the possibility
  to release such blocks again, it would be expensive (and
  not so often successful). Therefore, it is rarely done.

Read more in a book about memory management!

> ...
>am I right in saying that using
>the catalog involves lots of btree objects which are objects and would not
>be affected by the memory issue above?

"BTree" objects are objects and would be counted in
the "Reference Counts" display of "Control_Panel --> Debug Information".
You should be able to detect leaks involving them by analysing
this display (do not forget to flush the caches before you trust
the reference counts!).

> ...
>> Did I not tell you about a special Python build to
>> analyse memory leaks?
>
>No.

You can build Python to account for any allocated object.
However, all extensions must then be build with this same
option (otherwise, the object structures do not agree and chaos results).

You will need many hours to analyse the thousands of objects
to distinguish between the leaked objects and the one rightfully
used by the application...

-- 
Dieter