[Zope] A memory leak in Zope, PageTemplateFile and macros?

[Dieter Maurer]

Marco Bizzarri wrote at 2005-5-6 22:35 +0200:
> ...
>However, if you access start_html, which uses the standard_template_html 
>as a macro, you can see an increase.

I saw a small one after thousands of requests.
> ...

>You're mentioning that these could be normal causes, 
>but what are these causes?

Small memory consumption increases can be caused by an effect
known as memory fragmentation. The "C" runtime library (and Python
on top of it) does not support memory compaction. Therefore, free memory
areas can only be joined together if they are adjacent (and even then
memory management may not do it). As a consequence, the free memory
tends to be fragmented in a large number of small blocks.

> ...
>I've tried this both on 2.7.3 and 2.7.6, and in both cases I'm 
>experiencing the problem. Do you have any suggestion on what I could 
>possibly investigate?

The usual approach in investigating memory leaks works as follows:

  *  you flush the ZODB caches
     ("Control_Panel --> Database Management --> <your database> --> Flush cache
     --> "minimize").

  *  you visit "Control_Panel --> Debug information" and
     "update snapshot"

  *  you perform your test

  *  you flush the ZODB caches

  *  you visit "Control_Panel" --> "Debug information"
     and analyse the changes in reference counts (in the middle
     of the page).

This allows you to recognize leaks at the level of instances (of classes).
The "LeakFinder" product helps you with the above steps.


I did this for your case and could not determine leaking instances.


When instances of static types (e.g. strings, integers, dictionaries, ...)
leak, they cannot be detected in this way. Neither can
leaked caused by fragmented memory.

Python can be build in a special way to analyse leaks of
any Python objects. Note, however, that all "C" extensions
(especially Zope) must be then build in a consistent way.

Tracking down such leaks is even more expensive that
those involving standard Zope objects.


There are tools around to analyse leaks at the "C" level.
I had used the commercial product "purify" once (a long time ago).
Andreas (Jung) told me that some open source project (I think
it was KDE) has built a similar tool.

-- 
Dieter