[Zope] Memory and Large Zope Page Templates

[Marco Bizzarri]

Santi Camps wrote:
> Hi all,
> 
> I'm trying to understand some questions about Zope and Memory 
> management.   I've configured a Zope with minimal Cache settings: 100 
> objects of cache, 4 threads and 20 Mb of ZEO Cache.   The Storage is a 
> DirectoryStorage of 4Gb.   Monitoring the Zope process, all is running 
> fine: it takes about 30 Mb of RAM.  I've see that just 2 of the threads 
> are working.
> 
> The problem begins when I execute a report.  The report is a 
> PageTemplateFile that calls a method to obtain a list of processed 
> results and send them to the user.   It takes some time of process, but 
> it works fine.   What I don't understand is why, after this report,  
> Zope is taking 120 Mb of RAM !!  Too much for the cache I have.
> I've look for for a memory leak, showing refcounts in method like this:
> 
>    def _get_refcounts(self):
>        import sys
>        import types
>        d = {}
>        sys.modules
>        # collect all classes
>        for m in sys.modules.values():
>            for sym in dir(m):
>                o = getattr (m, sym)
>                if type(o) is types.ClassType:
>                    d[o] = sys.getrefcount(o)
>        # sort by refcount
>        pairs = map (lambda x: (x[1],x[0]), d.items())
>        pairs.sort()
>        pairs.reverse()
>        return pairs
> 
> But there is nothing rare.   I can see how the refcounts of some objects 
> are heavely incresed during the process of the report, but when finished 
> they are freed.  But RAM of the Zope process don't decrease. 
> Anybody have an explanation ?  It's recomended to run this kind of 
> reports in a new process ?  In this case, there are any products to help 
> this way ?
> 
> Thanks in advance
> 
> Santi Camps

If you look at Zope-dev, there have been some discussion regarding leaks 
and memory management.

IIRC, I think that once the size of your process increases, it won't 
shrink... however, this does not mean that it will not reuse the same 
memory again...

I mean, if you experience the following, it should be normal:

initial size of Zope -> 30 MB
you run the report: new size -> 150 MB
after a while, you run again the report: new size: 150 MB

if you run again the report, and you end up with a size of 270 MB (150 + 
120) this is a problem.

HTH

Regards
Marco