[Zope-dev] Memory Leak debug - help

[Júlio Dinis Silva]

>I use a catalog-intensive setup, and my memory footprint for Zope
>seems to slowly increase over several days.  Might this be a leak?
>On my 2 ZEO identical clients, I get about an average of 50-60
>catalog queries per minute each, and load is distributed equally
>(round-robin) over the 2 nodes by my proxy server.  Both nodes were
>restarted at the same time yesterday, yet the one node that performed
>a re-indexing of about 16,000 objects early this morning has an RSS
>of 242 MB vs. 219 MB on the other node.  The RSS on both nodes had
>grown to over 450 MB earlier yesterday before I restarted Zope, so I
>wonder if that is a leak, esp. in regard to re-indexing a catalog, or
>something else.

I know one can do some bad coding that leads into memory leaks.
The first one is when you perform a query in a zcatalog. Its very
common to put results in the REQUEST using REQUEST.set. This procedure
creates a potencial risk for memory leaks. I actually tracked one
down: I had a heavy query to a zcatalog that each time it was executed
the python footprint increased in 5 Mb when query started and decrease
3Mb as query ends "leaking" 2Mb. I was using the REQUEST.set code and
after converting this into a set of dtml-let and dtml-return the
"leak" from executing the mentioned query was no more, ceased to be.

Another thing I done was to track down code which create
objects and dont implement nothing to delete them from memory.
The tipical error is, for instance, when you in a external method do 
something like:

----------------

file = open('BIG','r')
buffer = file.read(4000000)

[... execute some code with buffer ...]

del buffer
return something

----------------

Most people forget the "del buffer" part.

Another possibility I ran was concerned with the new BTree
implementation since zope2-I-dont-remember-version. With this new
BTrees one can use a manage_convertBTrees to convert the existing
catalogs to the new implementation. I converted all my zcatalogs to
this new implementation and, I cant put my hands on fire for this but,
I think the leaks started to increase.
And this rumour gain another strength when I saw on Top of RefCounts
this BTree class, which is a C extension and probably impossible to
patch with the leakFinder, just like Shane said.

I'll continue to look on zcatalogs because of this relation
between the manage_convertBtrees and the finding of
BTrees.IIBtree.IIBucket at the top of my servers refcount.

Regards,
Julio Dinis Silva

_________________________________________________________________
Get your FREE download of MSN Explorer at http://explorer.msn.com/intl.asp