[Zope] zope vs. memory [was Re: Refcounts, ZODB Cache and memory leaks?]

[Dylan Jay]

Just in case anyone can shed anymore light on zope memory woes, here is 
where I've got to in my investigation:

- Memory grows and mostly doesn't shrink.
- Its not due to objects that can't be collected by the GC
- Its not due to the ZODB cache

That leaves the following causes I think
- RAMCache etc and module/class variables. Either mine, plone's or zope's.
- Pythons design of not releasing memory from small objects back to the OS
http://www.sauria.com/~twl/conferences/pycon2005/20050325/Improving%20Python's%20Memory%20Allocator.html

   Has anyone tried to determine how much of an average catalog search is 
<256kb? I'm guessing a lot.

- Some other kind of reference being help somewhere in the zope machinery.



Any tips anyone?


>> "Peter Bengtsson" <mail at peterbe.com> wrote in message 
>> news:loom.20050426T185741-270 at post.gmane.org...
>>> Dylan Jay <me <at> dylanjay.com> writes:
>>>
>>>> If anyone has experience with finding these kind of rouge references 
>>>> I'd
>>>> really appreciate any tips.
>>>
>>> I have no idea how to patch tings with LeakFinder to find your problems 
>>> because
>>> LeakFinder, albeit a great piece of code, attempts an incredibly 
>>> difficult task.
>>> Most likely, your memory leaks is due to an assignment of really large 
>>> lists on
>>> complex objects (eg. more complex than strings or numbers).
>>
>> Can you expound on this a little as this sounds very interesting? Why do 
>> assignments to a list normally result in leaks? Do you mean lists of 
>> complex objects or complex objects that have lists?
>>
>>> Look for any constructs where you extract large lists and do operations 
>>> on them.
>>> Once you have, if you ever, contain these, return to the mailing list 
>>> and we can
>>> try to solve it together.
>>
>> I certainly do operations on lists, partucularly LazyMap etc. I do 
>> operations with catalog results and map them to lazily to custom objects 
>> which are joined by aquisition to catalog brains.
>>
>> I'm new to memory leak debugging in python but have done it before in 
>> java.
>>>From what I understand garbage collection happens differently in java. In
>> java you have to avoid circular references, is the same true in python?
>>
>>
>>>
>>> _______________________________________________
>>> Zope maillist  -  Zope at zope.org
>>> http://mail.zope.org/mailman/listinfo/zope
>>> **   No cross posts or HTML encoding!  **
>>> (Related lists -
>>> http://mail.zope.org/mailman/listinfo/zope-announce
>>> http://mail.zope.org/mailman/listinfo/zope-dev )
>>>
>>
>>
>>
>> _______________________________________________
>> Zope maillist  -  Zope at zope.org
>> http://mail.zope.org/mailman/listinfo/zope
>> **   No cross posts or HTML encoding!  **
>> (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
>> http://mail.zope.org/mailman/listinfo/zope-dev )
>>
>
>
>
> _______________________________________________
> Zope maillist  -  Zope at zope.org
> http://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce
> http://mail.zope.org/mailman/listinfo/zope-dev )
>