Refcounts, ZODB Cache and memory leaks?
Hi, I'm trying to track down some memory leaks using LeakFinder. I get some custom objects with refcounts that keep going up on use. When I patch them using leakfinder I get no results which should be correct as they should be loaded from the ZODB rather than created. The problem is, why won't they go away after GC or clearing the ZODB cache? If anyone has experience with finding these kind of rouge references I'd really appreciate any tips. Dylan Jay Web Builder dylan@pretaweb.com --- www.pretaweb.com - Reliable & responsive CMS website development www.meetmemap.com - makes giving directions easy www.eatmanifesto.com - Traditional Sustainable and Delectable Ph:+61(0)423198306 ~ Y!+Skype:dylan_jay ~ ICQ:520341 ~ SIP:djay@babble.net
Dylan Jay <me <at> dylanjay.com> writes:
If anyone has experience with finding these kind of rouge references I'd really appreciate any tips.
I have no idea how to patch tings with LeakFinder to find your problems because LeakFinder, albeit a great piece of code, attempts an incredibly difficult task. Most likely, your memory leaks is due to an assignment of really large lists on complex objects (eg. more complex than strings or numbers). Look for any constructs where you extract large lists and do operations on them. Once you have, if you ever, contain these, return to the mailing list and we can try to solve it together.
"Peter Bengtsson" <mail@peterbe.com> wrote in message news:loom.20050426T185741-270@post.gmane.org...
Dylan Jay <me <at> dylanjay.com> writes:
If anyone has experience with finding these kind of rouge references I'd really appreciate any tips.
I have no idea how to patch tings with LeakFinder to find your problems because LeakFinder, albeit a great piece of code, attempts an incredibly difficult task. Most likely, your memory leaks is due to an assignment of really large lists on complex objects (eg. more complex than strings or numbers).
Can you expound on this a little as this sounds very interesting? Why do assignments to a list normally result in leaks? Do you mean lists of complex objects or complex objects that have lists?
Look for any constructs where you extract large lists and do operations on them. Once you have, if you ever, contain these, return to the mailing list and we can try to solve it together.
I certainly do operations on lists, partucularly LazyMap etc. I do operations with catalog results and map them to lazily to custom objects which are joined by aquisition to catalog brains. I'm new to memory leak debugging in python but have done it before in java.
From what I understand garbage collection happens differently in java. In java you have to avoid circular references, is the same true in python?
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
I've been going some testing with gc.set_debug etc and it looks like there isn't a memory leak or at least not one the garbage collector can find. That means that the data must be remaining reachable somehow. -- Dylan Jay --- http://www.meetmemap.com - makes giving directions easy http://www.pretaweb.com - Update-it-yourself, low cost websites http://www.eatmanifesto.com - against all dumbing down of food Y!:dylan_jay ICQ:520341 MSN:djsq@hotmail.com Ph:+61423198306 "Dylan Jay" <gmane@dylanjay.com> wrote in message news:d5pu2j$q7j$1@sea.gmane.org...
"Peter Bengtsson" <mail@peterbe.com> wrote in message news:loom.20050426T185741-270@post.gmane.org...
Dylan Jay <me <at> dylanjay.com> writes:
If anyone has experience with finding these kind of rouge references I'd really appreciate any tips.
I have no idea how to patch tings with LeakFinder to find your problems because LeakFinder, albeit a great piece of code, attempts an incredibly difficult task. Most likely, your memory leaks is due to an assignment of really large lists on complex objects (eg. more complex than strings or numbers).
Can you expound on this a little as this sounds very interesting? Why do assignments to a list normally result in leaks? Do you mean lists of complex objects or complex objects that have lists?
Look for any constructs where you extract large lists and do operations on them. Once you have, if you ever, contain these, return to the mailing list and we can try to solve it together.
I certainly do operations on lists, partucularly LazyMap etc. I do operations with catalog results and map them to lazily to custom objects which are joined by aquisition to catalog brains.
I'm new to memory leak debugging in python but have done it before in java.
From what I understand garbage collection happens differently in java. In java you have to avoid circular references, is the same true in python?
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Just in case anyone can shed anymore light on zope memory woes, here is where I've got to in my investigation: - Memory grows and mostly doesn't shrink. - Its not due to objects that can't be collected by the GC - Its not due to the ZODB cache That leaves the following causes I think - RAMCache etc and module/class variables. Either mine, plone's or zope's. - Pythons design of not releasing memory from small objects back to the OS http://www.sauria.com/~twl/conferences/pycon2005/20050325/Improving%20Python's%20Memory%20Allocator.html Has anyone tried to determine how much of an average catalog search is <256kb? I'm guessing a lot. - Some other kind of reference being help somewhere in the zope machinery. Any tips anyone?
"Peter Bengtsson" <mail@peterbe.com> wrote in message news:loom.20050426T185741-270@post.gmane.org...
Dylan Jay <me <at> dylanjay.com> writes:
If anyone has experience with finding these kind of rouge references I'd really appreciate any tips.
I have no idea how to patch tings with LeakFinder to find your problems because LeakFinder, albeit a great piece of code, attempts an incredibly difficult task. Most likely, your memory leaks is due to an assignment of really large lists on complex objects (eg. more complex than strings or numbers).
Can you expound on this a little as this sounds very interesting? Why do assignments to a list normally result in leaks? Do you mean lists of complex objects or complex objects that have lists?
Look for any constructs where you extract large lists and do operations on them. Once you have, if you ever, contain these, return to the mailing list and we can try to solve it together.
I certainly do operations on lists, partucularly LazyMap etc. I do operations with catalog results and map them to lazily to custom objects which are joined by aquisition to catalog brains.
I'm new to memory leak debugging in python but have done it before in java.
From what I understand garbage collection happens differently in java. In java you have to avoid circular references, is the same true in python?
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Dylan Jay wrote at 2005-5-13 14:01 +1000:
Just in case anyone can shed anymore light on zope memory woes, here is where I've got to in my investigation:
- Memory grows and mostly doesn't shrink.
That's normal behavior.
- Its not due to objects that can't be collected by the GC - Its not due to the ZODB cache
That leaves the following causes I think - RAMCache etc and module/class variables. Either mine, plone's or zope's.
When I remember right, then also the reference counts do not go up. In this case, the leak must be caused by elementary Python types (more precisely by objects the type of with is statically allocated) -- if Python objects cause the leak at all.
- Pythons design of not releasing memory from small objects back to the OS http://www.sauria.com/~twl/conferences/pycon2005/20050325/Improving%20Python's%20Memory%20Allocator.html
Python has no big chance. The memory management options of the underlaying operating systems are quite limited and Python uses the C memory management at its lowest level. Thus, it must live with what "C" provides.
Has anyone tried to determine how much of an average catalog search is <256kb?
What is an "average" catalog search? How do you measure the size of a catalog search?
- Some other kind of reference being help somewhere in the zope machinery.
Did I not tell you about a special Python build to analyse memory leaks? -- Dieter
participants (4)
-
Dieter Maurer -
Dylan Jay -
Dylan Jay -
Peter Bengtsson