Re: [Zope] memory leak in a very simple product. Way?
[Thomas B. Passin]
[Christoph Wierling]
I've writte a very simple product which creates 100 simple objects (testProd class instances) and each of these objects has a list-attribute which contains 200 other objects (item class instances). (Please find the code at the end of this mail.) By starting the addTestProd-function I create 20000 references to item objects, as the Zope-debug option as well as the leakFinder-product tell me. But I'll never get rid of these references and earlier or later I'll run out of memory.
Could anybody explain me, way this simple product is leaking like a sieve? And how I can get rid of the references?
here is the code:
...
class testProd(SimpleItem.SimpleItem): """ testProd docu """
meta_type = 'testProd'
def __init__(self, id, title = ''): self.id = id self.title = title
self._tree = []
for i in xrange(200): n = item(str(i),i) tree = self._tree tree.append(n) self._tree = tree
I think you really want to do this:
tree = [] for i in xrange(200): n = item(str(i),i) tree.append(n) self._tree = tree
Thanks for the suggestion. Your code does about the same as my did - it also reproduces the 20000 references to objects of the item-class. And I also don't get rid of the 20000 references. I tried it! :-( christoph
Christoph Wierling wrote:
[Thomas B. Passin]
class testProd(SimpleItem.SimpleItem): """ testProd docu """
meta_type = 'testProd'
def __init__(self, id, title = ''): self.id = id self.title = title
self._tree = []
for i in xrange(200): n = item(str(i),i) tree = self._tree tree.append(n) self._tree = tree
The only thing I can guess at is that xrange plays funnily with Zope in this case. For some reason the list items generated by xrange() might not be released for garbage colletcion. If so it is probably a bug. Try: class testProd(SimpleItem.SimpleItem): """ testProd docu """ meta_type = 'testProd' def __init__(self, id, title = ''): self.id = id self.title = title self._tree = [] for i in range(200): n = item(str(i),i) self._tree.append(n) On another note. Why do you do all the : tree = self._tree tree.append(n) self._tree = tree isn't: self._tree.append(n) Just as good, and 2 lines shorter? regards Max M
I dont think I understand the issue. You create some persistent objects. You have references to those objects in a list. You store the list in a persistent object too. You restart and you still have references to those objects. This isn't a leak, AFAICT. ----- Original Message ----- From: "Christoph Wierling" <wierling@molgen.mpg.de> To: <tpassin@mitretek.org> Cc: <zope@zope.org> Sent: Wednesday, January 16, 2002 5:19 AM Subject: Re: [Zope] memory leak in a very simple product. Way?
[Thomas B. Passin]
[Christoph Wierling]
I've writte a very simple product which creates 100 simple objects (testProd class instances) and each of these objects has a
list-attribute
which contains 200 other objects (item class instances). (Please find the code at the end of this mail.) By starting the addTestProd-function I create 20000 references to item objects, as the Zope-debug option as well as the leakFinder-product tell me. But I'll never get rid of these references and earlier or later I'll run out of memory.
Could anybody explain me, way this simple product is leaking like a sieve? And how I can get rid of the references?
here is the code:
...
class testProd(SimpleItem.SimpleItem): """ testProd docu """
meta_type = 'testProd'
def __init__(self, id, title = ''): self.id = id self.title = title
self._tree = []
for i in xrange(200): n = item(str(i),i) tree = self._tree tree.append(n) self._tree = tree
I think you really want to do this:
tree = [] for i in xrange(200): n = item(str(i),i) tree.append(n) self._tree = tree
Thanks for the suggestion. Your code does about the same as my did - it also reproduces the 20000 references to objects of the item-class. And I also don't get rid of the 20000 references. I tried it! :-(
christoph
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Chris McDonough wrote:
I dont think I understand the issue. You create some persistent objects. You have references to those objects in a list. You store the list in a persistent object too. You restart and you still have references to those objects. This isn't a leak, AFAICT.
Oh then I misunderstood the problem. I assumed that the memory was still being used after the object had been deleted. Bugger. regards Max M
On Wed, 16 Jan 2002, Max M wrote:
Chris McDonough wrote:
I dont think I understand the issue. You create some persistent objects. You have references to those objects in a list. You store the list in a persistent object too. You restart and you still have references to those objects. This isn't a leak, AFAICT.
that's right, but I don't restart the server in between.
I assumed that the memory was still being used after the object had been deleted.
Well, I just deleted the persistent objects (lets call them A) which have a list attribute which have references to other persistent objects (lets call them B). So I expect that all B's will also be deleted as well, because the references from the A's do no longer exist. But as the debug info as well as the leak finder product tell me the 20000 references to the B's still exist. christoph
[Christoph Wierling] [me]
I think you really want to do this:
tree = [] for i in xrange(200): n = item(str(i),i) tree.append(n) self._tree = tree
Thanks for the suggestion. Your code does about the same as my did - it also reproduces the 20000 references to objects of the item-class. And I also don't get rid of the 20000 references. I tried it! :-(
I was about to ask if your item() objects are persistent, when I read Chris McDonough's post. If you are storing al 20,000 items, no wonder you see memory uasge increasing. Cheers, Tom P
On Wed, 16 Jan 2002, Thomas B. Passin wrote:
[Christoph Wierling]
[me]
I think you really want to do this:
tree = [] for i in xrange(200): n = item(str(i),i) tree.append(n) self._tree = tree
Thanks for the suggestion. Your code does about the same as my did - it also reproduces the 20000 references to objects of the item-class. And I also don't get rid of the 20000 references. I tried it! :-(
I was about to ask if your item() objects are persistent, when I read Chris McDonough's post. If you are storing al 20,000 items, no wonder you see memory uasge increasing.
Yes, the item() objects are persistent. And if I do recursive creation and deletion of the objects which contains the list with the persistent item() objects the references to the item objects are still growing. I just made the item objects not persistent, and it seems to work, so no growing references. So I wonder way the reference counter for persistent item objects is still growing by the recursive process of creation and deletion. christoph
[Christoph Wierling]
On Wed, 16 Jan 2002, Thomas B. Passin wrote:
I was about to ask if your item() objects are persistent, when I read
Chris
McDonough's post. If you are storing al 20,000 items, no wonder you see memory uasge increasing.
Yes, the item() objects are persistent. And if I do recursive creation and deletion of the objects which contains the list with the persistent item() objects the references to the item objects are still growing. I just made the item objects not persistent, and it seems to work, so no growing references.
So I wonder way the reference counter for persistent item objects is still growing by the recursive process of creation and deletion.
Well, maybe the persistence system has some other references to the objects that are less apparent, and don't get cleaned up. Or, if they are stored in the ZODB database, maybe it just needs to get packed after deletion. Cheers, Tom P
Christoph Wierling wrote:
Yes, the item() objects are persistent. And if I do recursive creation and deletion of the objects which contains the list with the persistent item() objects the references to the item objects are still growing. I just made the item objects not persistent, and it seems to work, so no growing references.
So I wonder way the reference counter for persistent item objects is still growing by the recursive process of creation and deletion.
If your item is subclassing persistent you should expect them to stay in the zodb. And then you should not add them to a list in your own product, but create a subclass of ObjectManager and use the _setObject() method to add the item instances. Else your item class should not subclass persistent. If you want to do it your way by adding item instances to a list in your product you should also be aware that every time your product is changed, or even one of the item instances, the whole object is written to the Data.fs again. Leading to a bloated Data.fs ... if you don't pack it regularly. But doing something like that with that many objects is the wrong way to use the zodb. regards Max M
If your item is subclassing persistent you should expect them to stay in the zodb. And then you should not add them to a list in your own product, but create a subclass of ObjectManager and use the _setObject() method to add the item instances.
This seems to be the solution of my problem ! :-) Thanks for your help, christoph
participants (4)
-
Chris McDonough -
Christoph Wierling -
Max M -
Thomas B. Passin