RE: [Zope-dev] XMLDocument appears to leak memory
David Kankiewicz wrote:
Martijn Faassen wrote: I think I've seen an earlier report and I'd like to confirm this. XML Document appears to leak memory pretty badly:
* When rendering it with DTML to HTML, Zope's memory use often goes up a few kilobytes. Not all the time, it appears, but a reload often does the trick.
* It also appears that doing a DOM manipulation (for instance appendChild) increases Zope's memory use significantly; by a few 10s of k.
I've been working with the Zope Quick reference in XML and have found Zope's internal XML Documents are on about 10 times the size compared to stand alone file. (my guess, seems to be the DOM information?). With the manipulation, it might be the storage required to undo (not sure?).
Well, I suppose that it is possible there are memory leaks in XML Document, however I'd like to see stronger evidence before I jump to that conclusion. XML Document uses a lot of resources. DOM creates a lot of objects, and the estimate that a Zope DOM representation of an xml file is 10 times as large as the file is probably a reasonable estimate. Displaying an XML Documents wakes up quite a few objects. Editing XML Documents can create quite a few new objects since it is difficult for XML Document to know when to change existing Nodes and when to create entirely new Nodes. It is on my todo list to improve memory use in XML Document. For example steps can be taken to move temporarily needed objects out of memory. If anyone has more specific problems or reproducible cases of memory getting out of control, I'd love to hear them. Please remember, however, that XML Document is unsupported; I just work on in my spare time. Thanks! -Amos P.S. Another version of XML Document is coming out today. It is a *lot* better than the first alpha release.
Amos Latteier wrote: Ah, Amos, you're back! I hope you had a nice vacation -- us XMLers missed you. :) [snip]
Well, I suppose that it is possible there are memory leaks in XML Document, however I'd like to see stronger evidence before I jump to that conclusion.
Well, my Zope process grows by 10 megs in a few hours, by just using XML Document and the management interface. It's of course possible some of my helper external methods leak memory. I'll work on a test case either today or next week.
P.S. Another version of XML Document is coming out today. It is a *lot* better than the first alpha release.
I've been pointing everybody to use that one from CVS (and sending out demos to a multitude of people) for a while now. :) It's indeed a *lot* better. Regards, Martijn
Martijn Faassen wrote:
Amos Latteier wrote:
Well, I suppose that it is possible there are memory leaks in XML Document, however I'd like to see stronger evidence before I jump to that conclusion.
Well, my Zope process grows by 10 megs in a few hours, by just using XML Document and the management interface. It's of course possible some of my helper external methods leak memory. I'll work on a test case either today or next week.
I've done some more testing. Rendering from XML doesn't seem to be the problem, generally; though I get some inconclusive results (especially when acquisition seems to be involved, I'll try to figure out more details on that). Sometimes it adds a few k, but usually the process doesn't seem to leak a lot of memory. There seems to be a difference from what happens with my python test script that calls urls (little leaking of memory) and when I call it through the browser (this is multiframe so some other frames are called as a side effect -- some kilobytes appear to get leaked in some requests). I don't really understand this. However, the big leaks is when I do a transaction that changes the DOM (I do a removeChild, appendChild and a createTextNode in an external method). Calling this same external method multiple times in a single request does not seem to increase memory leaking. The memory leak here is significant though; in the order of a 100k per request, often. I can easily grow my procses by 10 megs this way, by repeated requests. I realize that this is still unclear, and I'll try to come up with something small that reproduces the leaks, if this information doesn't help. Incidentally, I think my testing has revealed a bug in the ExternalMethod product. Sometimes, but not consistently, I get an error at line 248 in ExternalMethod.py. I think this happens more often when there's multiple threads trying to get at the same document. I posted an example traceback to the collector; after more occurances I can add that it's always this line 248 that raises an exception. Regards, Martijn
participants (2)
-
Amos Latteier -
Martijn Faassen