Re: [Zope] Zope Eating Memory for Breakfast
From: Monty Taylor <mtaylor@goldridge.net>
Thanks for the feedback. It's very helpful.
(4) ZODB: I don't have anything that dynamically changes Zope's ZODB, that is, I don't upload files into ZODB (Do you?), I ... My site changes a bit more than that, but it's still fairly static. My web-content manager may post a new org-chart or internal news story, but we don't even have that many people connecting to read yet.
(5) Size of ZODB: my Data.fs is about 9 MB.
That would be 525Meg. My god, that's large. But I think that's mainly pictures.
Aha! There is the first clue. The memory increase has to do with Zope ZODB operations. Why does it have to do with ZODB operations? (1) There are Zope sites out there running for a long time and people aren't complaining about memory leak problems. These include some heavily-visited sites. (2) Everytime you restart Zope the memory usage starts out OK. (3) The size of the problem is proportional to the size of Data.fs (4) I played around a bit with my site, and when I am viewing and accessing my site as an external user, the memory usage did not go up. (5) But when I use Zope management interface to do some operations, like clicking on the Find tab (and then the Find button in that page), the memory usage jumped. I managed to raise the memory usage by 5 MB just by clicking the Find tab/button all over places and search for files containing a diverse text pattern. Could you try with "Finding" some items in your website by using Zope's "Find" feature, and let us know whether it also raises the memory usage? So at the moment the memory leak problem does not seem to be coming from usage of products (like database adapter), but from some of the management function calls in Zope. ----------------------------------------------------------- In your case, I would recommend you using external storage for images/files. And use a product like LocalFS (see http://www.zope.org/Members/sabaini/externalfiles-howto ) to handle large files. You can also write your own external methods to save/delete/update/upload/download the external files. That should bring your memory usage drastically down to megabytes instead of gigabytes. So, in a sense, your problem can be solved (since you have a few GB of RAM memory to play around... ha ha... I am not that lucky.) It's a harder question how we are going to pinpoint the exact source of the memory leak. regards, Hung Jung ______________________________________________________ Get Your Private, Free Email at http://www.hotmail.com
----- Original Message ----- From: Hung Jung Lu <hungjunglu@hotmail.com>
(5) But when I use Zope management interface to do some operations, like clicking on the Find tab (and then the Find button in that page), the memory usage jumped. I managed to raise the memory usage by 5 MB just by clicking the Find tab/button all over places and search for files containing a diverse text pattern.
If you Find from the root, then check memory, then Find again, does it go up after the second Find? I would expect it to go up after the first, since Find walks the entire subtree loading objects into memory in order to check them against the search condition. The second time, they should all be in memory (or the cache should be full) and no more growth should occur (I think). Cheers, Evan @ digicool & 4-am
Hung Jung Lu wrote:
Aha! There is the first clue. The memory increase has to do with Zope ZODB operations.
Why does it have to do with ZODB operations?
I'm not certain how you conclude this, I don't see specific evidence toward this.
(1) There are Zope sites out there running for a long time and people aren't complaining about memory leak problems. These include some heavily-visited sites.
Exactly.
(2) Everytime you restart Zope the memory usage starts out OK.
(3) The size of the problem is proportional to the size of Data.fs
The size of the Data.fs file has no correlation to the amount of memory consumption.
(5) But when I use Zope management interface to do some operations, like clicking on the Find tab (and then the Find button in that page), the memory usage jumped. I managed to raise the memory usage by 5 MB just by clicking the Find tab/button all over places and search for files containing a diverse text pattern.
This is because Find activated objects out of the database to see if they match your criteria. As it activates objects, it puts them in the object cache. The object cache holds up to N objects, not up to X megabytes, so if you have large objects, your memory usage will go up noticably. Using Find tends to fill *up* the object cache. After using a find operation, your memory usage will go up. But I bet if you keep finding an finding and finding, it will not go up indefinatly. Increased memory usage does not neccesarily mean there is a leak, it might just need that memory. -Michel
participants (3)
-
Evan Simpson -
Hung Jung Lu -
Michel Pelletier