[Zope] Zope Eating Memory for Breakfast

[Hung Jung Lu]

>From: Monty Taylor <mtaylor@goldridge.net>

Thanks for the feedback. It's very helpful.

> > (4) ZODB: I don't have anything that dynamically changes Zope's
> > ZODB, that is, I don't upload files into ZODB (Do you?), I
>...
>       My site changes a bit more than that, but it's still fairly
>static. My web-content manager may post a new org-chart or internal news
>story, but we don't even have that many people connecting to read yet.
>
> > (5) Size of ZODB: my Data.fs is about 9 MB.
>
>      That would be 525Meg. My god, that's large. But I think that's
>mainly pictures.

Aha! There is the first clue. The memory increase has to do
with Zope ZODB operations.

Why does it have to do with ZODB operations?

(1) There are Zope sites out there running for a long time and
people aren't complaining about memory leak problems. These
include some heavily-visited sites.

(2) Everytime you restart Zope the memory usage starts out OK.

(3) The size of the problem is proportional to the size of
    Data.fs

(4) I played around a bit with my site, and when I am viewing
and accessing my site as an external user, the memory
usage did not go up.

(5) But when I use Zope management interface to do some
operations, like clicking on the Find tab (and then the
Find button in that page), the memory usage jumped. I
managed to raise the memory usage by 5 MB just by clicking
the Find tab/button all over places and search for
files containing a diverse text pattern.

Could you try with "Finding" some items in your website
by using Zope's "Find" feature, and let us know whether
it also raises the memory usage?

So at the moment the memory leak problem does not seem to
be coming from usage of products (like database adapter),
but from some of the management function calls in Zope.

-----------------------------------------------------------

In your case, I would recommend you using external
storage for images/files. And use a product like LocalFS
(see http://www.zope.org/Members/sabaini/externalfiles-howto )
to handle large files. You can also write your own external
methods to save/delete/update/upload/download the external
files. That should bring your memory usage drastically down
to megabytes instead of gigabytes.

So, in a sense, your problem can be solved (since you have
a few GB of RAM memory to play around... ha ha... I am not
that lucky.) It's a harder question how we are going to
pinpoint the exact source of the memory leak.

regards,

Hung Jung

______________________________________________________
Get Your Private, Free Email at http://www.hotmail.com