[Zope] Zope memory usage out of control

[Ron Bickers]

> -----Original Message-----
> From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of
> Navindra Umanee
> Sent: Monday, February 05, 2001 7:37 PM
> To: zope@zope.org
> Subject: [Zope] Zope memory usage out of control
>

> Please tell me that's really a single process with several threads all
> sharing memory? :-)

Linux gives each thread a process ID, so ps shows each thread as a process.
There's no way (that I know of) to tell from ps if they're threads or
processes, but since we know Zope is threaded, we know those are threads.
If you happen to run MySQL, you'll see the same thing.

> dot.kde.org is certainly a very busy site, is this the kind of memory
> usage to be expected from Zope?  Has anyone seen this kind of problem
> before?

I just recently started paying attention to Zope memory use when I noticed
my process using over 100MB for a virtually empty site.  I have since
discovered that a couple dozen 300k images that I have are causing it.

Apparently, Zope loads objects in memory in order to send them.  This means
that several large objects can eat up memory fast.  When I ran a test with
the ExtImage product, I don't get the Zope bloat I do when using Zope's
builtin Image object.

This is a huge problem for me since I have dozens of images in Zope that are
viewed *maybe* once every few days.  After the first viewer, Zope shoots up
in memory use and never goes back down.

> We're using Zope 2.2.1 with the hotfixes on a Red Hat Linux box.

I'm using Zope 2.3.0, but I don't think anything has changed in this
respect.

> Any hints short of switching to another news system appreciated (the
> SlashCode guys have been on our case). ;)

As Chris pointed out, it could be a memory leak.  However, if it goes up to
a large amount and hangs around there for a while, it sounds more to me like
there are some large objects being served (or many objects being cached???).
Others know far more about this than I, but this has been my experience.

I'm very interested in having Zope serve Image/File objects without having
to read them into memory first.  Running ExtFile/Image is a nice option
since it also stores the large files in the file system, but I don't mind a
large Data.fs as much as I mind having all of my memory eaten up for no good
reason.  Perhaps the future Image/File objects can optionally be stored
externally and served without the memory bloat.  Should I add a feature
request to the collector, or does DC already have a solution in mind for
this issue?

_______________________

Ron Bickers
Logic Etc, Inc.
rbickers@logicetc.com