[Zope] Zope memory usage out of control

[Chris McDonough]

Ron, it would be good if you could add a feature request to the Collector
about large file object streaming.  I'm not superfamiliar with the code
there, but I *think* the Image stuff is supposed to break the Image object
into smaller subobjects and load each one serially out of the ZODB when it
needs to serve them.  If it's not doing this properly, it needs to get
fixed.

----- Original Message -----
From: "Ron Bickers" <rbickers@logicetc.com>
To: "Navindra Umanee" <navindra@cs.mcgill.ca>; <zope@zope.org>
Sent: Monday, February 05, 2001 8:58 PM
Subject: RE: [Zope] Zope memory usage out of control


> > -----Original Message-----
> > From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of
> > Navindra Umanee
> > Sent: Monday, February 05, 2001 7:37 PM
> > To: zope@zope.org
> > Subject: [Zope] Zope memory usage out of control
> >
>
> > Please tell me that's really a single process with several threads all
> > sharing memory? :-)
>
> Linux gives each thread a process ID, so ps shows each thread as a
process.
> There's no way (that I know of) to tell from ps if they're threads or
> processes, but since we know Zope is threaded, we know those are threads.
> If you happen to run MySQL, you'll see the same thing.
>
> > dot.kde.org is certainly a very busy site, is this the kind of memory
> > usage to be expected from Zope?  Has anyone seen this kind of problem
> > before?
>
> I just recently started paying attention to Zope memory use when I noticed
> my process using over 100MB for a virtually empty site.  I have since
> discovered that a couple dozen 300k images that I have are causing it.
>
> Apparently, Zope loads objects in memory in order to send them.  This
means
> that several large objects can eat up memory fast.  When I ran a test with
> the ExtImage product, I don't get the Zope bloat I do when using Zope's
> builtin Image object.
>
> This is a huge problem for me since I have dozens of images in Zope that
are
> viewed *maybe* once every few days.  After the first viewer, Zope shoots
up
> in memory use and never goes back down.
>
> > We're using Zope 2.2.1 with the hotfixes on a Red Hat Linux box.
>
> I'm using Zope 2.3.0, but I don't think anything has changed in this
> respect.
>
> > Any hints short of switching to another news system appreciated (the
> > SlashCode guys have been on our case). ;)
>
> As Chris pointed out, it could be a memory leak.  However, if it goes up
to
> a large amount and hangs around there for a while, it sounds more to me
like
> there are some large objects being served (or many objects being
cached???).
> Others know far more about this than I, but this has been my experience.
>
> I'm very interested in having Zope serve Image/File objects without having
> to read them into memory first.  Running ExtFile/Image is a nice option
> since it also stores the large files in the file system, but I don't mind
a
> large Data.fs as much as I mind having all of my memory eaten up for no
good
> reason.  Perhaps the future Image/File objects can optionally be stored
> externally and served without the memory bloat.  Should I add a feature
> request to the collector, or does DC already have a solution in mind for
> this issue?
>
> _______________________
>
> Ron Bickers
> Logic Etc, Inc.
> rbickers@logicetc.com
>
>
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://lists.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://lists.zope.org/mailman/listinfo/zope-announce
>  http://lists.zope.org/mailman/listinfo/zope-dev )
>