[ZODB-Dev] Zope memory usage

[Roché Compaan]

Hi Alan

Thanks for the notice. We'll give this a go and report back.

Do you know how exactly it is decided what stays in the cache? 

-- 
Roché Compaan
Upfront Systems                   http://www.upfrontsystems.co.za

On Wed, 2008-09-17 at 10:26 -0500, Alan Runyan wrote:
> There was a recent modification to limit the ZODB cache to a set size.  i.e.
> Limit the size of memory usage to 128MB.
> 
> The original feature was implemented here:
>   http://svn.zope.org/ZODB/branches/dm-memory_size_limited-cache/
> 
> You can get the feature+3.8 branch of the ZODB from:
>   http://svn.zope.org/ZODB/branches/zcZODB-3.8/
> 
> The changes are also on trunk (will be ZODB 3.9).
> 
> The goal of the modification is to try to put upper bounds on the size of
> the database cache.  Before this "size limited cache" the cache would grow
> and if you had very large objects in the zodb cache -- your memory usage would
> be surpringsly high.
> 
> Please use this feature in your testing at upfront.  And let Roche know this
> feature has landed (if we was not monitoring svn checkins).  The feature needs
> to be test driven.  Give it a go and report your experience.
> 
> cheers
> alan
> 
> 
> On Wed, Sep 17, 2008 at 5:10 AM, Izak Burger <izak at upfrontsystems.co.za> wrote:
> > Hi all,
> >
> > I'm sure this question has been asked before, but it drives me nuts so I
> > figured I'll ask again. This is a problem that has been bugging me for
> > ages. Why does zope memory use never decrease? Okay, I've seen it
> > decrease maybe by a couple megabyte, but never by much. It seems the
> > general way to run zope is to put in some kind of monitoring, and
> > restart it when memory goes out of bounds. In general it always uses
> > more and more RAM until the host starts paging to disk. This sort of
> > baby-sitting just seems wrong to me.
> >
> > It doesn't seem to make any difference if you set the cache-size to a
> > smaller number of objects or use a different number of threads. Over
> > time things always go from good to bad and then on to worse. I have only
> > two theories: a memory leak, or an issue with garbage collection (python
> > side).
> >
> > It is possible that this is not a bug, my reasoning goes like this: The
> > OS exposes a "virtual memory" picture to the application, ie, the
> > application does not know how much of the RAM is real and how much is
> > paged out. All it does is call malloc every now and then. Combined with
> > a kernel that allows overcommit (which is in general a good thing), I
> > see this going only one way.
> >
> > I'm hoping someone on this list has some insights into the issue.
> >
> > regards,
> > Izak
> > _______________________________________________
> > For more information about ZODB, see the ZODB Wiki:
> > http://www.zope.org/Wikis/ZODB/
> >
> > ZODB-Dev mailing list  -  ZODB-Dev at zope.org
> > http://mail.zope.org/mailman/listinfo/zodb-dev
> >
> 
> 
>