[Zope-dev] Memory leak, the resolution

[Leonardo Rochael Almeida]

On Thu, 2003-08-28 at 17:13, Shane Hathaway wrote:
> Leonardo Rochael Almeida wrote:
> > Let's assume we're not hitting a 2.6 bug. we could be hitting a 2.6
> > feature :-) I've read Casey's article about optimizing ZCatalog and I
> > know there has been a lot of ZCatalog optimization work. Has there been
> > any changes in 2.6 (in ZCatalog or thread cache or anywhere else) that
> > was a tradeoff of memory for speed?
> 
> Not that I know of.  To the contrary, I think--we sometimes can't do an 
> optimization because it would consume extra memory.

Well, as long as I can put an upper bound on the consumed memory, and
the tradeoff is reasonable, I'd be glad to trade memory for speed.

> > You mean that one ZCatalog object is in the cache, and (indirectly)
> > attached to it are thousands of DateTimes? Then we have a huge problem.
> 
> Why?  Although the catalog manages those DateTime objects, the DateTimes 
> are in many ZODB records.

The problem is: there should be a more direct relation between the
number of allowed objects in the cache and the number of refcounts. As
you say, the DateTimes are in ZODB Records, so if I see a DateTime
refcount of, say 120k, I should be seeing at least that many objects in
the caches, but I was seeing under 3k objects, unless there's something
else that could be counted as a refcount multiplying factor, say if
something was keeping a reference to the class for every instance
besides the instance itself.

> > I was counting on the "objects in cache" parameter to put an upper bound
> > in Zope memory consumption, even if this upper bound was a little fuzzy.
> 
> In the past, the cache size parameter was very fuzzy.  It was not an 
> upper bound.  Now, it is not only an upper bound, it also tends to act 
> as a lower bound.  (Note that the cache is still allowed to grow 
> indefinitely within the scope of a request, however.)


> > [..] but for now we need to deal with the emergency that is the fact that
> > migrating to 2.6.1 put us in the very hot spot of not having a way to
> > put an upper bound to Zope memory consumption.
> 
> I would characterize it the other way around; there was no way to 
> establish a hard boundary until now.  I suggest you reduce the ZODB 
> cache size to 2000 (or less) and see if the problem goes away.

The first time you sugested this I didn't even bother because we were
seeing our memory down the drain even while the caches where still at
1600 objects each (we had our limit at 5000).

But as we ran out of options, and seeing that flushing the cache got us
rid of the DateTimes. We decided to test the possibility of keeping Zope
thrashing its cache, even at the expense of performance, to see if it
could shed the DateTimes. 

So we set the cache limit to 1000 objects, and to our amazement, Zope
sat comfortably at 150MB RSS, and the performance was very good. Some
heavy ZCatalog pages still send Zope for a walk for a few seconds (for
those and all other requests) but the site becomes responsive again in
under 20 seconds. Not only that, but the DateTime refcount isn't even
among the top 5 (the BTrees are winning there now).

So we now have breathing space to do the great ZCatalog reform.

> > Anyone know off the top of their head how do I get to the object caches?
> 
> app._p_jar._cache will get you started.  It is dictionary-like.

Thanks, as soon as we get more RAM in the machine, I'll see if I can
switch the cache to a high number again and see where all those
DateTimes are hanging.

Cheers, Leo

-- 
Ideas don't stay in some minds very long because they don't like
solitary confinement.