[Zope-dev] Very severe memory leak

[Leonardo Rochael Almeida]

On Mon, 2003-08-25 at 23:35, Shane Hathaway wrote:
> On 08/25/2003 05:12 PM, Leonardo Rochael Almeida wrote:
> 
> > Let me restate something important, because I forget about it myself
> > sometimes when I'm thinking about this problem: *upgrading to 2.6.1 made
> > the situation definetly worse*. The site became lot faster, but the
> > memory situation became intolerable, so I believe there is something
> > introduced between 2.5.1 and 2.6.1 that is causing our problems. Is
> > there a fix in 2.6.2 for some leak introduced *in* the 2.6 series?
> 
> I think there are, actually, but AFAIK they were obscure and rare.

That would make the likelyhood of our problems being caused by them very
small. Is there any way to know if we're hitting them?

Let's assume we're not hitting a 2.6 bug. we could be hitting a 2.6
feature :-) I've read Casey's article about optimizing ZCatalog and I
know there has been a lot of ZCatalog optimization work. Has there been
any changes in 2.6 (in ZCatalog or thread cache or anywhere else) that
was a tradeoff of memory for speed?

If there was such a change, it would explain why we had a slow site that
did fit into memory while now we have a fast one that doesn't.

> > Another thing, we have already estabilished that clearing the caches
> > resets the DateTime refcount, so the DateTime must be anchored to the
> > caches somehow. So, if the DateTime refcounts are constantly in the
> > 50k-100k range, how come the cache object count for all threads is below
> > 5k?
> 
> If the DateTimes are in catalog metadata or indexes, this would make sense.
> 
> > If they're indirectly attached, it means that a lot of DateTimes are
> > anchored in just a few objects, so no matter how much I reduce the
> > target cache size, it might never be enough to throw out the
> > DateTimes...
> 
> They are probably in the ZCatalog.

You mean that one ZCatalog object is in the cache, and (indirectly)
attached to it are thousands of DateTimes? Then we have a huge problem.
I was counting on the "objects in cache" parameter to put an upper bound
in Zope memory consumption, even if this upper bound was a little fuzzy.


>  Remove all metadata fields you can. 
>   If you have index dates, use DateTimeIndexes rather than the standard 
> indexes.

Well, this is a classic ZCatalog+ZClass site. Almost all ZClasses are
CatalogAware and they're all cataloged in the monster ZCatalog named
"Catalog" in the folder that represents the root of the site (as served
by Apache) because that's how CatalogAware ZClasses work out of the box
(I know the catalog name can be changed). It has a huge amount of
Indexes and Metadata. The client grew the site organically, creating
ZClasses and hanging them in the same ZCatalog as he was needing them.

It's gonna take a lot of time to untangle it all and we're working on
it, but for now we need to deal with the emergency that is the fact tha
migrating to 2.6.1 put us in the very hot spot of not having a way to
put an upper bound to Zope memory consumption.

> I find it helpful to simply visit the information ZCatalog is storing 
> for just one cataloged object.  In the catalog contents view, click one 
> object.  Look at each field and consider how ZCatalog might misuse that 
> field for other objects.

What kind of misuse you have in mind?

> > hmm, any chance of this remote console showing up somewhere? :-)
> 
> I've attached what I threw together. 

Thanks!

> > I might use it to navigate the thread object cache.
> 
> It's unfortunate you have to go so deep, but that sounds like a good plan.

Anyone know off the top of their head how do I get to the object caches?

	Cheers, Leo

-- 
Ideas don't stay in some minds very long because they don't like
solitary confinement.