[Zope-dev] Very severe memory leak

[Leonardo Rochael Almeida]

On Mon, 2003-08-25 at 15:57, Shane Hathaway wrote:
> Leonardo Rochael Almeida wrote:
> > On Sat, 2003-08-23 at 22:18, Shane Hathaway wrote:
> >>When you flush the cache, those DateTimes should disappear.  If they 
> >>don't, the leak is keeping them.
> > 
> > They are disappearing. Too bad they return immediately after, as they're
> > comming from a very heavy catalog query result.
> 
> Hmm, that tends to indicate you really need that much memory.

If I just ask the client to put more memory in the machine, he'll
rightfully grill me because the site was working "less-worse" before the
upgrade. The fact remains that under 2.5.1 the leak would usually allow
the site to run for a number of hours before we needed to restart Zope. 

But maybe this means that the leak is not related to the DateTime
refcounts. It's just that the fast and continual increase in DateTime
objects is really bugging me. BTW, what is the usual DateTime refcount
in heavy ZCatalog sites you guys run?

Let me restate something important, because I forget about it myself
sometimes when I'm thinking about this problem: *upgrading to 2.6.1 made
the situation definetly worse*. The site became lot faster, but the
memory situation became intolerable, so I believe there is something
introduced between 2.5.1 and 2.6.1 that is causing our problems. Is
there a fix in 2.6.2 for some leak introduced *in* the 2.6 series?

Another thing, we have already estabilished that clearing the caches
resets the DateTime refcount, so the DateTime must be anchored to the
caches somehow. So, if the DateTime refcounts are constantly in the
50k-100k range, how come the cache object count for all threads is below
5k?

If they're indirectly attached, it means that a lot of DateTimes are
anchored in just a few objects, so no matter how much I reduce the
target cache size, it might never be enough to throw out the
DateTimes...

> [...]
> Maybe the individual ZODB objects are excessively large.  A few months 
> ago we found a bug in the new zope.org software that made it store file 
> uploads as a single chunk.  A lot of those files were over 1 MB in size. 
>   This exhausted 1 GB of RAM pretty quickly.  Splitting the files into 
> chunks, like Zope usually does automatically, solved the problem.

Nice to know, but we don't store large files or other objects.

> > it just plain monitor port?
> 
> I rolled my own remote console.  The monitor port is too buggy and Boa 
> is overkill for this.

hmm, any chance of this remote console showing up somewhere? :-)

I might use it to navigate the thread object cache.

-- 
Ideas don't stay in some minds very long because they don't like
solitary confinement.