[Zope] Zope leaking memory?

[Dennis Allison]

Chris -- 

Thanks for the assist.  I think that I found the source of the problem
last night and the the memory leak hypothesis was a cockpit error. 

The problem manifested itself when running a find script which was doing
an edit cleanup of a lot of stuff (thousands of ZUBBB discussions and ZUBB
postings). One of the things the script did was to reset all the ZUBB
methods to the default by copying in the half-dozen or so methods present
in every instance of the discussions.  Zope apparently treats the whole
edit as a single transaction and buffers up the changes until it comes
time to commit.

Rewriting the edit script to do things a bit differently and only make 
changes when they are needed dropped the memory growth and time by a
factor of eight or so.  While not really anything I'd want to run execpt
for the nonce to maintain the system, it did solve my immediate problem.
The huge memory that got allocated did get released upon successful
completion of the script (and when the script was aborted as well).

Is there some mechanism to force a ZODB commit explicitly?  If there were,
then the edit script would have the structure:

	find the thing to edit
		edit the thing
		commit the edits

and the resouce demands would be under control.


On Sun, 12 Sep 2004, Chris McDonough wrote:

> I've noticed what I believe are "request leaks" if any variety of CMF is
> involved in a request though I haven't tracked this down.  The symptoms
> are that every request causes the HTTPRequest class to have its
> reference count increased by one, plus a bunch of other objects are
> leaked because the request that was leaked apparently continues to hold
> on to references to them.
> 
> This may or may not be what you are seeing.
> 
> If you want to track something like this down, there is really no
> shortcut.  My strategy (which I've never successfully used to actually
> fix a leak, mind you! ;-) is this:
> 
> - Confirm that bare Zope doesn't leak by testing refcounts before
>   and after you load the quickstart page several times (remember
>   to clear the ZODB cache before you start hitting the site
>   whenever trying to test refcounts).  Look at the number of
>   references to all classes in the Debug tab.  These references
>   should not normally grow on every request.
> 
> - Do same for one of your "domain" objects.
> 
> If you see leakage when trying this against a method of one of your
> domain objects, a binary-exclusion method is usually the only way to pin
> down the leak.  Prevent half of the code that gets run when your domain
> object method is called from running.  If you still see the leak, the
> leak happens in the code you didn't prevent from running.  If you did,
> the leak happens in the code that ran.  And so forth.  It's tedious and
> time-consuming, which is probably why I've never successfully used it to
> fix a leak. ;-)
> 
> - C
> 
> 
> On Sun, 2004-09-12 at 01:33, Dennis Allison wrote:
> > Pound-current (V 1.7+)
> > Zope 2.6.4
> > Python 2.3.4
> > RH 7.3
> > Dual Athalon processors
> > 
> > Occasionally, I see Zope gathering very large amounts of real memory.
> > Normally buffering and the like seems to stabalize at around 500MB on a
> > 4GB memory dual processor machine, but occasionally grows to 2.5GB or
> > more.  At the same time, CPU usage tends to fully occupy one of the
> > processors for the threads holding all the memory.  Clearly there is a
> > problem.  Any hints as to where to track down the problem?
> > 
> >  
> > 
> > _______________________________________________
> > Zope maillist  -  Zope at zope.org
> > http://mail.zope.org/mailman/listinfo/zope
> > **   No cross posts or HTML encoding!  **
> > (Related lists - 
> >  http://mail.zope.org/mailman/listinfo/zope-announce
> >  http://mail.zope.org/mailman/listinfo/zope-dev )
> > 
>