[Zope] Zope leaking memory?

[Chris McDonough]

On Sun, 2004-09-12 at 13:02, Dennis Allison wrote:
> Chris -- 
> 
> Thanks for the assist.  I think that I found the source of the problem
> last night and the the memory leak hypothesis was a cockpit error. 
> 
> The problem manifested itself when running a find script which was doing
> an edit cleanup of a lot of stuff (thousands of ZUBBB discussions and ZUBB
> postings). One of the things the script did was to reset all the ZUBB
> methods to the default by copying in the half-dozen or so methods present
> in every instance of the discussions.  Zope apparently treats the whole
> edit as a single transaction and buffers up the changes until it comes
> time to commit.

It's the developer's responsibility to manage memory during large
transactions, yes.

> Rewriting the edit script to do things a bit differently and only make 
> changes when they are needed dropped the memory growth and time by a
> factor of eight or so.  While not really anything I'd want to run execpt
> for the nonce to maintain the system, it did solve my immediate problem.
> The huge memory that got allocated did get released upon successful
> completion of the script (and when the script was aborted as well).

Yup.

> 
> Is there some mechanism to force a ZODB commit explicitly?  If there were,
> then the edit script would have the structure:
> 
> 	find the thing to edit
> 		edit the thing
> 		commit the edits
> 
> and the resouce demands would be under control.

get_transaction().commit(1) commits a "subtransaction".  This is used
by, for example the Catalog, when doing batch reindexing.  You might
want to grep for it in the ZCatalog product.  There is probably some
surrounding hair to try to reduce memory consumption.

Note that explicit transaction control is quite dangerous in Zope if
used in the normal course of operations invoked by TTW users (for
reasons too obscure to go into here: see recent posts to ZODB-dev). 
Using it in a script is typically fine though.

- C