On Sun, 2004-09-12 at 13:02, Dennis Allison wrote:
Chris --
Thanks for the assist. I think that I found the source of the problem last night and the the memory leak hypothesis was a cockpit error.
The problem manifested itself when running a find script which was doing an edit cleanup of a lot of stuff (thousands of ZUBBB discussions and ZUBB postings). One of the things the script did was to reset all the ZUBB methods to the default by copying in the half-dozen or so methods present in every instance of the discussions. Zope apparently treats the whole edit as a single transaction and buffers up the changes until it comes time to commit.
It's the developer's responsibility to manage memory during large transactions, yes.
Rewriting the edit script to do things a bit differently and only make changes when they are needed dropped the memory growth and time by a factor of eight or so. While not really anything I'd want to run execpt for the nonce to maintain the system, it did solve my immediate problem. The huge memory that got allocated did get released upon successful completion of the script (and when the script was aborted as well).
Yup.
Is there some mechanism to force a ZODB commit explicitly? If there were, then the edit script would have the structure:
find the thing to edit edit the thing commit the edits
and the resouce demands would be under control.
get_transaction().commit(1) commits a "subtransaction". This is used by, for example the Catalog, when doing batch reindexing. You might want to grep for it in the ZCatalog product. There is probably some surrounding hair to try to reduce memory consumption. Note that explicit transaction control is quite dangerous in Zope if used in the normal course of operations invoked by TTW users (for reasons too obscure to go into here: see recent posts to ZODB-dev). Using it in a script is typically fine though. - C