[ZODB-Dev] Memory out of control when *NOT* changing objects

[Dieter Maurer]

Malcolm Cleaton wrote at 2004-10-12 15:22 +0100:
>I'm having a memory problem with a script for updating objects in the ZODB.
>
>I'm running on Zope 2.7.3b1, and invoking the script using zopectl run.
>
>The loop looks like this, although some method names have been
>changed to protect the innocent:
>
>results = portal_catalog.searchResults(...)
>num = 0
>for brain in results:
>	ob = brain.getObject()
>	if needsChanging(ob):
>		change(ob)
>	# else
>	# 	ob._p_deactivate()
>
>	num += 1
>	if num % 1000 == 0:
>		get_transaction().commit()
>
>With the code as shown, memory usage is fine if the objects actually need
>changing. However, if they don't, it spirals out of control, straight for
>swap death; I suspect the script may eventually finish but I don't have
>the patience.

This is unfortunately a problem with the current ZODB release.
Hopefully, it will get better with future ones:

  As you found out, your ZODB cache grows without limits.

  Usually, the ZODB connection performs a cache
  garbage collection at transaction boundaries.
  However, it can do this only, when it is informed about
  such boundaries.

  In the current ZODB (ZODB 3.2, at least), a ZODB connection
  is only informed about transaction boundaries when
  at least one of its objects registered with the transaction
  (because it was modified). If only reads are performed,
  the connection is not notified.

  This is a bug!


For my MVCC implementation, a connection must learn about each
transaction boundary whether or not one of its objects is modified.
Therefore, I force the connection to register itself with the
transaction. This way, it gets informed for every boundary.

Hopefully, future ZODB connections will do something similar...


You can work around this problem by calling "connection._incrgc()"
yourself after a transaction boundary.

If "obj" is a persistent object, you get its ZODB connection
via "obj._p_jar".

-- 
Dieter