[Zope-dev] zope2/zodb cache size question

[Marius Gedminas]

On Thu, Apr 22, 2010 at 03:00:16PM +0200, Jürgen Herrmann wrote:
> today i ran a check script which iterates over approx 150000 objects
> and does some sanity check calculations on these. during this loop
> i saw the zope process use up to about 4.5gb memory. the database 
> has ~3.5million objects in it.
> 
> i set the zodb cache size for the mount point in question to 10000
> objects. obviously this limit is not honoured during one transaction:
> 
> connection                     active objects      total objects
> ...
> <Connection at 24c10410>       263104              462834
> ...
> 
> so two questions here:
> - would the byte-limited zodb cache setting help here?
> - if no - how can i iterate over a big set of objects without
>   forcing them to stay in the cache for the whole transaction?
>   after all i just need each object once during the iteration.

Use savepoints:

   for n, obj in enumerate(your_objects):
       perform_sanity_check(obj)
       if n % 10000 == 0:
           transaction.savepoint()

Real-world example:
http://bazaar.launchpad.net/~schooltool-owners/schooltool/schooltool/annotate/head:/src/schooltool/generations/evolve26.py

Marius Gedminas
-- 
http://pov.lt/ -- Zope 3 consulting and development
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
Url : http://mail.zope.org/pipermail/zope-dev/attachments/20100422/153a3c9a/attachment-0001.bin