On Thu, Apr 22, 2010 at 1:19 PM, Jürgen Herrmann <Juergen.Herrmann@xlhost.de> wrote:
On Thu, 22 Apr 2010 12:54:55 -0400, Jim Fulton <jim@zope.com> wrote:
On Thu, Apr 22, 2010 at 9:00 AM, Jürgen Herrmann <Juergen.Herrmann@xlhost.de> wrote:
hi there!
today i ran a check script which iterates over approx 150000 objects and does some sanity check calculations on these. during this loop i saw the zope process use up to about 4.5gb memory. the database has ~3.5million objects in it.
i set the zodb cache size for the mount point in question to 10000 objects. obviously this limit is not honoured during one transaction:
Right.
so two questions here: - would the byte-limited zodb cache setting help here?
No. The limits aren't really limits. They are more like suggestions. :) In particular, they are only enforced at transaction (or subtransaction) boundaries, when a connection is opened or closed, or when applications call certain APIs explicitly. are these apis exposed somewhere? calling these periodically sounds cleaner than the aproach below.
- if no - how can i iterate over a big set of objects without forcing them to stay in the cache for the whole transaction? after all i just need each object once during the iteration.
As you're iterating through the objects, if you know you aren't going to need an object again or otherwise think it would be OK to free it from memory, you can call _p_deactivate on it. If the object hasn't been modified, then it's state will be released and it will become a ghost. as my objects contain references to OOBTrees, do i also have to call _p_deactivate() on these explicitly, or is it enough to call it on the containing object?
You need to call it on any objects you want to go away. Another option is to periodically call cacheGC() on the connection object: some_object._p_jar.cacheGC() This will apply the limits, to the degree it can. As with _p_deactivate, only unmodified objects will be removed from memory. I'm assuming in this discussion that you're only analyzing, not updating objects. Jim -- Jim Fulton