[Zope-dev] zope2/zodb cache size question

[Jim Fulton]

On Thu, Apr 22, 2010 at 1:19 PM, Jürgen Herrmann
<Juergen.Herrmann at xlhost.de> wrote:
>
> On Thu, 22 Apr 2010 12:54:55 -0400, Jim Fulton <jim at zope.com> wrote:
>> On Thu, Apr 22, 2010 at 9:00 AM, Jürgen Herrmann
>> <Juergen.Herrmann at xlhost.de> wrote:
>>>
>>> hi there!
>>>
>>> today i ran a check script which iterates over approx 150000 objects
>>> and does some sanity check calculations on these. during this loop
>>> i saw the zope process use up to about 4.5gb memory. the database
>>> has ~3.5million objects in it.
>>>
>>> i set the zodb cache size for the mount point in question to 10000
>>> objects. obviously this limit is not honoured during one transaction:
>>
>> Right.
>>
>>> so two questions here:
>>> - would the byte-limited zodb cache setting help here?
>>
>> No. The limits aren't really limits. They are more like
>> suggestions. :)  In particular, they are only enforced at
>> transaction (or subtransaction) boundaries, when a connection is
>> opened or closed, or when applications call certain APIs explicitly.
> are these apis exposed somewhere? calling these periodically sounds
> cleaner than the aproach below.
>>
>>> - if no - how can i iterate over a big set of objects without
>>>  forcing them to stay in the cache for the whole transaction?
>>>  after all i just need each object once during the iteration.
>>
>> As you're iterating through the objects, if you know you aren't going
>> to need an object again or otherwise think it would be OK to free it
>> from memory, you can call _p_deactivate on it. If the object hasn't
>> been modified, then it's state will be released and it will become a
>> ghost.
> as my objects contain references to OOBTrees, do i also have to call
> _p_deactivate() on these explicitly, or is it enough to call it on the
> containing object?

You need to call it on any objects you want to go away.

Another option is to periodically call cacheGC() on the
connection object:

  some_object._p_jar.cacheGC()

This will apply the limits, to the degree it can. As with _p_deactivate,
only unmodified objects will be removed from memory.  I'm assuming in this
discussion that you're only analyzing, not updating objects.

Jim

-- 
Jim Fulton