[Zope] [WARNING] Garbage collection may be disruptive

[Dieter Maurer]

knitti wrote at 2007-10-1 01:22 +0200:
> ...
>On 9/29/07, Dieter Maurer <dieter at handshake.de> wrote:
>>    Almost every non-trivial request (all requests not delivering
>>    static content) write to the ZODB. This means, we are
>>    using the ZODB in a high frequency write szenario -- one,
>>    it has not been designed for.
>
>as such as szenario might be relevant for some design decisions,
>could you tell, what "high frequency" roughly means?

Apparently, we have in the order of 10 writing transactions
per second during peak times.

>What would
>you implement differently, if you could start over?

Nothing. We like to stress Zope/ZODB and bring it to its boundaries.

However, we may need to look how we can scale our application
such that the load is spread over more than a single ZEO.
Thus, rather than 1 ZEO performing 10 writing transactions
per second, we may need 4 ZEOs, each performing about 2.5 writing
transaction per second...

Of course, we expect that the transaction number will grow and grow
and grow....


>if the gc has to remove many objects, perhaps you can use
>gc.set_treshold() to tune for a higher collection frequency. so each
>collection run might end up using less time.

Yes. That's what we did as an emergency measure.

Next step will be to get rid of the broken TextIndex and
replace it with one based on "Managable Index" with
"IncrementalSearch2" support.

Should then the garbage collection still make problems (not
to be expected in this application, but maybe in a different
one where we create lots of cyclic objects during XML processing),
then we will synchronize the garbage collection: delay, until
no request is processed or until no transaction is in progress.



-- 
Dieter