knitti wrote at 2007-10-1 01:22 +0200:
... On 9/29/07, Dieter Maurer <dieter@handshake.de> wrote:
Almost every non-trivial request (all requests not delivering static content) write to the ZODB. This means, we are using the ZODB in a high frequency write szenario -- one, it has not been designed for.
as such as szenario might be relevant for some design decisions, could you tell, what "high frequency" roughly means?
Apparently, we have in the order of 10 writing transactions per second during peak times.
What would you implement differently, if you could start over?
Nothing. We like to stress Zope/ZODB and bring it to its boundaries. However, we may need to look how we can scale our application such that the load is spread over more than a single ZEO. Thus, rather than 1 ZEO performing 10 writing transactions per second, we may need 4 ZEOs, each performing about 2.5 writing transaction per second... Of course, we expect that the transaction number will grow and grow and grow....
if the gc has to remove many objects, perhaps you can use gc.set_treshold() to tune for a higher collection frequency. so each collection run might end up using less time.
Yes. That's what we did as an emergency measure. Next step will be to get rid of the broken TextIndex and replace it with one based on "Managable Index" with "IncrementalSearch2" support. Should then the garbage collection still make problems (not to be expected in this application, but maybe in a different one where we create lots of cyclic objects during XML processing), then we will synchronize the garbage collection: delay, until no request is processed or until no transaction is in progress. -- Dieter