[ZODB-Dev] Re: What makes the ZODB slow?

[Chris Withers]

Jean Jordaan wrote:
>> The ZODB is actually very fast. [...]
>>
>> So you're probably observing slowness in the frameworks on top of it.
> 
> I'll believe this anytime :-]

;-)

> In our case, a transaction may be a workflow state change on say 50 objects.
> Two or three people try a transaction like that within a couple of seconds
> of one another, and ConflictErrors crop up.

Ah, conflict errors, the bane of any ZODB app. Makes me wonder why other 
optimistic concurrency databases (I believe PostGres is one of these?) 
don't seem to exhibit the same problems...

> In a log with 402 ConflictErrors, 225 are on BTrees (_IIBTree.IITreeSet,
> _IOBTree.IOBucket, _OOBTree.OOBTree, _OOBTree.OOBucket all feature). We
> assume these all relate to catalog indexing.

Now, here's the crux of the matter. 99% of conflicts I see happen in the 
indexing stuff (ie: ZCatalog) and I really think these shouldn't. Search 
should always be accurate, but may be slow. I reckon if we could sort 
out a way that conflicts in indexing objects can better be dealt with, 
99% of the problem goes away..

> Of the rest, 147 of the 177 are either Products.Transience.Transience.Increaser
> or Products.Transience.Transience.Length2

Bleugh. Sessions are evil.

> The role the framework (Plone, unsurprisingly) is playing in this case, is
> that it leans hard on the catalog during a transaction lasting a number of
> seconds.

...and Plone is evil too, in this context because of the amount of 
extraneous indexing it does, both in terms of calls to catalog_object 
and in terms on the number and type of indexes and catalogs used...

> To mitigate this, we want to create a savepoint and then commit more often
> while iterating and changing workflow, rolling back to the savepoint if
> necessary.

savepoints won't help you with conflicts...

Chris

-- 
Simplistix - Content Management, Zope & Python Consulting
            - http://www.simplistix.co.uk