[Zope-dev] [CRITICAL] Conflict Errors, Transactions, Retries, Oh My....

[Jeffrey P Shell]

On Wednesday, May 28, 2003, at 10:19  PM, Chris McDonough wrote:

>> This doesn't happen often, but (as stated), this is a critical
>> operation that needs to be better protected.  All other exceptions and
>> bits and pieces in the block of code in question has been tested
>> thoroughly and we have not had any other problems that cause erroneous
>> writes.  Is there a way I can protect against Conflict Error retries 
>> as
>> well?  Is there some sort of Try/Except or Try/Finally I can wrap
>> around the code that won't interfere with the ZODB?  Is there any 
>> other
>> sort of best-practice here that could help me (and others) who might
>> unknowingly trigger this problem?
>
> Not infallibly.   You can really never know where a ConflictError will
> might be raised.  Any concurrent access to a persistent object is a
> possible candidate.

Thanks for the information.  Is it safe at all to try to catch a 
ConflictError during the critical part of the code, log some 
information, and then reraise the error to let the system do what it 
needs?

I guess you're right though - it's hard to know when it will occur.

In the production system, in this particular method, there are only two 
known persistent object interactions.  At the end of the entire method, 
after a notification email has been sent, I have something like:

session['pieces'] = {}

(session['pieces'] was a dictionary of {item_id:integer} bits.  It 
never gets large for an individual user).  I think that the one recent 
case of desync'd data happened when we got to this point.  Since it's 
at the very end of the script (no more writes are expected beyond this 
point), I imagine that a get_transaction().commit() might be OK to 
precede this statement, just so that even if any conflicts happen when 
trying to write back to the session, we at least have synchronized data 
between the two systems.  Although, prior to this, there are a few 
reads of this session data.  Might it be safer to do something like 
this at the top of the method?:

pieces = session['pieces'].copy()

I apologize if this post is making little sense (or stupid sense) - 
dealing with threads, locks, conflicts, etc, has been the part of Zope 
I've understood the least.  I like that for the most part I don't have 
to think about it, but I don't know where to go for [fairly] current 
documentation on how to deal with it for those rare times I do.

The other persistent data write occurs earlier in the method, an object 
that generates serial numbers based off of some simple data in a 
PersistentMapping gets updated.  I think that PersistentMapping has 
become fairly large by now.  It maps the item_id referenced above to a 
regular dictionary containing three key/value pairs each.  I make sure 
to follow the rules of persistence when dealing with these 
dictionaries-with-a-PersistentMapping, but I'm guessing that an OOBTree 
might be better instead.  I still don't understand the potential 
pitfalls of Zope/ZODB BTrees (I keep reading about 'bucket splits' 
causing conflicts, and I don't know if that would be better or worse 
than any pitfalls a PersistentMapping gives).

Finally, the system in question has a few (three?  four?) public Zope 
sites using the same session storage.  Is there any documentation, 
notes, etc, about fine tuning the default session storage set up to 
handle large sites (or groups of sites) with less conflicts?

Thanks again for the help.  I'll take a look at MailDropHost.  Maybe 
I'll have to wrap another gateway around the gateway to the external 
system to try to catch these conflict situations.  Fortunately, the 
critical area only occurs once in the current copy of the code.  
Hopefully that will make it easier to protect.

Thanks again,
Jeffrey