[Zope-dev] How bad _are_ ConflictErrors

[Chris McDonough]

On Nov 21, 2005, at 2:10 PM, Dennis Allison wrote:
>
> Conflicts and how they interact with the database and sessioning  
> machinery
> is my hot button right at the moment )-:  I Hope I have not
> included too much information.
>
> I ran a quick report and we see about 1000 conflicts per hour at
> about 120000 hits per hour.

Is this the number of log messages that indicate a conflict error  
occurred (e.g. "x conflict errors since DATE" messages in the event  
log) or the number of conflict errors that are retried more than  
three times and thus make it "out" to the app user?  I'm guessing the  
former.

>   These are order of magnitude numbers and are
> highly variable.  The 1% number is way bigger than I am comfortable  
> with
> although I have no basis to scale my expectations.  I'd be much  
> happier were
> it a couple of orders of magnitude smaller.

I would be too.  It's considerably difficult when ZODB is used as the  
sessioning backend.  A lot of effort has been put in to reducing the  
potential for conflicts already.  It could of course be better if  
more time was put in, but there hasn't been any reason (besides a  
sense of accomplishment and contribution to the greater good,  
anyway ;-) to put in that effort since the last time this machinery  
was overhauled.

That said, if no conflict errors actually bubble up to the user using  
the application, the penalty is "just" app performance and "knowledge  
expense" (e.g. you can't use a nontransactional mailhost, you can't  
use a nontransactional database table, etc).  You've already paid for  
the latter the hard way. ;-)  I can't judge the expense of the former  
to you but I assume that's what you're primarily worried about now.

>
> Conflict errors are not always errors.

The real reason they're called "errors" is only because they're  
implemented as Python exceptions.  They are implemented as exceptions  
because it was the easiest mechanism to use (exceptions are already  
built into Python).

>   As I understand it, Zope retries
> when a conflict occurs and usually is able to commit both sides of the
> conflicting transaction.

There can be more than two sides (actually there always are... there  
are three.. the two conflicting in-progress connection states and the  
database state).

>   Sometimes Zope cannot commit conflicting
> transactions--and it is at that point that an error occurs.

An exception occurs, yes.

Oops, I just realized Tim responded to the rest of these points, so I  
won't go on.

> We do have occasional instances where unresolved conflicts raise user
> visible diagnostics.  These are real errors.  While I have not  
> explored
> the reasons why, it appears that at least some of these errors are not
> logged in event.log but only displayed to the user.

To be pedantic, if you're right about conflict error tracebacks being  
shown to end users, it's not because they are "unresolved" (in the  
sense that 'application-level conflict resolution' could have  
prevented them), it's because a request was issued that resulted in a  
conflict error, which was retried, and then that retried request  
raised a conflict error, and then twice more.  The only way to figure  
out what's going on here is to see the traceback.  IIRC, Zope logs  
conflict error tracebacks at the BLATHER log level (as well as a  
deluge of other ancillary info).

However, even if BLATHER logging mode is not on, if no obvious error  
is put in the event log when a conflict error is relayed to a user,  
that's definitely a bug.  I'd believe it in a second! ;-)

The Zope conflict exception catching code is written in such a  
complicated way (and without the benefit of any automated tests) that  
tracking that down could take an entire day which I don't have to  
burn ATM.  So I'm afraid the status quo will prevail until someone  
gets so indignant about it that they either pay for it to be fixed or  
fix it themselves.  Apologies for that. :-(

- C