[ZODB-Dev] BTree set question

[Chris McDonough]

On Thu, 2004-02-26 at 18:27, John Belmonte wrote:
> Well, I said that the rules are inconsistent, not that they will cause 
> an inconsistency in the database or something.  But I'll guess at your 
> meaning and give a simple example of why the mapping value modification 
> rule is dangerous.

Right, sorry for being imprecise.  I've always considered that the
actual problem caused by an "overaggressive" conflict resolution implied
some sort of data inconsistency.  Your example does prove that data can
become inconsistent as a result of CR or at least differ from the
programmer's intent.
 
> Take a mapping of words to the number of times they were encountered. 
> If transaction-A tries to increment count['idempotent'] by some amount, 
> and transaction-B increments it by some amount, we get a write conflict 
> as you'd hope-- except on the chance that they try to increment it by 
> the same amount.

This is a great, simple, practical example, thanks.  I can see how this
would be a problem for insertion of keys with the same value as well.  I
still don't quite understand why two transactions can't delete the same
key without generating a conflict, but maybe I just lack imagination.
;-)

> As Peter points out, you can probably accuse any such example of being 
> bad design.  In this case the design problem is obvious.  However, it 
> would be nice if we could write ZODB apps a little naively (or without 
> having enough time to attend to all details on the first pass), and as 
> the conflict errors crop up, design solutions for them.  This notion can 
> be supported by having conservative BTree merge rules.  Code that is bad 
> for ZODB, but otherwise perfectly "normal" if there wasn't concurrency, 
> should generate write conflicts if reasonably possible.

That makes sense to me.
 
I think the aggressiveness of the conflict resolution code can be traced
back to the fact that, for Zope, an unresolveable ConflictError is
extremely expensive.  A ConflictError is propagated up to the top-level
error handler, and when it is, Zope needs to retry the entire web
request that caused it to be generated.  It's also an issue that
WriteConflictErrors happen so late; a Zope request may do work that
takes several seconds to get to the point where its associated
transaction is ready to commit; doing all that work over again gets very
expensive.  I'm not saying that's an argument against being more
conservative, I'm just saying that was probably a factor in the decision
to make it less so.

It would be nice to be able to plug in a different policy, but I'm
pretty sure I'd just make a complete hash of things if I tried to make
that possible.  And I'm glad Tim stared at it long enough to make sense
out of the bucket_merge code; I wouldn't want to further punish him by
asking him to change it. ;-)  I'm also happy to work around the current
set of issues (now that I know them!) because I think it likely helps
Zope performance in highly concurrent situations.

- C