[ZODB-Dev] Potential BTrees splitting bug

[Tim Peters]

[Dieter Maurer]
> On one family of our servers, the test
> "ZEO.tests.testConnection.checkConcurrentUpdates2Storages"
> fails non-deterministically but with high probability.

Exactly how does it fail (depending on the exact version of ZODB you're
using, I realize it may not be easy to answer that question)?  It *can* fail
in many distinct ways (although it shouldn't fail at all, of course).

It's almost certainly the case that the problem isn't in the BTrees
splitting code.  checkConcurrentUpdates2Storages is a relatively recent
test, added to verify that some ZEO cache consistency bugs stay fixed.

> This is Dual 2GHz AMD, Debian Linux, Python 2.1.3, current Zope
> 2_6branch from CVS with my "No more ReadConflicts" patch installed.

The patch may be, or contribute to, the problem.  Could you please try it
without your patch too.

> I looked at about 40 failures. All followed the same pattern:
>
>   It was always the last key of a 17 element bucket that were
> unordered.

Sorry, I'm not clear on what that means.  Was the failure that a thread
believed it had added a key to the BTree but that the key wasn't actually
found there later?  Did the test produce any output when it failed (and if
so, can we see it -- the most recent versions of the tests do an exhaustive
dump of the BTree to stdout when they fail, which is most helpful)?

>   For me, it looked as if under some circumstances the last element of
> a splitting bucket were forgotten to be moved into the new bucket.
>
>
> The test passed always on Mono 1.5 Ghz AMD, SuSE Linux and otherwise
> identical parameters.

ZEO cache bugs are extremely sensitive to quirks of platform timing, so it's
not surprising that just changing boxes can change what you see.  For
example, the last batch of ZEO cache consistency bugs we fixed were
extremely easy to provoke on Win98SE(!), difficult to provoke on Win2K, and
seemingly impossible to provoke on Red Hat Linux, with comparable hardware
in all cases.