[ZODB-Dev] Re: BTrees strangeness (was [Zope-dev] Zope2.XBIGSession problems - blocker - our site dies - need helpofexperienceZope developer, please)

[Tim Peters]

[Chris McDonough]
> ...
> In Alex's case (the failing case), the ultimate consumer of this code
> is the Zope web publishing code, which does indeed pull the root
> persistent object and thus all of its subobjects (the "self" in my
> code being one of them) out of a database on a per-thread basis as
> opposed to operating on a shared global root.

That should be good enough indeed.  I agree we're left with A Mystery then.

> ...
> A snippet of logging data from code that exercises this via multiple
> threads by printing your suggestion at the top of _gc:
>
> thread.get_ident(): 81926 ; id(self._data): 1086726400
> thread.get_ident(): 65541 ; id(self._data): 1086703784

That's even more convincing than your word <wink>.

...

>> You understand Zope better than I do, so you're much closer to the
>> solution: just explain all the relevant hidden details of Zope, and
>> I'll give you a hundred ways that will work <wink>.

> Right.  Well, given the above output, is that enough to convince you
> that I probably shouldn't need a mutex here?

Yes it is -- thanks.

So what are we left with?  Cases of "spontaneous corruption" are usually
pinned on pilot error, but since you're using an IxBTree flavor that's hard
to swallow.  Other cases have been pinned on the BTree, ZODB, and ZEO
implementations.  We haven't managed to blame any on the BTree
implementation in well over a year, and that's been subjected to extreme
stress testing since then.  Several subtle timing holes have been plugged in
both ZODB and ZEO since then.  Which version of ZODB is in use, BTW?  I hope
it's the most recent of whatever flavor is involved, else there's no mystery
worth pursuing.  Your idea to try using FileStorage instead was stellar,
since that's been by far the most heavily tested in former
corruption-provoking ZODB and ZEO stress tests.

Something that could be very helpful is to add

     from BTrees.check import check, display
     ...

     try:
         self._check()
         check(self._data)
     except AssertionError:
         display(self._data)
         raise

That will show the internal structure of the BTree if it's damaged.  The
last several cases of corruption due to timing holes in invalidation
invariably resulted in BTrees with one bucket ending with something like

    ... 31 32 33 45 46 47

and then thn next bucket starting with something like

    34 35 36 37 38 39 ...

The check() function complains about the 45 46 47 in the first bucket then,
because they're larger than the 34 that starts the second bucket.  This can
happen when a bucket splits, and invalidation doesn't manage to force new
copies of all of {bucket that split, the other bucket it split into, the
parent node of the bucket that split} to get loaded.  If you see something
like that again, it will make Jeremy's day <wink>.