[ZODB-Dev] Re: Can __setstate__ trigger an RCE?

[Christian Robottom Reis]

On Tue, Jul 06, 2004 at 04:48:56PM -0400, Tim Peters wrote:
> >>     def __setstate__(self, state):
> >>         # Load the _change_buffer to ensure we have its state and
> >>         # avoid a ReadConflictError when handling it
> >>         list(self._change_buffer.items())
> 
> [Casey Duncan]
> > This code is probably more likely to cause an RCE rather than prevent it,
> > because it tries to load *all* of the buckets of the BTree whenever the
> > BTree's container is loaded. If somebody else somewhere has modified the
> > btree in any way since this transaction started, then you're toast (even
> > if this transaction would never actually read the changed buckets of the
> > tree).
> 
> Well, Christian said _change_buffer is bound to an OOBucket, not to a BTree.
> If that's correct, maybe he'd be better off switching to a BTree (depends on
> how big _change_buffer gets; if it's typically "very small", it's going to
> live in a single bucket regardless).

It's an OOBucket, and it never grows very large -- it just holds objects
that have changed since we last updated the catalog indexes, which in
this application usually means less than five since we update the
indexes every commit in a try/except loop designed to work with the
fact that CEs may happen when updating the indexes.

I chose to use an OOBucket because (at least from memory) Tim suggested
they were the best data structure to avoid conflicts, since they would
never split/merge and only really conflict if the same key was added or
deleted at once -- which in this case is guaranteed never to happen
unless two people alter the same object simultaneously, which is already
a CE for other reasons.

However, I just realized that Tim's advice probably does not apply to
ReadConflictErrors, given that these occur when reading dirty bits of
the bucket, regardless of what key I'm actually going to change. A
side-question is if Buckets are stored as single persistent objects, and
if so, this makes sense (and IIRC they indeed have smart conflict
resolution code).

At any rate, removing the "preloading" call causes RCEs to happen a lot
more frequently -- it was added for a reason, you know <wink>. I'm
convinced this is something of a race condition when someone commits a
change to the bucket right after someone else has just finished
unpersisting their "copy" of it. But this suggests to me that MVCC is
the only solution worth investigating, since tracking down everywhere
the OOBucket is changed means tracking down everywhere IndexedObject
attributes are changed -- IOW, all over the system.

Take care,
--
Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 3361 2331