[ZODB-Dev] Can __setstate__ trigger an RCE?

[Tim Peters]

[Christian Robottom Reis]
> I'm still trying to track down a problem that occurs with read conflicts
> in IndexedCatalog when using ZODB without MVCC enabled. The issue is that
> occasionally, I find in my client logs a conflict error on a OOBucket
> instance stored as _change_buffer in Catalog persistent instances. The
> interesting part is that the Catalog has the following code as part of
> its __setstate__ method:
>
>     def __setstate__(self, state):
>         # Load the _change_buffer to ensure we have its state and avoid
>         # a ReadConflictError when handling it
>         list(self._change_buffer.items())

I don't understand:  when did self._change_buffer get loaded?  Isn't that
the responsibility of this __setstate__ method?  Did you omit crucial parts
of this __setstate__ implementation?

> which should AFAIK load all the _change_buffer items and therefore avoid
> any potential conflict there.
>
> What's dogging me is that __setstate__ *itself* is generating RCEs, and
> this is something I don't understand. Shouldn't we be inherently
> protected from RCEs when __setstate__ is called, given that we're at a
> transaction boundary already?

I don't think so, but I don't know why you believe you're at a transaction
boundary.  __setstate__ gets called to materialize the state of a persistent
object, and loads routinely occur at times arbitrarily far removed from the
most recent transaction boundary.  In the absence of MVCC, RCE is expected
if an object O has been modified between the time the current transaction
began and the time O.__setstate__() is called.  For example,

1. a transaction ends
2. a new transaction (implicitly) begins then
3. someone else modifies an OOBucket (commits a change to it)
4. this transaction tries to do O.__setstate__, where O._change_buffer
   is the OOBucket someone else modified above
5. ReadConflictError is expected then:  in the absence of MVCC, ZODB
   won't let you load state that was modified since the transaction began

Any amount of time can pass between #2 and #3, and between #3 and #4.  "The
transaction boundary" is the instant between #1 and #2.