[ZODB-Dev] Re: Can __setstate__ trigger an RCE?

[Casey Duncan]

On Tue, 6 Jul 2004 18:04:07 -0300
Christian Robottom Reis <kiko at async.com.br> wrote:

> On Tue, Jul 06, 2004 at 04:48:56PM -0400, Tim Peters wrote:
[..]
> It's an OOBucket, and it never grows very large -- it just holds
> objects that have changed since we last updated the catalog indexes,
> which in this application usually means less than five since we update
> the indexes every commit in a try/except loop designed to work with
> the fact that CEs may happen when updating the indexes.

Sounds like a classic hot-spot. I imagine it gets changed quite often
then?
 
> I chose to use an OOBucket because (at least from memory) Tim
> suggested they were the best data structure to avoid conflicts, since
> they would never split/merge and only really conflict if the same key
> was added or deleted at once -- which in this case is guaranteed never
> to happen unless two people alter the same object simultaneously,
> which is already a CE for other reasons.

Yup, they would be good for avoiding write conflicts.
 
> However, I just realized that Tim's advice probably does not apply to
> ReadConflictErrors, given that these occur when reading dirty bits of
> the bucket, regardless of what key I'm actually going to change. A
> side-question is if Buckets are stored as single persistent objects,
> and if so, this makes sense (and IIRC they indeed have smart conflict
> resolution code).

Buckets are really just parallel arrays of keys and values sorted by key
AFAIK.
 
> At any rate, removing the "preloading" call causes RCEs to happen a
> lot more frequently -- it was added for a reason, you know <wink>.

I figured it was worth a try...

> I'm
> convinced this is something of a race condition when someone commits a
> change to the bucket right after someone else has just finished
> unpersisting their "copy" of it. But this suggests to me that MVCC is
> the only solution worth investigating, since tracking down everywhere
> the OOBucket is changed means tracking down everywhere IndexedObject
> attributes are changed -- IOW, all over the system.

Can you explain the exact purpose of this bucket? Is it some kind of
queue for deferred indexing? If so, you may be able to get away with
making it independant and rely on (custom) write conflict resolution to
handle all updates (ala QueueCatalog). This is (of course) a non-trivial
exercise, however.

-Casey