[ZODB-Dev] Re: Can __setstate__ trigger an RCE?

[Christian Robottom Reis]

On Tue, Jul 06, 2004 at 05:15:30PM -0400, Casey Duncan wrote:
> > On Tue, Jul 06, 2004 at 04:48:56PM -0400, Tim Peters wrote:
> [..]
> > It's an OOBucket, and it never grows very large -- it just holds
> > objects that have changed since we last updated the catalog indexes,
> > which in this application usually means less than five since we update
> > the indexes every commit in a try/except loop designed to work with
> > the fact that CEs may happen when updating the indexes.
> 
> Sounds like a classic hot-spot. I imagine it gets changed quite often
> then?

Oh, you know, just about every time an attribute in an instance (in a
system where there are around 500,000 of them) is changed <wink>

> > I'm convinced this is something of a race condition when someone
> > commits a change to the bucket right after someone else has just
> > finished unpersisting their "copy" of it. But this suggests to me
> > that MVCC is the only solution worth investigating, since tracking
> > down everywhere the OOBucket is changed means tracking down
> > everywhere IndexedObject attributes are changed -- IOW, all over the
> > system.
> 
> Can you explain the exact purpose of this bucket? Is it some kind of
> queue for deferred indexing? If so, you may be able to get away with
> making it independant and rely on (custom) write conflict resolution to
> handle all updates (ala QueueCatalog). This is (of course) a non-trivial
> exercise, however.

Yes, that's precisely what it is -- a queue for deferred indexing, quite
similar to QueueCatalog (in fact, it's implemented as something called
queued_mode in IndexedCatalog.Catalog). 

_p_independent, huh? I seem to recall this being evil, but let's have a
look at how QueueCatalog implements its conflict resolution to get an
idea of just how non-trivial you're implying.

Take care,
--
Christian Robottom Reis | http://async.com.br/~kiko/ | [+55 16] 3361 2331