[ZODB-Dev] Re: Can __setstate__ trigger an RCE?

[Jeremy Hylton]

On Tue, 2004-07-06 at 17:21, Christian Robottom Reis wrote:
> On Tue, Jul 06, 2004 at 05:15:30PM -0400, Casey Duncan wrote:
> > > On Tue, Jul 06, 2004 at 04:48:56PM -0400, Tim Peters wrote:
> > [..]
> > > It's an OOBucket, and it never grows very large -- it just holds
> > > objects that have changed since we last updated the catalog indexes,
> > > which in this application usually means less than five since we update
> > > the indexes every commit in a try/except loop designed to work with
> > > the fact that CEs may happen when updating the indexes.
> > 
> > Sounds like a classic hot-spot. I imagine it gets changed quite often
> > then?
> 
> Oh, you know, just about every time an attribute in an instance (in a
> system where there are around 500,000 of them) is changed <wink>

So how often do you see conflict errors?  I don't recall from earlier in
the thread what percentage of transactions see a conflict.  It's hard to
avoid them entirely when you have concurrency in your application;
they're the mechanism for concurrency control after all.

> Yes, that's precisely what it is -- a queue for deferred indexing, quite
> similar to QueueCatalog (in fact, it's implemented as something called
> queued_mode in IndexedCatalog.Catalog). 

The logic for the queued catalog's conflict resolution is fairly
complex.  The last time I studied it in detail (maybe 9 months ago), we
concluded it had some bugs; I can't recall if they were fixed or not.

> _p_independent, huh? I seem to recall this being evil, but let's have a
> look at how QueueCatalog implements its conflict resolution to get an
> idea of just how non-trivial you're implying.

I sketched an alternative implementation for queue catalog that might be
worth trying (http://www.python.org/~jeremy/weblog/031031c.html).  The
idea is to use a set of "buckets," (not necessarily a BTree bucket, just
a container) where each client gets its own bucket.  The indexing thread
periodically drains an entire bucket of all its events, which could
cause a conflict, but you'd try to avoid two clients sharing the same
bucket.  The idea is an attempt to maximize scalability by avoiding
conflict resolution entirely.  Caveats: At least as complicated as
queued catalog.  Idea never tested in practice :-).

Jeremy