[Zope-dev] @#$%^&*( Zope Concurrency

[Phillip J. Eby]

At 06:39 AM 11/9/99 +1100, gtk wrote:
>Bait: I am willing to SPEND MONEY to get this bastard solved.

Me too.  That's why my company is paying DC for some consulting work in
this area, although at this point just to discuss possible solutions to
this and related subjects.


>- The retry mechanism doesn't work
>- The user sees extremely messy errors

These sounds to me like bug(s), which I'm sure the DC folks will fix once
they're sufficiently pinned down.


>- This seriously limits Zope's usability for write-intensive tasks like
>conferencing
>- Since all of my plans are for write-intensive tasks, that makes me
>uncomfortable

Ditto.  Ty Sarna and I have done some extensive analysis and testing of
ZODB and have found pretty much the same.  Any write-intensive data
structure will have conflict issues.  We came up with several ideas for
solutions, but they all require changes to the ZODB architecture, either to
allow a lock to be placed on a persistent object (which would be held to
the end of the transaction, and guarantee that the in-memory version for
that thread was up-to-date), to delay certain operations until a different
transaction phase, or to allow objects to be given a chance to resolve
conflicts by themselves (e.g. serialize writes that don't really conflict.)
 This is why we're consulting with DC; to find out which approaches are
most feasible.

(Side note: DC themselves have noted that currently they recommend that
people use external databases for write-intensive objects.)



>- Zope cannot deal with overlapping modifications to one object
>- That includes adding children to containers

Yep.  This also includes Catalogs, unfortunately.


>I've spent my last 24 hours of work on Zope trying to figure out where to
>hack in any kind of method-aware queueing which could -- at its most
>simple -- simply not attempt to place more than one call to an object at
>once. From there, I'd be able to hack the routine for more subtlety.

It's not quite that simple.  Technically, you'd need all the fundamental
ZODB interfaces (Connection, Database, and Storage) to support a lock()
operation, which would store the appropriate information at that level, and
be removed upon termination (successful or otherwise) of the transaction.
It's also probable that you'd need deadlock checking.