[ZODB-Dev] High Write Applications

[Christian Reis]

On Mon, Aug 04, 2003 at 03:03:31PM -0400, Shane Hathaway wrote:
> Right.  I'm suggesting we make the back-end smarter.  ZCatalogs are the 
> best candidate for this.  When you call catalog_object() or 
> uncatalog_object(), the client would create a command object, add it to 
> the transaction queue, and execute that command locally without sending 

(What do you mean locally here? Local to the *client*? How/Why?)

> change events to the transaction.  Upon commit, the ZEO server would 
> receive that command object, execute (replay) it, and store object 
> states that are potentially different from what the client saw.
> 
> This would benefit operations that touch many objects.  Hot spots could 
> be moved into the storage server.

I've also toyed with the idea of moving IndexedCatalog indexing and
queries to the server-side. This would allow us to really work as a
database -- all query processing would be done server-side, and we could
do caching of search results, and so forth.

I'm wondering (blue-sky kind of thing) how hard this would be using the
current zRPC stuff.

> The main risk is that the ZEO server, now smarter, also has a much 
> greater chance of crashing or stalling on user code.  (The 
> application-level conflict resolution feature introduced that risk to a 
> lesser degree.)  I wonder if we'd have to split the storage server into 
> two processes, one of which we can feel free to restart on a whim.

That's definitely an idea -- one for processing specialized client
queries (in the *fast* backyard of the storage's filesystem), which
communicates with the main process (using what protocol?). Now which of
them would send data back to the client? ;)

Take care,
--
Christian Reis, Senior Engineer, Async Open Source, Brazil.
http://async.com.br/~kiko/ | [+55 16] 261 2331 | NMFL