[ZODB-Dev] High Write Applications

[Shane Hathaway]

Christian Reis wrote:
> On Mon, Aug 04, 2003 at 03:03:31PM -0400, Shane Hathaway wrote:
> 
>>Right.  I'm suggesting we make the back-end smarter.  ZCatalogs are the 
>>best candidate for this.  When you call catalog_object() or 
>>uncatalog_object(), the client would create a command object, add it to 
>>the transaction queue, and execute that command locally without sending 
> 
> 
> (What do you mean locally here? Local to the *client*? How/Why?)

Local to the client, meaning that the client makes the changes 
immediately.  This is so that future calls to searchResults() within the 
same transaction use updated indexes.  Unfortunately it means that the 
work of updating indexes happens twice (once on the client and once on 
the storage server.)  Ideally, it should happen twice only in the event 
of a conflict.

>>change events to the transaction.  Upon commit, the ZEO server would 
>>receive that command object, execute (replay) it, and store object 
>>states that are potentially different from what the client saw.
>>
>>This would benefit operations that touch many objects.  Hot spots could 
>>be moved into the storage server.
> 
> 
> I've also toyed with the idea of moving IndexedCatalog indexing and
> queries to the server-side. This would allow us to really work as a
> database -- all query processing would be done server-side, and we could
> do caching of search results, and so forth.
> 
> I'm wondering (blue-sky kind of thing) how hard this would be using the
> current zRPC stuff.

It wouldn't be very hard at all.  The harder part is deciding whether 
there's any actual benefit. ;-)

>>The main risk is that the ZEO server, now smarter, also has a much 
>>greater chance of crashing or stalling on user code.  (The 
>>application-level conflict resolution feature introduced that risk to a 
>>lesser degree.)  I wonder if we'd have to split the storage server into 
>>two processes, one of which we can feel free to restart on a whim.
> 
> 
> That's definitely an idea -- one for processing specialized client
> queries (in the *fast* backyard of the storage's filesystem), which
> communicates with the main process (using what protocol?). Now which of
> them would send data back to the client? ;)

The main process.  I was thinking the second process would be almost a 
ZEO client.

Shane