[Zope] Re: ZEO with Multiple Storages

[Dieter Maurer]

Brian Brinegar wrote at 2006-9-14 14:49 -0400:
>> > One ZEO server can deal with multiple storages just fine.
>> > Transactions involving multiple storages commit just fine.
>>
>> Caveat: You buy nothing by having a single ZEO process serve several
>> ZODBs when you're trying to decrease commit times. You should run a
>> ZEO process per ZODB you're serving out, that way the writes can be
>> segregated and parallelized. That also makes it easier to move
>> processes/databases around disks/hosts to spread I/O load.
>
>Just to clarify, you are saying that a single ZEO server would not be
>able to deal with multiple storages simultaneously and that a
>transaction in one storage would block transactions in all other storages.

Jens might be saying this -- but it is not (completely) true:

   A transaction is not an atomic operation for the ZEO server.
   Therefore, other transactions can proceed.

   A ZODB transaction commit consists of the following stages:

     tpc_begin	      # the first transaction stage begins
     ... commit all modified object ...
     vote	      # first stage finishes
     tpc_finish	      # second stage finishes

   ZEO processes these stages as follows:

     tpc_begin -- a commit log is created
     ... commit ... -- the new state is stored in the commit log
     vote -- ZEO acquires a storage look -- no further commits for this storage possible
             storage "tpc_begin" called
	     objects from commit log stored
	     storage "vote" called
     tpc_finish -- storage "tpc_finish" called; storage lock released


Jens is right that ZEO is (essentially) a single threaded server.
When it processes a request other requests must wait until this
request is finished. Using multiple ZEO servers will increase parallelism.


However, a transaction usually consists of many requests to
a ZEO server and between such requests ZEO is happy to perform
other requests even those for other transaction commits.

Only once "vote" started, a storage is "commit" blocked until the following
"tcp_finish|abort". Other storages can be processed in a normal
way; they can even commit transactions.


By the way: for large transactions "vote" may take a long time
(and prevent other requests from being processed).

That's why I implemented "vote" in our (private) Zope copy
as a "long running request". Such requests are run in a
separate thread (ZEO is not really a single threaded server --
it is only that most commands are run in the main thread)
and allow other requests to be processed in the meantime.



-- 
Dieter