[ZODB-Dev] Re: ZEO and Twisted

[Jim Fulton]

Rodrigo Dias Arruda Senra wrote:
> You probably already know all of these, but just in case ...
> 
> [ Jim Fulton ]:
> ---------------
> |
> |  I am strongly against linking ZEO to an application's main loop.
> 
> Twisted performs better if the protocols build upon it rely on its
> reactor (asynchronous loop). That means breaking long computations
> to use promises (deferreds in Twisted vocabulary), therefore increasing
> the interleaving of data handlers.
 >
> Implementing ZEO in Twisted would probably isolate the zeo protocol
> from the underlying transport (tcp, udp, whatever). OTOH, the
> implementation would be dependent on the typical deferred patterns
> (only common in Twisted nowadays). Twisted also support threads, but
> that defeats most of the other benefits of using Twisted in the first
> place (like not bothering about critical sections and other synch
> stuff), so they are just used when blocking calls must be made.

I'm familiar with how Twisted works,  ZEO currently uses asyncore,
which is similar in philosophy.  They are both based on the reactor
pattern, which doesn't really fit ZODB very well.  There are operations
that can take a long time.  In Zope, we got around this by interfacing
a separate thread pool with the asyncore main loop.

ZODB will never use Twisted's computational model internally.  Forget
it.  It's not on the table.  It's not even in the same room.

That's not to say that I'd be opposed to leveraging Twisted for
ZEO's networking.  This would be in a dedicated thread just for
this I/O.  This would be separate from any main loop the application
might run.

> 
> |  If twisted supports using multiple independent main loops, then it
> | would be an option.
> 
> I know it supports multiple implementations of reactors, each
> specialized for a particular purpose, for instance to conciliate
> two different loops into a single reactor, which is the case of
> twisted_networking_loop + GUI_event_loop.
> Therefore, it would be possible (my educated guess) to create a
> zeoreactor, would that suffice ?  
> 
> That was done to marry Twisted to: gtk, gtk2, glib2, wxPython,
> win32event loops.

I would want to use a dedicated loop just for ZEO's networking.
In fact, if an application used multiple ZEO storages, it would have
a separate loop for each storage.  This would be in addition to
any main loop the application might use for it's own purposes.

...


> [ David Pratt ]:
> -----------------
> |  > Is there is any strong opposition to using the twistd daemon for a 
> |  > twisted zeo service?
> 
> I volunteer to help in that effort to the best of my modest skills
> and limited (just like everybody else) spare time.

Subject to what I've said above, that would be great some time
in the future.  I'd be happy to have Twisted as an option.
But before I'd consider that, there really needs to be a more
basic refactoring of the ZEO networking architecture.

Jim

-- 
Jim Fulton           mailto:jim at zope.com       Python Powered!
CTO                  (540) 361-1714            http://www.python.org
Zope Corporation     http://www.zope.com       http://www.zope.org