[Zope-dev] Re: Event & Timer Service for Zope 2.8

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Dylan Jay wrote:
> Chris McDonough wrote:
> 
>> On Sat, 2005-01-29 at 05:49, Andrew Veitch wrote:
>>
>>> I noticed there was a thread last month on an event service for
>>> Zope.  Did that reach any conclusion?
>>>
>>> From what I can see the present products are:
>>>
>>> - http://www.zope.org/Members/lstaffor/Xron - Zope 2.6 only, not
>>> being  developed
>>> - http://zope.org/Members/naumen/NauScheduler - shipped with one of
>>> the  Zope CMS's works fine with 2.7
>>> - http://dev.legco.biz/products/ZopeScheduler and 
>>> http://dev.legco.biz/products/timerservice - works fine with 2.7 too
>>> - http://cvs.zope.org/Products/Event/ - event service without a time 
>>> service, not sure of development status
>>> - http://mail.zope.org/pipermail/zope-cmf/2005-January/021747.html - 
>>> Dieter's project to be based on Python's sched module
>>> -  http://api.nuxeo.org/CPS3/public/
>>> CPSCore.EventServiceTool.EventServiceTool-class.html - CPS's event 
>>> service
>>> -  http://cvs.sourceforge.net/viewcvs.py/*checkout*/mailmanager/
>>> mailmanager/MailCheckThread.py?rev=1.15 - our custom scheduler for 
>>> MailManager
>>> - Zope X3's event service
>>
>>
>>
>> It has proven (at least for me) to be useful to make the distinction
>> between "event service", "scheduler", and "clock" in the past.  I tend
>> to think of each this way:
>>
>> Event Service:
>>
>> Framework software that allows objects to communicate between each other
>> indirectly by sending and receiving "events".  An event service's
>> responsibility is only to receive and send messages.  Note that it is
>> not within the scope of an event service's responsibility to run tasks
>> at timed or scheduled intervals.
>>
>> Scheduler:
>>
>> Software that ensures tasks may be run at timed or scheduled intervals.
>>
>> Clock:
>>
>> Software that intermittently "pokes" Zope, telling it to wake up and "do
>> something" every so often.  A scheduler needs a clock.  A clock can be
>> implemented either as an external source (e.g. wget every 10 seconds) or
>> internally (by coopting the asyncore loop).
>>
>> Schedulers usually require clocks, but neither clocks nor schedulers
>> require event services.  Similarly, event services require neither a
>> clock nor a scheduler.
>>
>>
>>> It would be great to some consolidation of all of this into the core.
>>> I  would also strongly favour a timer service along with the event 
>>> service. I know that the argument has always been that operating 
>>> systems provide time services but this is not very convenient for
>>> cross  platform development and on some Windows platforms it's
>>> difficult.
>>
>>
>>
>> If we use the definitions above, I think what you want is a "scheduler"
>> and a "clock" to be available for Zope, but not necessarily an event
>> service.  Don't want to be pedantic about it, but when you use the term
>> "event service" it tends to mean different things to different people.
>>
>> To that end, I'd suggest using the Scheduler product at
>> http://cvs.zope.org/Products/Scheduler.  Note that this product *does*
>> currently depend on the "Event" product at
>> http://cvs.zope.org/Products/Event but it's only by accident (the Event
>> product is usable separately).  There is an external clock
>> implementation that ships with the Scheduler product.
>>
>> I agree with you that there should be a Zope "internal" clock.  I
>> created one not too long ago, but it's a bit invasive.  Read
>> http://www.plope.com/Members/chrism/more_scheduling and
>> http://www.plope.com/Members/chrism/scheduling_service for an overview
>> of the strategy I was thinking of when I did it.
>>
>> Personally, I think maybe the internal clock service can also be
>> implemented as a Product and not added to Zope proper.  It would be
>> great if someone were to maybe volunteer to take the work that I did
>> towards it and productify it.
> 
> 
> +1 for using http://cvs.zope.org/Products/Scheduler
> I think the time of a non-core scheduler needs to come to an end. Its
> crazy the number of implementations out there.\
> I'm using the zope cvs version and I've enhanced it to make it
> serialised and non reentrant. I'd like to check that in.

I'd want to see the rationale and the patch, first.  The product keeps
the schedule itself in the ZODB, which means that the transactional
semantics are going to fight with several types of "serialization".

> In order to simplify installation I do think a Product version of a
> clock would be required. Perhaps the clock can optional be promoted from
> outside to ensure its thread is still alive?
> 
> I'm willing to have a crack at it as long as that effort isn't going to
> yet another non-core scheduler/clock.

- -10 for incluing in the core any implementation which depends on the
existence of a "wild" thread inside the appserver:  coping with threads
and async together is a *very* tricky dance, and when it goes south, you
can't do anything to debug it, or fix it;  all you can do is restart the
appserver.

Chris' ClockServer removes the need for such a thread, by hooking
ZServer's mainloop to generate the "faux" request needed to kick off
async processing.  A "crontab" - like schedule can be driven equally
well from ClockScheduler as from a separate thread.

For real scalability, you need to make the long-running async processing
run in a separate process (and maybe a separate machine!):  in this
case, you can:

  - use ClockServer within the dedicated appserver process

  - run 'wget' or another URL-based trigger against the "trigger" URL
    of the dedicated appserver process

  - run one or more of Gary Poster's "zasync" processes, which aren't
    appservers at all (they don't listen for HTTP/FTP, etc.);  instead,
    they connect to the ZODB and pull tasks from a queue stored there.
    Getting zasync set up correctly can be tricky, but it provides the
    most scalability bang of any of these solutions (e.g., using Tarek
    Ziyade's "zasync load balancer").


Tres.
- --
===================================================================
Tres Seaver          +1 202-558-7113          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.5 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFC3G/l+gerLs4ltQ4RAvhaAJwIpR7edzP50xkNSxDtYaoQQMcVNwCdF0az
DX+0O9D3SmTOhXMk020Guk8=
=eDaJ
-----END PGP SIGNATURE-----