The Xron product seems rather "fragile" in use. That is, if things go wrong when an Xron DTML Method is triggered, that method doesn't get rescheduled. An example of this is that my intranet DNS server needed to be rebooted. Xron couldn't look up the appropriate domain name, and stopped working. ------ 2000-10-11T08:16:34 PROBLEM(100) Products.Xron.Loggerr Trigger event: http://my.development.server:4080/my_xron_method Trigger time: 2000/10/11 09:14:00 GMT+1 Failed to trigger event. Type=bci.NotAvailable Val=host not found (File: http://my.development.server/my_xron_method/trigger Line: []) None None for None ------ 2000-10-11T08:19:04 PROBLEM(100) Products.Xron.Loggerr Failed to disarm event Also, if something bad happens in the Dispatcher thread, the thread dies. Before I leap in and start patching Xron, I'd like to have a discussion about how Xron should handle problems. My first thought is that on errors in the dispatcher thread, Xron should enter an "error state" where it probes every so often to see whether it can resume normal operation. The length of time between probes could increase with each probe, to give good performance with transient problems whilst preventing Xron from choking resources. -- Steve Alexander Software Enigneer Cat-Box limited http://www.cat-box.net
Steve, I am also interested in scheduling... though I haven't looked closely at Xron.
The Xron product seems rather "fragile" in use.
That is, if things go wrong when an Xron DTML Method is triggered, that method doesn't get rescheduled.
Does Xron take an optimistic approach to repeating jobs? In other words, does it assume every job is a one-time job and that the last duty that a repeating job performs is to reschedule itself?
An example of this is that my intranet DNS server needed to be rebooted. Xron couldn't look up the appropriate domain name, and stopped working.
------ 2000-10-11T08:16:34 PROBLEM(100) Products.Xron.Loggerr Trigger event: http://my.development.server:4080/my_xron_method Trigger time: 2000/10/11 09:14:00 GMT+1 Failed to trigger event. Type=bci.NotAvailable Val=host not found (File: http://my.development.server/my_xron_method/trigger Line: []) None None for None ------ 2000-10-11T08:19:04 PROBLEM(100) Products.Xron.Loggerr Failed to disarm event
Also, if something bad happens in the Dispatcher thread, the thread dies.
What does the Dispatcher thread do? Does it fire off worker threads?
Before I leap in and start patching Xron, I'd like to have a discussion about how Xron should handle problems.
My first thought is that on errors in the dispatcher thread, Xron should enter an "error state" where it probes every so often to see whether it can resume normal operation. The length of time between probes could increase with each probe, to give good performance with transient problems whilst preventing Xron from choking resources.
What are other threads in Xron doing while the Dispatcher thread is hosed? What are the other threads?
Chris McDonough wrote:
Steve,
I am also interested in scheduling... though I haven't looked closely at Xron.
The Xron product seems rather "fragile" in use.
That is, if things go wrong when an Xron DTML Method is triggered, that method doesn't get rescheduled.
Does Xron take an optimistic approach to repeating jobs? In other words, does it assume every job is a one-time job and that the last duty that a repeating job performs is to reschedule itself?
It is the Xron DTML Method's responsibility to reschedule itself. It could be its first duty, however. Then again, as all the work happens inside a ZODB transaction, it doesn't matter when the reschedule happens.
What does the Dispatcher thread do? Does it fire off worker threads?
Xron has a single dispatcher thread. This thread knows how long to sleep for until the next job needs to run. It uses Client.py to run a job. Jobs are Xron DTML Methods. They have a "trigger" method that gets called through the web from localhost, by the Dispatcher thread.
Before I leap in and start patching Xron, I'd like to have a discussion about how Xron should handle problems.
My first thought is that on errors in the dispatcher thread, Xron should enter an "error state" where it probes every so often to see whether it can resume normal operation. The length of time between probes could increase with each probe, to give good performance with transient problems whilst preventing Xron from choking resources.
What are other threads in Xron doing while the Dispatcher thread is hosed? What are the other threads?
There are no other threads in Xron. Xron creates a single thread to do all event dispatching. The work is done in Zope ZODB threads (or whatever the correct terminology is). The dispatcher thread communicates with a ZODB thread by using Client to make a web request. Xron usually makes requests as the anonymous user. However, I've patched my Xron to make requests as XronUser by hardcoding (!) the username and password for that in the RPC.py module of Xron. Then, I've manually created a XronUser user in my root user folder. I give XronUser the local roles needed to run methods I want to be scheduled via Xron. This is not the ideal way to make Xron use authentication. I posted a proposed design for that to zope-dev in August. http://lists.zope.org/pipermail/zope-dev/2000-August/006548.html http://lists.zope.org/pipermail/zope-dev/2000-August/006549.html http://lists.zope.org/pipermail/zope-dev/2000-August/006550.html -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net
It is the Xron DTML Method's responsibility to reschedule itself. It could be its first duty, however. Then again, as all the work happens inside a ZODB transaction, it doesn't matter when the reschedule happens.
I understand. As a side note, I do not like the fact that Xron requires you to use special DTML methods. I suppose this is a requirement in this architecture due to the fact they need to be autocataloged, but I don't really like that feature either :-).
What does the Dispatcher thread do? Does it fire off worker threads?
Xron has a single dispatcher thread. This thread knows how long to sleep for until the next job needs to run.
Do you think this is this any more effective than having a producer thread do a lookup every minute to see what jobs are current, after which it should put the current jobs in a queue? E.g., as the "run" method of a separate producer thread: def run(self,localtime=time.localtime,time=time.time,sleep=time.sleep): while 1: t = localtime(time()) sleeptime = (60 - t[SECOND]) sleep(sleeptime) for event in self.eventdata.values(): if event.isCurrent(): self.outputqueue.put(EventWrapper(event,self.reportqueue))
It uses Client.py to run a job.
Why is this? This doesn't make any sense to me on its face. Why not just call the method from a separate thread?
What are other threads in Xron doing while the Dispatcher thread is hosed? What are the other threads?
There are no other threads in Xron. Xron creates a single thread to do all event dispatching. The work is done in Zope ZODB threads (or whatever the correct terminology is). The dispatcher thread communicates with a ZODB thread by using Client to make a web request.
I see... would it be naive if I were to say that I think that a scheduler should fire off threads of its own and not rely on an external facility to run the jobs?
Xron usually makes requests as the anonymous user. However, I've patched my Xron to make requests as XronUser by hardcoding (!) the username and password for that in the RPC.py module of Xron. Then, I've manually created a XronUser user in my root user folder. I give XronUser the local roles needed to run methods I want to be scheduled via Xron.
If this were handled not using Client.py, I think we could just use the security manager to change users.
Chris McDonough wrote:
It is the Xron DTML Method's responsibility to reschedule itself. It could be its first duty, however. Then again, as all the work happens inside a ZODB transaction, it doesn't matter when the reschedule happens.
I understand.
As a side note, I do not like the fact that Xron requires you to use special DTML methods. I suppose this is a requirement in this architecture due to the fact they need to be autocataloged, but I don't really like that feature either :-).
My two biggest problems with the XRON facility are the Catalog and the need for speciality methods. I was thinking a more useful model might be something along the lines of a unix cron. each user having access to their own cron tab, via some manage methods, and the methods listed would execute as the user. the hole trigger/disarm methodology to me seems a bit hokey and fragile. i can see the utility in offering up traditional zope semantics of users creating objects, but i really think that scheduling is something that should be part of the core and offered as a service to users.
What does the Dispatcher thread do? Does it fire off worker threads?
Xron has a single dispatcher thread. This thread knows how long to sleep for until the next job needs to run.
this model has problems when new jobs (for immeadiate execution) are entered after the system sleeps. i think it should it taken into account that people will use this dispatching thread to achieve async operations on a request, and that these might need to be done immeadiately.
Do you think this is this any more effective than having a producer thread do a lookup every minute to see what jobs are current, after which it should put the current jobs in a queue? E.g., as the "run" method of a separate producer thread:
the same problem applies, a solution might be a timer and a signal handler to catch a signal event when a new job is entered into the system. events could be managed as some sort of collection of linked lists (or whatever ds).
def run(self,localtime=time.localtime,time=time.time,sleep=time.sleep): while 1: t = localtime(time()) sleeptime = (60 - t[SECOND]) sleep(sleeptime) for event in self.eventdata.values(): if event.isCurrent():
self.outputqueue.put(EventWrapper(event,self.reportqueue))
It uses Client.py to run a job.
Why is this? This doesn't make any sense to me on its face. Why not just call the method from a separate thread?
i played around with this in a monitor thread and you just can't achieve the proper semantics to execute an arbitray zope method. you have to recreate the request, response as well as have all the required support modules (DTML, tracebacks, etc) to successfully call a method, you are in essence reduplicating the publishing machinery. i tried to think of a way to get around this but i came up short. you can still access variables and simple methods.
What are other threads in Xron doing while the Dispatcher thread is hosed? What are the other threads?
There are no other threads in Xron. Xron creates a single thread to do all event dispatching. The work is done in Zope ZODB threads (or whatever the correct terminology is). The dispatcher thread communicates with a ZODB thread by using Client to make a web request.
I see... would it be naive if I were to say that I think that a scheduler should fire off threads of its own and not rely on an external facility to run the jobs?
the dispatcher thread itself has a connection to the ZODB, its using client as RPC to invoke the methods on Publishing threads. Since there is a hardcoded limit to ZODB connections firing off more connections does not seem like a good idea.
Xron usually makes requests as the anonymous user. However, I've patched my Xron to make requests as XronUser by hardcoding (!) the username and password for that in the RPC.py module of Xron. Then, I've manually created a XronUser user in my root user folder. I give XronUser the local roles needed to run methods I want to be scheduled via Xron.
If this were handled not using Client.py, I think we could just use the security manager to change users.
the xron dispatcher has access to ZODB attrs, and simple methods. the security manager would probably need to support semantics of retrieving passwords so they can be set in the client. btw, i'm interested in seeing scheduling get into the zope core as well. Kapil
Do you think this is this any more effective than having a producer thread do a lookup every minute to see what jobs are current, after which it should put the current jobs in a queue? E.g., as the "run" method of a separate producer thread:
the same problem applies, a solution might be a timer and a signal handler to catch a signal event when a new job is entered into the system. events could be managed as some sort of collection of linked lists (or whatever ds).
I think this problem is solved by: - polling for "current" events every 60 seconds *exactly on the 0th second of the minute* (this ensures that stopping and starting the scheduler won't accidently cause a "current" repeating event to fire twice). - allowing one-time events (which are most likely to need to be executed immediately) to be "current" for the minute implied by the event's timestamp as well as the minute just before that one. This ensures that an event that has a timestamp that was "current" when entered, but has subsequently been passed due to event polling overhead will be executed (even if it is entered at hh:mm:59.999 or similar). It also assumes that the scheduling poller will never take more than 60 seconds to poll all events, which is a fairly safe assumption. - the executor of the event is required to invalidate one-time events once they've been executed (or before they're executed) so they don't get fired twice. I don't believe there is any need for a signal handler or anything of the sort, the problem seems to just require careful synchronization.
i played around with this in a monitor thread and you just can't achieve the proper semantics to execute an arbitray zope method. you have to recreate the request, response as well as have all the required support modules (DTML, tracebacks, etc) to successfully call a method, you are in essence reduplicating the publishing machinery. i tried to think of a way to get around this but i came up short. you can still access variables and simple methods.
This would need to be thought through... the same problem exists for unit testing. :-(
I see... would it be naive if I were to say that I think that a scheduler should fire off threads of its own and not rely on an external facility to run the jobs?
the dispatcher thread itself has a connection to the ZODB, its using client as RPC to invoke the methods on Publishing threads. Since there is a hardcoded limit to ZODB connections firing off more connections does not seem like a good idea.
AFAIK, threads will block if they can't get a DB connection. What risks does that imply? Is it a problem?
From: "Kapil Thangavelu" <kthangavelu@earthlink.net> [snip]
Xron has a single dispatcher thread. This thread knows how long to sleep for until the next job needs to run.
this model has problems when new jobs (for immeadiate execution) are entered after the system sleeps. i think it should it taken into account that people will use this dispatching thread to achieve async operations on a request, and that these might need to be done immeadiately.
This is not (or should not be) a problem in current Xron design. The Dispatcher wakes up early if the Schedule catalog is changed. -- Loren
A smoke break later, some possible more thoughts and snake oil, there are two general purpose facilities embodied in xron that i would like to see made available as zope services, async zope processing and scheduling. async-- i'd like to be able to designate TTW that a method execute outside of the current request/response pair to do some long/heavy processing. the design might be easier if this functionality was separated out. a small dynamic pool of publishing threads to execute async functionality (hmmm.. why reinvent the wheel, zope threads are already in place... but what are the consequences of tying up the fixed number of zope's standard publishing threads on a busy site). and a scheduling thread to manage scheduled events/scheduling and dispatching to publishing threads. the possiblities of such are complicated by the magic number limit of thread connections to the ZODB. I was wondering how this magic number came about, is it a performance number dictated by the python interpretor lock?
Chris McDonough wrote:
As a side note, I do not like the fact that Xron requires you to use special DTML methods. I suppose this is a requirement in this architecture due to the fact they need to be autocataloged, but I don't really like that feature either :-).
I think Xron could work with anything that has "trigger" and "disarm" methods, that gets put into the Schedule catalog.
What does the Dispatcher thread do? Does it fire off worker threads?
Xron has a single dispatcher thread. This thread knows how long to sleep for until the next job needs to run.
Do you think this is this any more effective than having a producer thread do a lookup every minute to see what jobs are current, after which it should put the current jobs in a queue?
What if you want a small job to be done every 30 seconds?
It uses Client.py to run a job.
Why is this? This doesn't make any sense to me on its face. Why not just call the method from a separate thread?
Which separate thread? Do you create a new ZODB thread? That sounds as if it might use quite a few resources. Is there a Zope API for making ZODB threads call methods on objects, as signalled by other threads? I'm guessing the easiest way to do that is to use Client.py to make a new http request.
I see... would it be naive if I were to say that I think that a scheduler should fire off threads of its own and not rely on an external facility to run the jobs?
I'd really like to see a scheduler as a standard part of Zope. I think it would be good to have scheduled methods called from within Zope, rather then going through http. I don't know enough about how ZPublisher/ZODB works to know where to start this though. -- Steve Alexander Software Engineer Cat-Box limited http://www.cat-box.net
As a side note, I do not like the fact that Xron requires you to use special DTML methods. I suppose this is a requirement in this architecture due to the fact they need to be autocataloged, but I don't really like that feature either :-).
I think Xron could work with anything that has "trigger" and "disarm" methods, that gets put into the Schedule catalog.
I guess I'm not real real hot on the dependency on a catalog at all. I'm having a hard time imagining a situation where you have so many events that you need to index them -- especially using a ZCatalog. It just seems like too many moving parts and overkill for the job. :-)
Xron has a single dispatcher thread. This thread knows how long to sleep for until the next job needs to run.
Do you think this is this any more effective than having a producer thread do a lookup every minute to see what jobs are current, after which it should put the current jobs in a queue?
What if you want a small job to be done every 30 seconds?
I generally don't... I'm satisfied with cron granularity, although others might not be. Do you have this requirement?
It uses Client.py to run a job.
Why is this? This doesn't make any sense to me on its face. Why not just call the method from a separate thread?
Which separate thread? Do you create a new ZODB thread? That sounds as if it might use quite a few resources.
Is there a Zope API for making ZODB threads call methods on objects, as signalled by other threads? I'm guessing the easiest way to do that is to use Client.py to make a new http request.
Well... AFAIK, there is no such thing as a ZODB thread... there are a number of Python threads used by Zope. We'd just fire off another one to perform a task as part of the thing that takes an event off the queue. This might be naive. I need to try it.
I'd really like to see a scheduler as a standard part of Zope. I think it would be good to have scheduled methods called from within Zope, rather then going through http. I don't know enough about how ZPublisher/ZODB works to know where to start this though.
I'll need to try to call get_transaction().commit() from a manually-constructed Python thread fired off from Zope and see what happens. I think it'll just work. I'll find out in a few minutes. :-)
Chris McDonough wrote:
Xron has a single dispatcher thread. This thread knows how long to
sleep
for until the next job needs to run.
Do you think this is this any more effective than having a producer thread do a lookup every minute to see what jobs are current, after which it should put the current jobs in a queue?
What if you want a small job to be done every 30 seconds?
I generally don't... I'm satisfied with cron granularity, although others might not be. Do you have this requirement?
It uses Client.py to run a job.
Why is this? This doesn't make any sense to me on its face. Why not just call the method from a separate thread?
Which separate thread? Do you create a new ZODB thread? That sounds as if it might use quite a few resources.
Is there a Zope API for making ZODB threads call methods on objects, as signalled by other threads? I'm guessing the easiest way to do that is to use Client.py to make a new http request.
Well... AFAIK, there is no such thing as a ZODB thread... there are a number of Python threads used by Zope. We'd just fire off another one to perform a task as part of the thing that takes an event off the queue. This might be naive. I need to try it.
the real problem with a scheduler thread redesign is being able to call a zope method directly through a zodb connection (and also the connection limit). the zpublishing machinery is sitting atop the zodb, and to really make the method work you need to reduplicate it AFAIK. This just seems inescapable to me as its fundamental to the way zope works, i'd love to be enlightened... Kapil
From: "Chris McDonough" <chrism@digicool.com>
As a side note, I do not like the fact that Xron requires you to use special DTML methods. I suppose this is a requirement in this architecture due to the fact they need to be autocataloged, but I don't really like that feature either :-).
I think Xron could work with anything that has "trigger" and "disarm" methods, that gets put into the Schedule catalog.
I guess I'm not real real hot on the dependency on a catalog at all. I'm having a hard time imagining a situation where you have so many events that you need to index them -- especially using a ZCatalog. It just seems like too many moving parts and overkill for the job. :-)
ZCatalog is overkill, but it was convenient for quick development of Xron. The essential characteristics of the Schedule are: 1. Quickly determine what the next scheduled event is. 2. Automatically update when scheduling parameters of sheduled methods change. 3. Quickly determine how long Dispatcher should sleep (if sleeping model is used). 4. Manage concurrency of Schedule changes initiated by different scheduled methods -- Loren
What if you want a small job to be done every 30 seconds?
I generally don't... I'm satisfied with cron granularity, although others might not be. Do you have this requirement?
I thought a little about the granularity issue and decided to go with the most general solution. I have no requirement for 30-second intervals, but I wanted to avoid support requests from those who might have such a need. Also, I have no need for large numbers of scheduled methods. But I used a heavy-duty data structure (Catalog) and the sleeping dispatcher model, just so there would be no arbitrary limits that would cause me support headaches later.
It uses Client.py to run a job.
Why is this? This doesn't make any sense to me on its face. Why
not
just
call the method from a separate thread?
Which separate thread? Do you create a new ZODB thread? That sounds as if it might use quite a few resources.
Is there a Zope API for making ZODB threads call methods on objects, as signalled by other threads? I'm guessing the easiest way to do that is to use Client.py to make a new http request.
Yes, using Client.py was the easiest way to do that. The biggest problem with Client.py is its dependency on HTTP. If your server is set up to only accept HTTPS, then you can't use Xron -- not a desirable trade-off. We need another mechanism that achieves the same goal -- simulating the environment of a request -- without going all the way back to the socket. -- Loren
Yes, using Client.py was the easiest way to do that. The biggest problem with Client.py is its dependency on HTTP. If your server is set up to only accept HTTPS, then you can't use Xron -- not a desirable trade-off.
If security is the problem, you could configure the HTTP part to listen only on a loopback interface. (yes, this is supported on NT and windows 95 too)
We need another mechanism that achieves the same goal -- simulating the environment of a request -- without going all the way back to the socket.
I always thought the use of Client was very elegent. The performance hit is negligible, and all the plausible alternatives need many more lines of code. Another advantage is that it promotes testability of the scheduled events since it is easy (trivial) to reproduce the environment in which they run when scheduled. Toby Dickenson tdickenson@geminidataloggers.com
From: "Chris McDonough" <chrism@digicool.com> [snip]
As a side note, I do not like the fact that Xron requires you to use special DTML methods. I suppose this is a requirement in this architecture due to the fact they need to be autocataloged, but I don't really like that feature either :-).
The use of special DTML methods was a intended as a "proof of concept" as well as as a simple implementation route to get something running in the short term. It would be preferable to have the ability to schedule any method -- including python methods, perl methods, external methods, .... However, any more general implementation has to address the same issues addressed by the special DTML method of Xron: 1. Automatic reflection of changes to a method's scheduling parameters in the dispatcher's table of scheduled methods. Xron DTML methods are catalog aware for this reason. Autocataloging could clearly be implemented by other means. 2. Making rescheduling decisions inside the scope of the current transaction. Waking up the dispatcher (if the sleeping model is used) when the schedule table changes. For these reasons, Xron DTML Methods have "trigger" methods. A trigger method is like a __call__ method, but it deals with the mechanics of scheduling while leaving __call__ available for calling the method in cases (like when debugging the core logic of the method) when you don't want to mess with the schedule.
What does the Dispatcher thread do? Does it fire off worker threads?
Xron has a single dispatcher thread. This thread knows how long to sleep for until the next job needs to run.
Do you think this is this any more effective than having a producer thread do a lookup every minute to see what jobs are current, after which it should put the current jobs in a queue? E.g., as the "run" method of a separate producer thread:
def run(self,localtime=time.localtime,time=time.time,sleep=time.sleep): while 1: t = localtime(time()) sleeptime = (60 - t[SECOND]) sleep(sleeptime) for event in self.eventdata.values(): if event.isCurrent():
self.outputqueue.put(EventWrapper(event,self.reportqueue))
I don't see much of a functional difference between a polling and a sleeping model for the dispatcher. Both need a list of scheduled methods (events). I chose the sleeping model because it eliminates any granularity for the wakeup times. But it's not enough more complicated that I would consider it a design issue. [snip] -- Loren
participants (5)
-
Chris McDonough -
Kapil Thangavelu -
Loren Stafford -
Steve Alexander -
Toby Dickenson