Re: [Zope-dev] http://cvs.zope.org/Products/Scheduler vs Xron for cron-like functionality. Advice?
The Xron product release is almost a year old, but Scheduler seems to be only available via CVS.
Can anyone comment as to the maturity/stability of these products? Any advice would be much appreciated.
I can't vouch for either of the products you mentioned, but I've had good luck with cron on linux teamed with wget :-)
There doesn't seem to be much support in Zope for use cases where an action is not event driven, ie not a request/response pair.
*** Vaporware Follows*** What *I* want, (but haven't written yet) is a product that can do continuous low level scraping of legacy data sources and feed this data into an RDBMS or the ZODB. A kind of "helper daemon". How each instance is scheduled would be one of many adjustable per instance config options :-) However, assuming that I do get around to building this product, I don't know where on the scale between ugly hack and elegant, reusable solution it will fall. This will also determine whether you ever see it on "zope.org" :-) Adam
Hi All, I'm interested in starting to maintain Xron, if it has potential to be a stable products. From the code it doesn't seem to do any strange things, but I would like to know if anybody has experience of using it in a production environment, or any other experiences and that would recommend not using it in a production environment. The central feature I'm interested in is the ability to setup packing policies inside Zope. I know many people uses Cron or Scheduler but I like to have one platform independent solution and preferably inside the Zope process. Regards, Johan Carlsson
From the code it doesn't seem to do any strange things, but I would like to know if anybody has experience of using it in a production environment,
We have seen the following happen a couple of times: we have Reminders, subclassed from XronDTMLMethod. When a Reminder triggers, it sends email (using MailHost). When this fails, Xron goes into a spin, the Reminder retriggering forever, bloating Data.fs with a Gb or more overnight. We've started looking at the code a couple of times, but everything looked fine .. -- Jean Jordaan Upfront Systems http://www.upfrontsystems.co.za
At 10:30 2002-10-05 +0200, Jean Jordaan wrote:
From the code it doesn't seem to do any strange things, but I would like to know if anybody has experience of using it in a production environment,
We have seen the following happen a couple of times: we have Reminders, subclassed from XronDTMLMethod. When a Reminder triggers, it sends email (using MailHost). When this fails, Xron goes into a spin, the Reminder retriggering forever, bloating Data.fs with a Gb or more overnight. We've started looking at the code a couple of times, but everything looked fine ..
Can't see the reason, yet anyway. But Client.call of the RPC will always return something correct, even if the method that is being called fails? Using Traversal instead, if the call would the Dispatcher would know about it. A when I should be able to stop the event from triggering, and potentially send an e-mail to the admin (which must not fail ;-). I'm trying replacing RPC with restrictedTraverse right now. I'm not sure if I have a security manage in the context of Dispatcher. But it works. Some thoughts: Now it seems like XcronMethods rely on REQUEST to change timer, because it needs it to be able to override for ZClasses. I don't have a REQUEST object, because I'm calling from the Dispatcher thread. This is how I think I'm gone solve it. Split trigger in two separate methods: trigger and reschedule. This also means if trigger raises an exception, reschedule will not be called preventing trigger from being called and fail again. It will still be allowed to override trigger, but reschedule will not be allowed. I think there should be a very narrow bit of code that actually can change the settings of an event. I got a felling there maybe potential security and stability problems here. And I'm very concerned about the ZCatalog being safe here. Is it possible to spoof a catalogObject call from un-secure ttw-code? Also I'm think the trigger time should be changed to a start time and a recurrent setting. The next event should always be calculated from these setting, never be written. Which mean an event could be a read only operation, avoiding unnecessary ZODB bloating. The recurrent setting should be iCal compatible to be able to use Xron for managing triggers in a future Calender server, but this is current not a prioritized feature for me. I think it will happen though in the future. Also, a bit of a contradiction to the read-only events above, I think there is allot of unnecessary blather to the zLOG. I would rather let the Event keep an history over not critical events. It could be optional, I still want the Event to keep a volatile history log to prevent it from being called several times for the same occurrence. This wouldn't be perfect because volatile attributes are thread specific but it could prevent the object from going stall. Best Regards, Johan Carlsson -- Torped Strategi och Kommunikation AB Johan Carlsson johanc@torped.se Mail: Birkagatan 9 SE-113 36 Stockholm Sweden Visit: Västmannagatan 67, Stockholm, Sweden Phone +46-(0)8-32 31 23 Fax +46-(0)8-32 31 83 Mobil +46-(0)70-558 25 24 http://www.torped.se http://www.easypublisher.com
Hi All, I'm interested in starting to maintain Xron, if it has potential to be a stable products. From the code it doesn't seem to do any strange things, but I would like to know if anybody has experience of using it in a production environment, or any other experiences and that would recommend not using it in a production environment.
We are using it, but there seem to be some problems. One of them is that I frequently have cases where Xron doesn't reschedule properly (I am using the improved user interface stuff for Xron, so the bug could also be in there). What happens is that an event that is scheduled for daily execution at midnight will not be rescheduled for the next day after it was executed, but reset to the year 1970, which actually means it is switched off. The other problem is more obvious, but still it is a major issue: Xron seems to be incompatible with ZEO at the moment. The reason why is that the Xron process starts on every ZEO client machine, so everything is executed more than once. I could think of two policies to get around that, both of which should be selectable as an option on a per-event base: - Xron just runs on one server, e.g. the one that is on the fastest machine; this is useful for cases where you need the events to be executed on the same machine all the time, e.g. if you want to write stuff to the server's local file system - Xron runs on all servers (i.e. ZEO clients), but on a first come first serve base, i.e. the server that executes the event first blocks the others from doing so, too. Another issue is that Xron will use the current virtual host settings when it executes and reschedules an event. That means that the entries in the Xron Schedule ZCatalog will have different URLs. In some cases the URL that is used to execute an event could be important. E.g., we use Apache with some tricky rewrite rules in front of Zope, and to get around the Apache server the Zope server has to be called from a different URL. I am willing to help with maintaining Xron, especially the part concerning ZEO, because we really need it ... Joachim
Joachim Werner wrote:
Hi All, I'm interested in starting to maintain Xron, if it has potential to be a stable products. From the code it doesn't seem to do any strange things, but I would like to know if anybody has experience of using it in a production environment, or any other experiences and that would recommend not using it in a production environment.
We are using it, but there seem to be some problems. One of them is that I frequently have cases where Xron doesn't reschedule properly (I am using the improved user interface stuff for Xron, so the bug could also be in there). What happens is that an event that is scheduled for daily execution at midnight will not be rescheduled for the next day after it was executed, but reset to the year 1970, which actually means it is switched off.
We are also using Xron in a production environment, and have the same problem, and also the reverse: Xron doesn't reschedule properly, does not reset the scheduled date, and when the server is restarted tries to "catch up" on all missed events. pretty annoying. I think the problem is related to conflict errors that cause Zope to reissue the request.
Another issue is that Xron will use the current virtual host settings when it executes and reschedules an event. That means that the entries in the Xron Schedule ZCatalog will have different URLs. In some cases the URL that is used to execute an event could be important. E.g., we use Apache with some tricky rewrite rules in front of Zope, and to get around the Apache server the Zope server has to be called from a different URL.
This is a pain, Xron tries to invoke the actions from the front side of zope, ie by issuing a HTTP request. since we use a Zope+Apache http&https setup, we just hacked Xron to rewrite the urls to http://127.0.0.1:8080/...... Anyway, Xron is a backend zope product and shouldn't try to do it's thing going through the main gates, especially if the site is only reachable through https. We really would like to something that just repeats itself after it's done, instead of just doing something every X hours.
I am willing to help with maintaining Xron, especially the part concerning ZEO, because we really need it ...
Joachim
same here because, at this point, we are considering moving to a cron based approach, and it would be nice if we would not need to do it. Sloot.
Romain Slootmaekers wrote:
This is a pain, Xron tries to invoke the actions from the front side of zope, ie by issuing a HTTP request. since we use a Zope+Apache http&https setup, we just hacked Xron to rewrite the urls to http://127.0.0.1:8080/......
Another problem of Xron using standard http access is that you can't use authentication -- at least I could not figure out, how to add a user name and password to the http request issued by Xron. Ok, restricting access to localhost for the "page" requested by Xron can perhaps close this security hole in many circumstances, but while playing with Xron I felt a bit uncomfortable imagining that a melevolent user could easily screw up things like timed publication or removal of a certain page... Abel
At 11:32 2002-10-05 +0200, Joachim Werner said:
Hi All, I'm interested in starting to maintain Xron, if it has potential to be a stable products. From the code it doesn't seem to do any strange things, but I would like to know if anybody has experience of using it in a production environment, or any other experiences and that would recommend not using it in a production environment.
We are using it, but there seem to be some problems. One of them is that I frequently have cases where Xron doesn't reschedule properly (I am using the improved user interface stuff for Xron, so the bug could also be in there). What happens is that an event that is scheduled for daily execution at midnight will not be rescheduled for the next day after it was executed, but reset to the year 1970, which actually means it is switched off.
The other problem is more obvious, but still it is a major issue: Xron seems to be incompatible with ZEO at the moment. The reason why is that the Xron process starts on every ZEO client machine, so everything is executed more than once. I could think of two policies to get around that, both of which should be selectable as an option on a per-event base:
- Xron just runs on one server, e.g. the one that is on the fastest machine; this is useful for cases where you need the events to be executed on the same machine all the time, e.g. if you want to write stuff to the server's local file system
Seems like the most sensible to me. This should be possible to do by splitting the product in two? The Dispatcher and the Scheduler. The Scheduler gets installed on both server and clients but the Dispatcher only gets installed on the server that should run the triggers. I also would like to have a Dispatch monitor ttw where the thread can be start and stopped and the log could be read. Possibly having multiple Dispatcher (for Virtual Hosting situations), which creates another problem because the Dispatcher is started during product instanciation there must be a registry of Dispatchers or someway to start the distributed Dispatcher. For example if the server is restarted. One potential problem would also be notification of failure. I belive I've seen a zLOG email notification product what might be useful.
- Xron runs on all servers (i.e. ZEO clients), but on a first come first serve base, i.e. the server that executes the event first blocks the others from doing so, too.
This would need some kind of inter-process locking, which I think should be provided by ZEO. I'm presently not aware of any such services in ZEO?
Another issue is that Xron will use the current virtual host settings when it executes and reschedules an event. That means that the entries in the Xron Schedule ZCatalog will have different URLs. In some cases the URL that is used to execute an event could be important. E.g., we use Apache with some tricky rewrite rules in front of Zope, and to get around the Apache server the Zope server has to be called from a different URL.
I've noticed that Xcron uses ZPublishers client to trigger events. This seems a bit out-dated to me, wouldn't it be better to use restrictedTraverse? Also all events must be given by it's physicalPath, something we learned the hard way. VirtualPath may be translated to PhysicalPaths when needed, something I think we added to our version of VHM, which we would be happy to share. Who should the events be run as? To day the are run as Anonymous even though there are arguments for passing user name and password I can find any in the UI. I think Event should be run as Owner or ProxyRole. But I'm not sure?
I am willing to help with maintaining Xron, especially the part concerning ZEO, because we really need it ... Joachim
Excellent! :-) Best Regards, Johan Carlsson -- Torped Strategi och Kommunikation AB Johan Carlsson johanc@torped.se Mail: Birkagatan 9 SE-113 36 Stockholm Sweden Visit: Västmannagatan 67, Stockholm, Sweden Phone +46-(0)8-32 31 23 Fax +46-(0)8-32 31 83 Mobil +46-(0)70-558 25 24 http://www.torped.se http://www.easypublisher.com
Hi! OT: I am going to switch to some other mail client soon. It's unbelievable that Outlook Express still doesn't do the ">" thing right :-(
- Xron just runs on one server, e.g. the one that is on the fastest machine; this is useful for cases where you need the events to be executed on the same machine all the time, e.g. if you want to write stuff to the server's local file system
"Seems like the most sensible to me. This should be possible to do by splitting the product in two? The Dispatcher and the Scheduler. The Scheduler gets installed on both server and clients but the Dispatcher only gets installed on the server that should run the triggers." The reason why I think this behaviour should be optional is that there might be different scenarios. There even might be a scenario where an event should be triggered on all of the servers. But for a start I'd be happy if at least the events are not called more than once. "I also would like to have a Dispatch monitor ttw where the thread can be start and stopped and the log could be read." Would be nice indeed. "Possibly having multiple Dispatcher (for Virtual Hosting situations), which creates another problem because the Dispatcher is started during product instanciation there must be a registry of Dispatchers or someway to start the distributed Dispatcher. For example if the server is restarted." Yep :-| "One potential problem would also be notification of failure. I belive I've seen a zLOG email notification product what might be useful." Yes.
- Xron runs on all servers (i.e. ZEO clients), but on a first come first serve base, i.e. the server that executes the event first blocks the others from doing so, too.
"This would need some kind of inter-process locking, which I think should be provided by ZEO. I'm presently not aware of any such services in ZEO?" ZEO can't really do that at the moment. It's really just doing ZODB stuff, which is very limited in scope. What you can do of course is just use a global status parameter in the ZODB, but the only effect that would have is that the first one to set that status parameter would make the other processes fail and roll back. But a lot of stuff, like sending mails, can hardly be rolled back ...
Another issue is that Xron will use the current virtual host settings when it executes and reschedules an event. That means that the entries in the Xron Schedule ZCatalog will have different URLs. In some cases the URL that is used to execute an event could be important. E.g., we use Apache with some tricky rewrite rules in front of Zope, and to get around the Apache server the Zope server has to be called from a different URL.
"I've noticed that Xcron uses ZPublishers client to trigger events. This seems a bit out-dated to me, wouldn't it be better to use restrictedTraverse?" After looking at Xron in more detail and comparing it to the Scheduler stuff in the Zope CVS, I guess that going the Scheduler way is probably more promising in the long run. Xron seems to have quite a few major design flaws. One I haven't mentioned yet is that you can only trigger DTML methods. In a modern Zope installation, you'd probably want to call Scripts (Python) more often, though of course you can call everything FROM DTML ... Joachim
At 20:40 2002-10-05 +0200, Joachim Werner said:
"I've noticed that Xcron uses ZPublishers client to trigger events. This seems a bit out-dated to me, wouldn't it be better to use restrictedTraverse?"
After looking at Xron in more detail and comparing it to the Scheduler stuff in the Zope CVS, I guess that going the Scheduler way is probably more promising in the long run. Xron seems to have quite a few major design flaws. One I haven't mentioned yet is that you can only trigger DTML methods. In a modern Zope installation, you'd probably want to call Scripts (Python) more often, though of course you can call everything FROM DTML ..
Yes. I also see at least one pre-built events like a DBPackEvent and possibly more. Events may need to be restricted to only be created in the Scehduler. Not sure that we need this one. Just an idea that pop by :-) Regards, Johan Carlsson
-- Torped Strategi och Kommunikation AB Johan Carlsson johanc@torped.se Mail: Birkagatan 9 SE-113 36 Stockholm Sweden Visit: Västmannagatan 67, Stockholm, Sweden Phone +46-(0)8-32 31 23 Fax +46-(0)8-32 31 83 Mobil +46-(0)70-558 25 24 http://www.torped.se http://www.easypublisher.com
participants (5)
-
abel deuring -
Jean Jordaan -
Joachim Werner -
Johan Carlsson [Torped] -
Romain Slootmaekers