dealing with scripts that take too long
Hi there, we're experiencing problems with certain maintenance scripts, which just take too long to complete, so that the browser resets the connection and Zope aborts the transaction. Short of splitting the scripts up into smaller pieces and running them individually (which would be a pain), what can we do? I was thinking we could send data back to the browser, but I cannot figure out a way to do this from a TTW Python script. How can I send data immediately, not only when I 'return printed' after all the processing is done. NPH or so, I believe this was called with plain CGIs. Also, I would be interested in how other people approach this problem. `zopectl run` may be an alternative, but we'd rather not require filesystem access. -- martin; (greetings from the heart of the sun.) \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck invalid/expired pgp (sub)keys? use subkeys.pgp.net as keyserver! spamtraps: madduck.bogus@madduck.net "arrogance on the part of the meritorious is even more offensive to us than the arrogance of those without merit: for merit itself is offensive." -- friedrich nietzsche
<snip> we're experiencing problems with certain maintenance scripts, which just take too long to complete, so that the browser resets the connection and Zope aborts the transaction. Short of splitting the scripts up into smaller pieces and running them individually (which would be a pain), what can we do? I was thinking we could send data back to the browser, but I cannot figure out a way to do this from a TTW Python script. How can I send data immediately, not only when I 'return printed' after all the processing is done. NPH or so, I believe this was called with plain CGIs. Also, I would be interested in how other people approach this problem. `zopectl run` may be an alternative, but we'd rather not require filesystem access. </snip> We are running a *nix environment, and use cron with script files that use either perl LWP or CURL to make http requests that invoke the zope methods/scripts. LWP, CURL (and others) give you the ability to control the timeout parameters. hth Jonathan
also sprach Jonathan <dev101@magma.ca> [2006.01.25.2330 +0100]:
We are running a *nix environment, and use cron with script files that use either perl LWP or CURL to make http requests that invoke the zope methods/scripts. LWP, CURL (and others) give you the ability to control the timeout parameters.
So does wget. The problem is that our users are on Windows machines without access to or knowledge of a Unix prompt. Thus, NPH would be a nice way to go about it. -- martin; (greetings from the heart of the sun.) \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck invalid/expired pgp (sub)keys? use subkeys.pgp.net as keyserver! spamtraps: madduck.bogus@madduck.net bill gates, 1984: "640 k ought to be enough" bill gates, 1995: "the internet is not a primary goal for pc usage" bill gates, 1999: "linux has no impact on microsoft's strategy"
martin f krafft wrote:
also sprach Jonathan <dev101@magma.ca> [2006.01.25.2330 +0100]:
We are running a *nix environment, and use cron with script files that use either perl LWP or CURL to make http requests that invoke the zope methods/scripts. LWP, CURL (and others) give you the ability to control the timeout parameters.
So does wget. The problem is that our users are on Windows machines without access to or knowledge of a Unix prompt. Thus, NPH would be a nice way to go about it.
My suggestion would be to use ZEO and Stepper... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
also sprach Chris Withers <chris@simplistix.co.uk> [2006.01.30.1854 +0100]:
My suggestion would be to use ZEO and Stepper...
Oooh, this is definitely worth a closer inspection. Thanks a lot. -- martin; (greetings from the heart of the sun.) \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck invalid/expired pgp (sub)keys? use subkeys.pgp.net as keyserver! spamtraps: madduck.bogus@madduck.net "we are trapped in the belly of this horrible machine, and the machine is bleeding to death." -- godspeed you black emperor!
On Jan 25, 2006, at 5:17 PM, martin f krafft wrote:
we're experiencing problems with certain maintenance scripts, which just take too long to complete, so that the browser resets the connection and Zope aborts the transaction.
If these are maintenance scripts that are kicked off manually by admins (as opposed to things that can be automated with cron) then maybe you can use a technique similar to one we use here. For certain long running actions that our users need, we use the Zope Scheduler product. The clicks a button on a UI form, the form then adds the script to the schedule queue as a one shot action. The scheduler clock is actually running on an entirely separate Zope instance on a different machine, so that machines sole duty is to handle these async requests. The script, once completed updates its status, so revisiting the initial form will let the user know the status of their request. (we also give them a view into the schedule queue so if things are taking too long at least they know where they stand.) The Zope server that processes the scheduler queue is set up differently, since it is never has to handle user input. It only has a single thread, and the cache size is larger than normal (the stock zope config is for four threads, and each thread has its own cache pool.)
martin f krafft schrieb:
Hi there, ... I was thinking we could send data back to the browser, but I cannot figure out a way to do this from a TTW Python script. How can I send data immediately, not only when I 'return printed' after all the processing is done. NPH or so, I believe this was called with plain CGIs.
Actually it does not have anything to do with NPH, but thats another story. You can just write via context.REQUEST.RESPONSE.write(somestring) When you start doing this, you are switching to "streaming" mode and anything you return from that script is discarded. This is especially important to know when exceptions occur - you wont see them unless you take precaution try: nasty_things() except Exception,x: response.write("Error: %r\n" % x) # or something raise x # dont forget to reraise! Or you look into the error_log object.
Also, I would be interested in how other people approach this problem. `zopectl run` may be an alternative, but we'd rather not require filesystem access.
zopectl run is actually fine for maintenance. Its also easier to avoid running the same script 100x the same time - as it can happen with HTTP requests. A simple cron job, some locking, report via email ... and you are done. HTH Tino
also sprach Tino Wildenhain <tino@wildenhain.de> [2006.01.25.2345 +0100]:
Actually it does not have anything to do with NPH, but thats another story.
You can just write via context.REQUEST.RESPONSE.write(somestring)
This is exactly what I was looking for, I just couldn't remember the process. I know it's not NPH, but reminiscent of it at least. :) Thanks to you all for the quick responses! -- martin; (greetings from the heart of the sun.) \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck invalid/expired pgp (sub)keys? use subkeys.pgp.net as keyserver! spamtraps: madduck.bogus@madduck.net "a kiss may ruin a human life." -- oscar wilde
On 25 Jan 2006, at 22:17, martin f krafft wrote:
we're experiencing problems with certain maintenance scripts, which just take too long to complete, so that the browser resets the connection and Zope aborts the transaction.
I am assuming you use broken browsers that will time out, like IE? Use a more suitable browser like Firefox, those don't time out by default. jens
also sprach Jens Vagelpohl <jens@dataflake.org> [2006.01.25.2352 +0100]:
I am assuming you use broken browsers that will time out, like IE? Use a more suitable browser like Firefox, those don't time out by default.
As far as I can remember, Firefox was being used, and it didn't time out per se (no error), but the script apparently died some way through; the transaction was aborted. -- martin; (greetings from the heart of the sun.) \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck invalid/expired pgp (sub)keys? use subkeys.pgp.net as keyserver! spamtraps: madduck.bogus@madduck.net humpty was pushed.
On 25 Jan 2006, at 22:56, martin f krafft wrote:
also sprach Jens Vagelpohl <jens@dataflake.org> [2006.01.25.2352 +0100]:
I am assuming you use broken browsers that will time out, like IE? Use a more suitable browser like Firefox, those don't time out by default.
As far as I can remember, Firefox was being used, and it didn't time out per se (no error), but the script apparently died some way through; the transaction was aborted.
If the script dies by itself for some reason then invoking it in other ways might not help. You should concentrate on getting error tracebacks or other evidence that shows *why* the script dies, and go from there. jens
also sprach Jens Vagelpohl <jens@dataflake.org> [2006.01.26.0002 +0100]:
If the script dies by itself for some reason then invoking it in other ways might not help. You should concentrate on getting error tracebacks or other evidence that shows *why* the script dies, and go from there.
I have not analysed the scripts myself, but they appear to be bulk processing jobs. If they use things like objectIds with proper filters, I don't see how it could fail after n iterations... But yeah, the scripts need to be checked... -- martin; (greetings from the heart of the sun.) \____ echo mailto: !#^."<*>"|tr "<*> mailto:" net@madduck invalid/expired pgp (sub)keys? use subkeys.pgp.net as keyserver! spamtraps: madduck.bogus@madduck.net "most people become bankrupt through having invested too heavily in the prose of life. to have ruined one's self over poetry is an honour." -- oscar wilde
participants (6)
-
Andrew Langmead -
Chris Withers -
Jens Vagelpohl -
Jonathan -
martin f krafft -
Tino Wildenhain