Hi all, I need to run a python script called via xml-rpc that is kicked off by a cron job. This script will be making several dozen, perhaps over 100, different http requests to external servers, some of which can be quite slow on the return. I am curious to know if I am likely to receive a http timeout on this if the xml-rpc request takes a long time -- seems to me that that would happen. Also, curious to know whether anyone has done something like this before and may know of a better way, perhaps using threads, or another mechanism that would let the xml-rpc call return (without a timeout) while the remainder of the work is being done. I'd like to avoid changing the timeout value on Apache since I don't want to change this across the board, if possible. Any ideas? Any known recipes out there? Thanks, Kevin
I need to run a python script called via xml-rpc that is kicked off by a cron job. This script will be making several dozen, perhaps over 100, different http requests to external servers, some of which can be quite slow on the return.
I am curious to know if I am likely to receive a http timeout on this if the xml-rpc request takes a long time -- seems to me that that would happen. Also, curious to know whether anyone has done something like this before and may know of a better way, perhaps using threads, or another mechanism that would let the xml-rpc call return (without a timeout) while the remainder of the work is being done. I'd like to avoid changing the timeout value on Apache since I don't want to change this across the board, if possible.
Depending on what you use the received HTML pages for, you could also periodically request them outside of zope with a simple script (wget or curl etc) and dump them onto the file system. Then zope could read them from there. jens
On Mon, Aug 04, 2003 at 12:03:54PM -0400, Kevin Carlson wrote:
Hi all,
I need to run a python script called via xml-rpc that is kicked off by a cron job. This script will be making several dozen, perhaps over 100, different http requests to external servers, some of which can be quite slow on the return.
I am curious to know if I am likely to receive a http timeout on this if the xml-rpc request takes a long time -- seems to me that that would happen.
It can happen. We get this sometimes when using zsyncer to sync large amounts of stuff. It seems to depend on the client: IE seems to have its own timeout, regardless of what the server does; whereas Mozilla seems to be happy to wait for ages.
Also, curious to know whether anyone has done something like this before and may know of a better way, perhaps using threads, or another mechanism that would let the xml-rpc call return (without a timeout) while the remainder of the work is being done. I'd like to avoid changing the timeout value on Apache since I don't want to change this across the board, if possible.
Any ideas? Any known recipes out there?
I'm not clear - are you worried about each xml-rpc call timing out, or the whole batch of them? If the latter, you could just call RESPONSE.write() after each xml-rpc call. If the individual calls DO take too long, you might try working with threads (tricky!). Or run it as a separate process, which could be launched by an external method e.g. using os.system("command &") to put the separate process in the background. This way, the external method can return immediately. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's ZOOMING INDEFATIGABLE HEATHEN POLAR BEAR! (random hero from isometric.spaceninja.com)
Paul Winkler wrote:
I'm not clear - are you worried about each xml-rpc call timing out, or the whole batch of them? If the latter, you could just call RESPONSE.write() after each xml-rpc call.
I'm worried about the entire batch, actually. A single xml-rpc call would generate multiple requests to other sites. Each of those requests will return a document that is added to a folder and catalogued.
If the individual calls DO take too long, you might try working with threads (tricky!). Or run it as a separate process, which could be launched by an external method e.g. using os.system("command &") to put the separate process in the background. This way, the external method can return immediately.
I'd like to try working with threads to see what is possible, but I'm not sure where to begin with threads and Zope. I've also concluded that I can make a single xml-rpc call to get data on the additional calls and then call each one individually as an additional xml-rpc call. This would be much less likely to timeout. I'd like to try and do this with threads, but after googling on threads in Zope and getting back hundreds of results on "discussion threads" I am hoping someone can point me to a resource on this... Thanks, Kevin
On Mon, Aug 04, 2003 at 02:27:08PM -0400, Kevin Carlson wrote:
would be much less likely to timeout. I'd like to try and do this with threads, but after googling on threads in Zope and getting back hundreds of results on "discussion threads" I am hoping someone can point me to a resource on this...
i suggest the zope-dev list archives at nipltd: http://zope.nipltd.com/public/lists/dev-archive.nsf/$$Search?OpenForm that might get you started. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's THE TONE-DEAF RAINCOAT! (random hero from isometric.spaceninja.com)
participants (3)
-
Jens Vagelpohl -
Kevin Carlson -
Paul Winkler