[Zope-dev] howto: background processes, queued events and ZEO?

[Steve Alexander]

Jay, Dylan wrote:

> I have a very computationally expensive task that I need to perform for
> users of my system. The idea is that when they access the page that needs
> the results of this process they will be returned a note saying to wait
> while the calculations are done. The calculation request is queued and
> eventually picked up by a background process and run. Only one calculation
> is run at once.
> 
> My current attempt at implementation is using Xron DTML Methods. However
> from looking at the code I gather that this will not scale well if I decide
> to use ZEO (I think every client will have a background dispatcher so the
> task would be run multiple times). Using ZEO I'd like to nominate one or
> more servers to be the workhorses and do all the calculations. Anyone tried
> doing something like this or can suggest the best approach?


I'm doing somthing like this using a long-running python process 
(nothing to do with Zope) that watches various directories for the 
appearance of xml files that tell it what to do.

When it sees new xml files, it processes them in order of last 
modification time. The xml file contains a URL to go to to notify the 
Zope that produced the xml file that processing is complete.

All the information needed to do the calculation is in the xml file, so 
my long running python process doesn't need to know about Zope or ZEO at 
all.


In practice, users request a report from through the web. They are given 
a page saying "this may take some time. You will receive an email when 
it is complete, or look at your homepage for a progress indication."

Zope constructs a file in a "working" directory, and moves it into the 
watched directory. The long running process sees this file, processes 
it, and tells Zope when it is done. Zope updates the user's homepage, 
and sends the user an email.

--
Steve Alexander
Software Engineer
Cat-Box limited