[Zope] Re: zope fork in external method - mysql connection dropped

[Daniel Lopez]

I made one tweak to the double-fork procedure, adding a waitpid call in the 
grandparent process (the original zope thread) before it returns out of the 
external method... the code then looked something like:

[...prefork code up to here...]
pid1 = os.fork()
if pid1 > 0:
    #grandparent waits for its child before returning
    os.waitpid(pid1, 0)
    return RESPONSE.redirect("wait_page")
pid2 = os.fork()
os.setsid()
if pid2 > 0:
    #child quits, orphaning grandchild
    sys.exit(0)
[...grandchild-only code after here...]

The waitpid call seems to be preventing the zombies... this is good! 
(though if you find something bad about this approach, please do speak up)

But in the process, a new bug was created, having to do with the MySQL 
connection.  I now receive a "Lost connection to MySQL server during 
query" error in what appears to be a final db flush from the grandparent's 
publish function (ZPublisher.Publish, line 104).  The good news is that 
the grandchild continues to do it's work, but the bad news is that the 
user receives an error page instead of the redirect to the "wait_page".

My guess is that this has something to do with the various forked 
processes sharing the MySQL connection, and in particular, I'm guessing 
the child process that exit(0)'s is closing the connection, which is then 
unavailable when the grandparent tries to use it.

Here's the interesting part: if I put a sleep(1) line in between the 
waitpid() line and the return line, I get no error.  It would appear that 
the 1 second pause provides enough time for a new MySQL connection to be 
made after the previous one is killed (less than 1 second is not enough).

This is a very ugly fix, though... I can't guarantee that 1 second will 
always be enough... can anyone suggest a cleaner solution?  Perhaps I 
shouldn't kill the child with sys.exit(0)?

-Daniel



On Fri, 13 Oct 2006, Jonathan wrote:

> Pls keep your posts on the list  - so that others can help and so that others 
> can search for problems/solutions!
> 
> ----- Original Message -----
> Sent: Friday, October 13, 2006 12:13 PM
> Subject: zope fork in external method
> 
> 
>> 
>> Jonathan-
>> 
>> I'm emailing you because I saw your post a few months back at:
>> 
>> http://mail.zope.org/pipermail/zope/2006-May/166574.html
>> 
>> I've been doing my best to find answers on existing posts, but to no avail. 
>> Perhaps you can send further pointers?
>> 
>> On my system, a user can hit a page that initiates a long database query. If 
>> the user hits the stop button and tries to refresh, it messes things up (by 
>> interrupting the code upon return of the query), so I came up with a 
>> solution that almost works:
>> 
>> I put all the code into an external method, which forks.  The parent returns 
>> a redirect to a "waiting" throbber page that uses Ajax to poll whether the 
>> query is done.  The child then goes on to do the hard work.
>> 
>> This is working as desired, with a single exception: zombie child processes. 
>> Where can I learn more about forking processes in zope external methods? 
>> I've read about the double-fork method, but that hasn't resolved the 
>> problem...
> 
> A few ideas for reaping dead child processes:
> 
> 1) implement a SIGCHLD handler
> 
> 2) when a spawned child process is finished its 'zope processing' have it 
> write its process id to a file (be careful when accessing files, you will 
> need locking to eliminate problems that may be caused by zope's multiple 
> threads) and then have a clean up routine which kills all of the processes 
> listed in the file (this could be an independant clean-up routine which wakes 
> up on a regular basis, or it could be built into the 'spawning' process which 
> does the clean up before it spawns a child).
> 
> 3) as the last thing it does, have the child process issue a kill -9 system 
> command to kill its own process (I haven't tried this myself, but it may work 
> and it would be simple to implement)
> 
> Good luck!
> 
> Jonathan
> 
> 
>