[Zope] zope fork in external method - mysql connection dropped

[Daniel Lopez]

I spoke too soon.  It was working, but consistent with the Curse of Fork, 
the mysql error eventually started showing up inconsistently.  At first I 
thought it might be dependent on zope's debug mode, but it started showing 
up either way.

So I did what I didn't want to do: split out the 'grandchild' code into a 
command-line script... with the resulting basic structure:

>> external_method.py:
#do some zope stuff here
rv = os.system("shell_script.sh")
if rv:
   #handle error here, such as command not found
return RESPONSE.redirect("wait_page")

>> shell_script.sh
#do some shell stuff here
python_script.py &
exit 0

>> python_script.py
#all the grandchild code here


My understanding of this is that (1) os.system creates a shell in a new 
process [without copying the parent's stack and files?], (2) the ampersand 
line in the shell script creates a grandchild process, and (3) the shell 
script exits, orphaning the python_script and allowing the grandparent to 
proceed.

Maybe it's a little roundabout, but this seems to be cleaner and more 
reliable so far.  Comments?

-Daniel



On Tue, 17 Oct 2006, Daniel Lopez wrote:

>
> You're good, my friend.  I was hoping something like this existed, and so 
> far, it seems to be working exactly as desired.
>
> One note for others: it's actually os._exit() not sys._exit()
>
>
> Re: the other suggestions:
>
> (1) I was already using runzope
>
> (2) subprocess module: I'm using an older version of zope and python, and 
> can't currently risk upgrading core components.
>
> For true cleanliness, in order to avoid the use of fork(), I could try to 
> pull out the grandchild's code into a self-standing script that could get 
> spawned by the grandparent, but that would require a great deal of work, as 
> the grandchild makes a few calls back into zope, along with using some 
> information in the request, which I'd need to pass to the standalone script. 
> I'm happy the lazier approach worked.
>
> Thanks to all...
> -D
>
>
> On Tue, 17 Oct 2006, Dieter Maurer wrote:
>
>> Daniel Lopez wrote at 2006-10-16 13:31 -0700:
>>> I made one tweak to the double-fork procedure, adding a waitpid call in 
>>> the
>>> grandparent process (the original zope thread) before it returns out of 
>>> the
>>> external method... the code then looked something like:
>>> 
>>> [...prefork code up to here...]
>>> pid1 = os.fork()
>>> if pid1 > 0:
>>>    #grandparent waits for its child before returning
>>>    os.waitpid(pid1, 0)
>>>    return RESPONSE.redirect("wait_page")
>>> pid2 = os.fork()
>>> os.setsid()
>>> if pid2 > 0:
>>>    #child quits, orphaning grandchild
>>>    sys.exit(0)
>>> [...grandchild-only code after here...]
>>> 
>>> The waitpid call seems to be preventing the zombies... this is good!
>>> (though if you find something bad about this approach, please do speak up)
>>> 
>>> But in the process, a new bug was created, having to do with the MySQL
>>> connection.  I now receive a "Lost connection to MySQL server during
>>> query" error in what appears to be a final db flush from the grandparent's
>>> publish function (ZPublisher.Publish, line 104).  The good news is that
>>> the grandchild continues to do it's work, but the bad news is that the
>>> user receives an error page instead of the redirect to the "wait_page".
>> 
>> Maybe, the MySQL client library installed an "atexit" hook that closes
>> the database connection.
>> 
>> You may try "sys._exit(0)" (instead of "sys.exit(0)") to finish
>> the child. "_exit" is for purposes when you do not want standard
>> exit handling (such as flushing buffers and calling "atexit" hooks).
>> 
>> 
>> 
>> -- 
>> Dieter
>> 
>> 
>