Problem with zope.sendmail in Python >= 2.5.1
Hi all. One of the contributors to the Launchpad project has identified a problem with zope.sendmail and thread changes in Python
= 2.5.1. Here is his description:
""" Description of the situation: Prior to Python 2.5.1, the atexit handlers were executed when the *main* thread exits. However,http://bugs.python.org/issue1566280 was then filed, the essence of which is that the atexit handlers could tear down cross-thread resources that were still in use by other threads. For this reason, the shutdown procedure was modified in Python 2.5.1 - now, when the main thread exits, it first makes a private call to threading._shutdown(), which waits until all non-daemon threads have exited, and *only then* proceeds to run the atexit handlers. Herein lies the problem. zope.sendmail.delivery.QueueProcessorThread attempts to use atexit to notify itself when it should shut down. However, the Python runtime >= 2.5.1 will wait for the QueueProcessorThread to exit before it calls the atexit handlers! Potential solutions: (1) To gain behaviour most similar to earlier Python versions, an evil monkeypatch into Python internals like this: http://bazaar.launchpad.net/~maxb/zope3/launchpad-3.4-py2.5/revision/9works . (2) Make the QueueProcessorThread a daemon thread. It will be terminated without notification when the interpreter exits, but the current QueueProcessorThread does not attempt to ensure its queue is completely flushed before shutdown anyway, so that should not matter. I will attach a small Python demo program useful for illustrating and exploring the issue. """ You can see the bug report, and his demo program, here: https://bugs.edge.launchpad.net/zope3/+bug/413335 I wasn't going to send this until I had investigated this myself, but I have been taking too long and thought I'd at least send this out to see if anyone has any thoughts. Otherwise, I'll look into it ASAP and report back. Thanks Gary
FWIW, we forked zope.sendmail a while back (with the intent of eventually merging these changes back upstream) as "repoze.sendmail". It does not use any thread to do queue processing. Instead, a separate process can be run to handle queue processing. This is being actively maintained. http://repoze.org/viewcvs/repoze.sendmail/trunk/README.txt?rev=3688&view=mar... On 9/3/09 10:07 PM, Gary Poster wrote:
Hi all. One of the contributors to the Launchpad project has identified a problem with zope.sendmail and thread changes in Python
= 2.5.1. Here is his description:
""" Description of the situation:
Prior to Python 2.5.1, the atexit handlers were executed when the *main* thread exits. However,http://bugs.python.org/issue1566280 was then filed, the essence of which is that the atexit handlers could tear down cross-thread resources that were still in use by other threads.
For this reason, the shutdown procedure was modified in Python 2.5.1 - now, when the main thread exits, it first makes a private call to threading._shutdown(), which waits until all non-daemon threads have exited, and *only then* proceeds to run the atexit handlers.
Herein lies the problem. zope.sendmail.delivery.QueueProcessorThread attempts to use atexit to notify itself when it should shut down. However, the Python runtime>= 2.5.1 will wait for the QueueProcessorThread to exit before it calls the atexit handlers!
Potential solutions:
(1) To gain behaviour most similar to earlier Python versions, an evil monkeypatch into Python internals like this: http://bazaar.launchpad.net/~maxb/zope3/launchpad-3.4-py2.5/revision/9works .
(2) Make the QueueProcessorThread a daemon thread. It will be terminated without notification when the interpreter exits, but the current QueueProcessorThread does not attempt to ensure its queue is completely flushed before shutdown anyway, so that should not matter.
I will attach a small Python demo program useful for illustrating and exploring the issue. """
You can see the bug report, and his demo program, here:
https://bugs.edge.launchpad.net/zope3/+bug/413335
I wasn't going to send this until I had investigated this myself, but I have been taking too long and thought I'd at least send this out to see if anyone has any thoughts. Otherwise, I'll look into it ASAP and report back.
Thanks
Gary _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
On Sep 3, 2009, at 10:53 PM, Chris McDonough wrote:
FWIW, we forked zope.sendmail a while back (with the intent of eventually merging these changes back upstream) as "repoze.sendmail". It does not use any thread to do queue processing. Instead, a separate process can be run to handle queue processing.
This is being actively maintained.
http://repoze.org/viewcvs/repoze.sendmail/trunk/README.txt?rev=3688&view=mar...
Thank you Chris. On the face of it, it sounds like it could be merged in with zope.sendmail very easily, if you have the same interfaces. The zope.sendmail package already advertises the ability to use an external process for processing queued mail, but that appears to be an aspect of the design and the interfaces rather than in the implementation. For this particular problem I faced, though, sticking with threads was easier for now. Gary
participants (2)
-
Chris McDonough -
Gary Poster