On Aug 13, 2009, at 5:12, Tres Seaver <tseaver@palladion.com> wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

+1 for the approach in general (I haven't looked at the patch in detail). Assuming all tests pass, and that you have a test exercising the ':' bug you describe, I would just commit it on the trunk (I think such a change should be in the upcoming 2.12 beta).

+1 for the approach and for adding it to 2.12 if it is backwards- compatible. jens

On Wed, Aug 12, 2009 at 9:42 PM, Andreas Jung<lists@zopyx.com> wrote:

On 13.08.09 01:03, Alec Mitchell wrote:

...

Hello,

I've been working on making Plone use the standard Zope MailHost  in place of the custom Products.SecureMailHost we've been using since Plone 2.1 (See: http://dev.plone.org/plone/ticket/8814).  During this process I've run into a couple bugs in the MailHost implementation and I believe it is missing some essential functionality.

The most significant issue is that if you call send() with a messageText containing just the message body, and that body has a ':' in it (e.g. a url) the body will be treated as a header and you'll send a nonsense message.  The current implementation of send() also puts a fairly large burden on developers who want to generate simple, correctly encoded messages.  Finally, send() relies heavily on the long deprecated 'rfc822' and 'mimetools' modules which have been removed from Python 3.0.

I've attached a patch that updates MailHost to use the 'email' module for parsing and generating messages.  In addition to fixing the issues that I ran across, and maintaining compatibility, it provides a number of new features:

* send and sendTemplate accept an optional charset argument.  Using this will set the content-type charset, as well as trigger appropriate encoding if needed.

* send and sendTemplate accept an optional msg_type argument which will set the content type header for the message.

* The messageText, mfrom, mto, and subject arguments may now be unicode or encoded non-ascii strings, provided a charset is given. Any unicode input will be automatically encoded to the provided charset (or the default charset).  Headers will be further encoded in compliance with rfc2822.  The message body will be further encoded using a transfer encoding determined by the email.Charset registry (e.g. 7bit for us-ascii, quoted-printable for utf-8 and iso8859, base64 for most other encodings).

* The messageText argument now accepts email.Message.Message objects directly.

I'm attaching a patch that includes these changes as well as tests for all new functionality.  I hope to integrate these changes into Zope 2.12 before final release, but would like to hear the opinions of Zope developers before committing.  Though these are fairly significant changes, I believe they provide very useful functionality as well as at least one critical fix, while maintaining 100% compatibility.

This comes very, very late. We are pretty close to a release. Please put the changes on the trunk only. We will check after my vacation if we can move it into the 2.12 beta.

I've put my latest changes on Zope trunk. All the existing tests pass (with a couple essentially cosmetic modifications in the MailHost tests), and there are 14 new tests which verify both existing and new functionality, as well as the fixed bugs. The new behavior should be identical to the existing behavior when the new charset and msg_type parameters aren't used, with the following exceptions: 1) Passing a message body containing a ':' no longer produces a nonsense email. 2) Providing unicode strings for the text or headers no longer results in a garbage message (it may produce a UnicodeEncodeError though). 3) 8bit (encoded) strings provided as headers will be converted to 7bit, using encoding determined in messageText headers or the default encoding. It would be very helpful to have these changes in Zope 2.12; otherwise, Plone 4.0 will be stuck with our unmaintained SecureMailHost product for yet another release in order to provide equivalent functionality. Moving to a standard Zope MailHost would be a big benefit for Plone, and all Zope users will benefit from the ability to easily send properly formed non-ASCII messages. Alec P.S. Andreas, if there's an address where I can send some nice wine to help facilitate the merge into 2.12, let me know ;-)

On Thu, 13 Aug 2009 21:22:07 -0700, Alec Mitchell <apm13@columbia.edu> wrote:

On Wed, Aug 12, 2009 at 9:42 PM, Andreas Jung<lists@zopyx.com> wrote:

...

On 13.08.09 01:03, Alec Mitchell wrote:

...

Hello,

I've been working on making Plone use the standard Zope MailHost  in place of the custom Products.SecureMailHost we've been using since Plone 2.1 (See: http://dev.plone.org/plone/ticket/8814).  During this process I've run into a couple bugs in the MailHost implementation and I believe it is missing some essential functionality.

The most significant issue is that if you call send() with a messageText containing just the message body, and that body has a ':' in it (e.g. a url) the body will be treated as a header and you'll send a nonsense message.  The current implementation of send() also puts a fairly large burden on developers who want to generate simple, correctly encoded messages.  Finally, send() relies heavily on the long deprecated 'rfc822' and 'mimetools' modules which have been removed from Python 3.0.

I've attached a patch that updates MailHost to use the 'email' module for parsing and generating messages.  In addition to fixing the issues that I ran across, and maintaining compatibility, it provides a number of new features:

* send and sendTemplate accept an optional charset argument.  Using this will set the content-type charset, as well as trigger appropriate encoding if needed.

* send and sendTemplate accept an optional msg_type argument which will set the content type header for the message.

* The messageText, mfrom, mto, and subject arguments may now be unicode or encoded non-ascii strings, provided a charset is given. Any unicode input will be automatically encoded to the provided charset (or the default charset).  Headers will be further encoded in compliance with rfc2822.  The message body will be further encoded using a transfer encoding determined by the email.Charset registry (e.g. 7bit for us-ascii, quoted-printable for utf-8 and iso8859, base64 for most other encodings).

* The messageText argument now accepts email.Message.Message objects directly.

I'm attaching a patch that includes these changes as well as tests for all new functionality.  I hope to integrate these changes into Zope 2.12 before final release, but would like to hear the opinions of Zope developers before committing.  Though these are fairly significant changes, I believe they provide very useful functionality as well as at least one critical fix, while maintaining 100% compatibility.

This comes very, very late. We are pretty close to a release. Please put the changes on the trunk only. We will check after my vacation if we can move it into the 2.12 beta.

I've put my latest changes on Zope trunk. All the existing tests pass (with a couple essentially cosmetic modifications in the MailHost tests), and there are 14 new tests which verify both existing and new functionality, as well as the fixed bugs. The new behavior should be identical to the existing behavior when the new charset and msg_type parameters aren't used, with the following exceptions:

1) Passing a message body containing a ':' no longer produces a nonsense email. 2) Providing unicode strings for the text or headers no longer results in a garbage message (it may produce a UnicodeEncodeError though). 3) 8bit (encoded) strings provided as headers will be converted to 7bit, using encoding determined in messageText headers or the default encoding.

It would be very helpful to have these changes in Zope 2.12; otherwise, Plone 4.0 will be stuck with our unmaintained SecureMailHost product for yet another release in order to provide equivalent functionality. Moving to a standard Zope MailHost would be a big benefit for Plone, and all Zope users will benefit from the ability to easily send properly formed non-ASCII messages.

There's one additional significant change to Zope behavior here that I forgot to mention. The current implementation sets the python default email transfer and header encoding for 'utf-8' messages to 'quoted-printable', it normally defaults to 'base64'. This is essentially a cosmetic change and makes reading and debugging email messages much more straightforward. It also makes encoded mail less likely to be caught by SPAM filters (some of which dislike base64 mail on principle). At present this change causes one test in CMFDefault to fail, which I'm happy to fix. But it's also not a problem to just remove the line that sets the new 'utf-8' encoding, though I think it dpes have some important advantages. Alec