-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, I've created a branch svn+ssh://svn.zope.org/repos/main/Zope/branches/ajung-zpt-encoding-fixes that should fix several encoding and webdav issues that unfortunately were discovered before the final 2.10 release. What has changed? Until Zope 2.10 the ZopePageTemplate class had no notion of an encoding. You could upload ascii, iso-8859-15, utf8...or whatever...everything has been stored as a Python byte string. For backward compatibility we did not convert the data into Python unicode strings with the introduction of the Zope 3 ZPT implementation. Although the ZopePageTemplate implementation already had a 'strict' flag (to enforce the conversion to Python unicode) the Zope 2 implementation did not enforce unicode as storage for a ZPT. This caused several encoding problems when editing a ZPT through the ZMI and in addition the WebDAV support was broken in Zope 2.10.0 and Zope 2.10.1. A ZPT has now something as an output_encoding. When you create a ZPT through the ZMI you'll be asked about the encoding (which is utf8 by default). The pt_render() method now converts the internal unicode representation back to the output encoding. This is basically the behavior of the old ZPT implementation. In addition the __call__() method sets the 'charset' property of the content-type header according to the configured output encoding. WebDAV: the PUT factory was using the write() method to store uploaded content. This method wasn't aware of the output encoding. PUT() now uses pt_edit(). This implies that the uploaded content must have the same encoding as the output encoding. Means: when you create a ZPT with encoding UTF-8 you can't upload new content with a different encoding. This is a slightly different behavior from older Zope versions and might break backward compatibility. Anyone having such a usecase? One might check in addition for the 'encoding' attribute inside the XML preamble or for the 'charset' property inside a <meta http-equiv="content-type" ..> tag for HTML documents. Any objections merging the changes on the head after further polishing and writing some more test? Andreas - -- ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany Web: www.zopyx.com - Email: info@zopyx.com - Phone +49 - 7071 - 793376 E-Publishing, Python, Zope & Plone development, Consulting -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (Darwin) iD8DBQFFhWvECJIWIbr9KYwRAg5xAJ9LEYbyy2TCrKj1b4VUcqE90qqqkwCeLr6B F3YYBC8V2oPEaOq1xH1FeFg= =68R/ -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Andreas Jung wrote:
Hi,
I've created a branch
svn+ssh://svn.zope.org/repos/main/Zope/branches/ajung-zpt-encoding-fixes
that should fix several encoding and webdav issues that unfortunately were discovered before the final 2.10 release.
What has changed?
Until Zope 2.10 the ZopePageTemplate class had no notion of an encoding. You could upload ascii, iso-8859-15, utf8...or whatever...everything has been stored as a Python byte string. For backward compatibility we did not convert the data into Python unicode strings with the introduction of the Zope 3 ZPT implementation. Although the ZopePageTemplate implementation already had a 'strict' flag (to enforce the conversion to Python unicode) the Zope 2 implementation did not enforce unicode as storage for a ZPT. This caused several encoding problems when editing a ZPT through the ZMI and in addition the WebDAV support was broken in Zope 2.10.0 and Zope 2.10.1.
A ZPT has now something as an output_encoding. When you create a ZPT through the ZMI you'll be asked about the encoding (which is utf8 by default). The pt_render() method now converts the internal unicode representation back to the output encoding. This is basically the behavior of the old ZPT implementation. In addition the __call__() method sets the 'charset' property of the content-type header according to the configured output encoding.
That is dangerous, because a page template may be called without being the "main" driver for a request; the response encoding should be used, if already set, rather than the value set on the template.
WebDAV: the PUT factory was using the write() method to store uploaded content. This method wasn't aware of the output encoding. PUT() now uses pt_edit(). This implies that the uploaded content must have the same encoding as the output encoding. Means: when you create a ZPT with encoding UTF-8 you can't upload new content with a different encoding. This is a slightly different behavior from older Zope versions and might break backward compatibility. Anyone having such a usecase? One might check in addition for the 'encoding' attribute inside the XML preamble or for the 'charset' property inside a <meta http-equiv="content-type" ..> tag for HTML documents.
PUT should always extract the encoding from the upload request, and use it to decode the template to unicode for storage. While saving that encoding as the "outpout encoding" for a newly-created template is reasonable, modifying the "output encoding" for an existing template is riskier. I'm not even sure there is a usecase for a per-template output encoding -- it seems more like a site-wide policy to me (perhaps configured as a view?)
Any objections merging the changes on the head after further polishing and writing some more test?
Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFhX3C+gerLs4ltQ4RAgA+AJ9HnkSxrN6DgiY1jujxSGXJ324E2QCgjstU gQdBYp9Uay/+tDApM+uPYNI= =27Or -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - --On 17. Dezember 2006 12:26:26 -0500 Tres Seaver <tseaver@palladion.com> wrote:
A ZPT has now something as an output_encoding. When you create a ZPT through the ZMI you'll be asked about the encoding (which is utf8 by default). The pt_render() method now converts the internal unicode representation back to the output encoding. This is basically the behavior of the old ZPT implementation. In addition the __call__() method sets the 'charset' property of the content-type header according to the configured output encoding.
That is dangerous, because a page template may be called without being the "main" driver for a request; the response encoding should be used, if already set, rather than the value set on the template.
Ok, you're right. Is there an reliable way to distinguish between a direct call and a non-direct call?
WebDAV: the PUT factory was using the write() method to store uploaded content. This method wasn't aware of the output encoding. PUT() now uses pt_edit(). This implies that the uploaded content must have the same encoding as the output encoding. Means: when you create a ZPT with encoding UTF-8 you can't upload new content with a different encoding. This is a slightly different behavior from older Zope versions and might break backward compatibility. Anyone having such a usecase? One might check in addition for the 'encoding' attribute inside the XML preamble or for the 'charset' property inside a <meta http-equiv="content-type" ..> tag for HTML documents.
PUT should always extract the encoding from the upload request, and use it to decode the template to unicode for storage.
This works already. However there will be cases when the code can't autodetect an encoding.
While saving that encoding as the "outpout encoding" for a newly-created template is reasonable, modifying the "output encoding" for an existing template is riskier.
Newly created content-type is still an open point..look as if I have to write a bunch of unittests for all edgecases. - -aj -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (Darwin) iD8DBQFFhYD0CJIWIbr9KYwRAgNHAKCPkhRE5By/ORg6HUgf/Hz3a29t2wCgxRnp 16FC8lfCNkxvPwxXHsOKMMQ= =IxWq -----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 - --On 17. Dezember 2006 18:40:03 +0100 Andreas Jung <lists@zopyx.com> wrote:
That is dangerous, because a page template may be called without being the "main" driver for a request; the response encoding should be used, if already set, rather than the value set on the template.
Ok, you're right. Is there an reliable way to distinguish between a direct call and a non-direct call?
I reverted the change. The __call__() and pt_render() method now returns the rendered PT *always* as unicode string. It is now up to the ZPublisher or the template itself to set the 'charset' as needed.
WebDAV: the PUT factory was using the write() method to store uploaded content. This method wasn't aware of the output encoding. PUT() now uses pt_edit(). This implies that the uploaded content must have the same encoding as the output encoding. Means: when you create a ZPT with encoding UTF-8 you can't upload new content with a different encoding. This is a slightly different behavior from older Zope versions and might break backward compatibility. Anyone having such a usecase? One might check in addition for the 'encoding' attribute inside the XML preamble or for the 'charset' property inside a <meta http-equiv="content-type" ..> tag for HTML documents.
PUT should always extract the encoding from the upload request, and use it to decode the template to unicode for storage.
This works already. However there will be cases when the code can't autodetect an encoding.
While saving that encoding as the "outpout encoding" for a newly-created template is reasonable, modifying the "output encoding" for an existing template is riskier.
Newly created content-type is still an open point..look as if I have to write a bunch of unittests for all edgecases.
I've added a bunch of unittest and some logic to cover all edgecases when editing a ZPT either through-the-web or through WebDAV/FTP. The output_encoding is now used in two places: for the ZPT edit form and when accessing a ZPT through WebDAV/FTP. The output_encoding is determined by the content-type and the content itself. For text/xml the output_encoding is always set to utf-8. For text/html the ZPT engine checks for a <meta http-equiv="content-type" ...> header and tries to extract the 'charset'. If it is not available the output_encoding will be iso-8859-15. Before merging the changes must be tested with CMF. Andreas -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (Darwin) iD8DBQFFhndtCJIWIbr9KYwRAtnvAJ45QnO2IhCNKwiyFhyskgHA5QbXmgCeOs/Z gfwjBTyds+MYaT6Pf6UFUhg= =itbR -----END PGP SIGNATURE-----
participants (2)
-
Andreas Jung -
Tres Seaver