As you might know I worked on the integration of the Zope 3 ZPT implementation for Zope 2.10. Before commiting the changes to the trunk I would like discuss my approach for Zope 2.10 - the ZopePageTemplate and PageTemplateFile classes will be replaced with wrapper classes around corresponding Zope 3 classes (no changes to the API, the existing unittest do pass with the wrappers) - the wrapper implementations store the content as Python unicode strings. The encoding of the content must be specified when creating/uploading new instances (either through the ZMI or through the 'encoding' parameters of the constructors). If the content is XML-ish then the encoding is taken from the XML preamble (defaults to utf8) - the ZMI screens for ZPTs use a _fixed_ UTF-8 encoding (is this also fine for asian users?) - deprecation of _all_ other methods, modules etc. from Products/PageTemplates (removal in 2.12 except of the wrapper classes) - deprecation of the TAL module (removal in 2.12) Open: - The ZPublisher will convert a 'unicode' ZPT using default_zpublisher_encoding (zope.conf) to a byte stream. The current encoding is iso-8859-15. I think it would make sense to change it to utf8. Otherwise we must explicitly set the content-type with charset=utf8 set. Thoughts? ToDo: in-place conversion of persistent ZPT instance through setstate() or so... Andreas
--On 5. Januar 2006 12:47:20 +0100 Andreas Jung <lists@andreas-jung.com> wrote:
As you might know I worked on the integration of the Zope 3 ZPT implementation for Zope 2.10. Before commiting the changes to the trunk I would like discuss my approach for Zope 2.10
I forgot to mention a major point: compatibility. When a ZPT is internally stored a unicode string then content returned by methods called through the ZPT will be implicitly converted to unicode. This will definitely raise UnicodeDecodeErrors. So how to deal with this issue? - allowing only unicode textual content when calling macros, PyScript etc. - converting non-unicode to unicode inside the TAL code using some encoding. The encoding could be specified as property of the called method (function properties) or object. We really _need_ to discuss this issue early to minimize side effects and to be able to provide the best compatibility possible. -aj
Andreas Jung wrote at 2006-1-5 13:39 +0100:
... I forgot to mention a major point: compatibility.
When a ZPT is internally stored a unicode string then content returned by methods called through the ZPT will be implicitly converted to unicode. This will definitely raise UnicodeDecodeErrors. So how to deal with this issue?
- allowing only unicode textual content when calling macros, PyScript etc.
definitely not.
- converting non-unicode to unicode inside the TAL code using some encoding. The encoding could be specified as property of the called method (function properties) or object.
Using a site encoding would probably be the best way (as done by e.g. Plone and Archetypes). Most sites use in fact a fixed encoding most of the time. At the places where a different encoding is used, it can be employed explicitely to convert to unicode. As pointed out in private email ("charset negociation will not work"), the response charset has a significant impact on the charset used in form data sent back to the server. This may pose severe problems when the response charset is not the same as the site encoding (for textual form data). -- Dieter
--On 5. Januar 2006 21:12:08 +0100 Dieter Maurer <dieter@handshake.de> >>
- allowing only unicode textual content when calling macros, PyScript etc.
definitely not.
I know that this is a hard requirement but it is an implicit requirement in Zope 3.
- converting non-unicode to unicode inside the TAL code using some encoding. The encoding could be specified as property of the called method (function properties) or object.
Using a site encoding would probably be the best way (as done by e.g. Plone and Archetypes).
Most sites use in fact a fixed encoding most of the time. At the places where a different encoding is used, it can be employed explicitely to convert to unicode.
A site encoding as default makes sense however there are situations when the encoding of a string from is different. E.g. when you deal with XMLHTTPRequest the browser expects an utf8 fragment which might conflict with an iso-8859-15 site-encoding. So there should be some mechanism to specify the encoding. For PyScript one could introduce a property to specify the encoding and methods of products I can imagine using function properties..and this is possibly something some must be solved on the TAL level.
As pointed out in private email ("charset negociation will not work"), the response charset has a significant impact on the charset used in form data sent back to the server. This may pose severe problems when the response charset is not the same as the site encoding (for textual form data).
This issue is connected to the ongoing/planned project to use the Z3 publisher in Zope 2.10. There will be a sprint at PyCon next month afaik... -aj
Andreas Jung wrote at 2006-1-6 10:13 +0100:
... A site encoding as default makes sense however there are situations when the encoding of a string from is different. E.g. when you deal with XMLHTTPRequest the browser expects an utf8 fragment which might conflict with an iso-8859-15 site-encoding. So there should be some mechanism to specify the encoding. For PyScript one could introduce a property to specify the encoding and methods of products I can imagine using function properties..and this is possibly something some must be solved on the TAL level.
I do not like this idea: If I need to modify individual objects (such as Python Scripts), rather than fiddling with properties, I can simply modify the script to return Unicode. I do not think that we need additional special features beside support for some kind of site encoding.
As pointed out in private email ("charset negociation will not work"), the response charset has a significant impact on the charset used in form data sent back to the server. This may pose severe problems when the response charset is not the same as the site encoding (for textual form data).
This issue is connected to the ongoing/planned project to use the Z3 publisher in Zope 2.10. There will be a sprint at PyCon next month afaik...
I do not see that the problem is related to any specific publisher. Instead, it is a conceptual problem: When form data comes back, the server *MUST* know the encoding used by the client. For "POST", the encoding is hopefully specified in the "Content-Type" request header, but for "GET" there is almost surely no information available: the server must assume that the client used the same charset as the page it replied to. However, the server does not have this information if it uses charset negotiation. -- Dieter
Andreas Jung wrote:
--On 5. Januar 2006 12:47:20 +0100 Andreas Jung <lists@andreas-jung.com> wrote:
As you might know I worked on the integration of the Zope 3 ZPT implementation for Zope 2.10. Before commiting the changes to the trunk I would like discuss my approach for Zope 2.10
I forgot to mention a major point: compatibility.
When a ZPT is internally stored a unicode string then content returned by methods called through the ZPT will be implicitly converted to unicode. This will definitely raise UnicodeDecodeErrors. So how to deal with this issue?
Ah, I wrote my reply before reading this.
- allowing only unicode textual content when calling macros, PyScript etc.
- converting non-unicode to unicode inside the TAL code using some encoding. The encoding could be specified as property of the called method (function properties) or object.
In effect Python already does this, it just decodes to unicode using a strict ASCII encoding. Making this configurable per page template might be good, though I'm worried about supporting implicit behavior leading to bad coding patterns. I'd prefer code to be Python unicode clean, but allowing in, say, UTF-8 strings, into a page template and then implicitly converting them to unicode, is inviting people to persist in not understanding the way to write good unicode code.
We really _need_ to discuss this issue early to minimize side effects and to be able to provide the best compatibility possible.
Agreed! Regards, Martijn
Andreas Jung wrote:
ToDo: in-place conversion of persistent ZPT instance through setstate() or so...
Could we avoid doing things in __setstate__ please? It imposes a runtime cost that doesn't disappear. I'd rather have a one-time update method that trawls the database, like is done for Python Scripts recompiling for instance. Or I can call it myself in my upgrade procedures on the exact objects I know will need updates. Florent -- Florent Guillaume, Nuxeo (Paris, France) CTO, Director of R&D +33 1 40 33 71 59 http://nuxeo.com fg@nuxeo.com
--On 5. Januar 2006 17:36:45 +0100 Florent Guillaume <fg@nuxeo.com> wrote:
Andreas Jung wrote:
ToDo: in-place conversion of persistent ZPT instance through setstate() or so...
Could we avoid doing things in __setstate__ please? It imposes a runtime cost that doesn't disappear.
I'd rather have a one-time update method that trawls the database, like is done for Python Scripts recompiling for instance. Or I can call it myself in my upgrade procedures on the exact objects I know will need updates.
I've implemented the setstate() variant as proof-of-concept. I am not sure if there is overhead for CMF-based sites since they FSPageTemplates inherit from PageTemplateFile which has nothing in mind with persistence. So only persistent ZopePageTemplate would be affected. If there is need for a dedicated migration we could easily write one..that's the lamest task of the whole ZPT unicode issue :-) -aj
Andreas Jung wrote at 2006-1-5 17:44 +0100:
...
I'd rather have a one-time update method that trawls the database, like is done for Python Scripts recompiling for instance. Or I can call it myself in my upgrade procedures on the exact objects I know will need updates.
I've implemented the setstate() variant as proof-of-concept. I am not sure if there is overhead for CMF-based sites since they FSPageTemplates inherit from PageTemplateFile which has nothing in mind with persistence. So only persistent ZopePageTemplate would be affected. If there is need for a dedicated migration we could easily write one..that's the lamest task of the whole ZPT unicode issue :-)
Modern ZODB versions allow modifications made in "__setstate__" to be reliably persisted. This means that "__setstate__" updates can now be one time (though the test whether an upgrade is necessary is done always). The "recompile" hack for "PythonScript"s would no longer be necessary. -- Dieter
Dieter Maurer wrote:
I'd rather have a one-time update method that trawls the database, like is done for Python Scripts recompiling for instance. Or I can call it myself in my upgrade procedures on the exact objects I know will need updates.
I've implemented the setstate() variant as proof-of-concept. I am not sure if there is overhead for CMF-based sites since they FSPageTemplates inherit from PageTemplateFile which has nothing in mind with persistence. So only persistent ZopePageTemplate would be affected. If there is need for a dedicated migration we could easily write one..that's the lamest task of the whole ZPT unicode issue :-)
Modern ZODB versions allow modifications made in "__setstate__" to be reliably persisted. This means that "__setstate__" updates can now be one time (though the test whether an upgrade is necessary is done always). The "recompile" hack for "PythonScript"s would no longer be necessary.
I know but that means you have backward compatibility cruft that stays there forever. And no good way to tell the administrators how to upgrade before a given date. Florent -- Florent Guillaume, Nuxeo (Paris, France) CTO, Director of R&D +33 1 40 33 71 59 http://nuxeo.com fg@nuxeo.com
Florent Guillaume wrote:
Andreas Jung wrote:
ToDo: in-place conversion of persistent ZPT instance through setstate() or so...
Could we avoid doing things in __setstate__ please? It imposes a runtime cost that doesn't disappear.
I'd rather have a one-time update method that trawls the database, like is done for Python Scripts recompiling for instance. Or I can call it myself in my upgrade procedures on the exact objects I know will need updates.
Perhaps we can use the generations framework. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Hello.
- the ZMI screens for ZPTs use a _fixed_ UTF-8 encoding (is this also fine for asian users?) This is fine for me, as one of asian user.
- The ZPublisher will convert a 'unicode' ZPT using default_zpublisher_encoding (zope.conf) to a byte stream. The current encoding is iso-8859-15. I think it would make sense to change it to utf8. Otherwise we must explicitly set the content-type with charset=utf8 set. Thoughts?
I've tried to the branch and I found a typo in zope.conf. I think default-zpublisher-encoding is a correct directive name. By the way, I have a request related to this changes. If the rootfolder has `management_page_charset` property as default, this will very convenient for me. Because I can use japanese in ZMI without some setup things. Is this possible? -- TAHARA Yusei tahara@timedia.co.jp
--On 6. Januar 2006 19:55:56 +0900 TAHARA Yusei <tahara@timedia.co.jp> wrote:
I've tried to the branch and I found a typo in zope.conf. I think default-zpublisher-encoding is a correct directive name.
This is already fixed.
By the way, I have a request related to this changes. If the rootfolder has `management_page_charset` property as default, this will very convenient for me. Because I can use japanese in ZMI without some setup things. Is this possible?
This whole management_page_charset is some kind of hack. I also had some trouble getting the ZMI for ZPT to use UTF-8 (which another hack). The current ZMI screen call zpt/read which returns a unicode string (to be presented within the edit textarea). I don't know if any further conversions have to be performed when using japanese in the ZMI...please try yourself (ptEdit.pt). -aj
Andreas Jung wrote:
Thoughts?
All these changes seem to be the right thing. This will make unicode life in Zope a lot easier. I worry about backward compatibility though. Some code (such as PlacelessTranslationService) is doing wild things like monkeypatching the ZPT engine so that incoming unicode is encoded into UTF-8 during page template execution. I.e. the principle is quite different from that of Zope 2 itself, where the publisher takes care of translating things into an encoded string upon output. Since Silva doesn't use PTS anymore I don't worry about this, but Plone developers might. Changing the default encoding of Zope to UTF-8 might break a lot of assumptions in people's code. What about input? If I have an input form, browsers tend to submit in the encoding that the form as in, for instance UTF-8. This means I get UTF-8 strings into my request. Now, if I have code that takes something from that request and displays it in a unicode page template, you'd have a problem, as you'd be mixing UTF-8 with unicode there. Again this might result in a lot of broken code. Regards, Martijn
--On 13. Januar 2006 13:25:19 +0100 Martijn Faassen <faassen@infrae.com> wrote:
Andreas Jung wrote:
Thoughts?
All these changes seem to be the right thing. This will make unicode life in Zope a lot easier.
I worry about backward compatibility though.
Some code (such as PlacelessTranslationService) is doing wild things like monkeypatching the ZPT engine so that incoming unicode is encoded into UTF-8 during page template execution. I.e. the principle is quite different from that of Zope 2 itself, where the publisher takes care of translating things into an encoded string upon output. Since Silva doesn't use PTS anymore I don't worry about this, but Plone developers might.
Changing the default encoding of Zope to UTF-8 might break a lot of assumptions in people's code.
What about input? If I have an input form, browsers tend to submit in the encoding that the form as in, for instance UTF-8. This means I get UTF-8 strings into my request.
Now, if I have code that takes something from that request and displays it in a unicode page template, you'd have a problem, as you'd be mixing UTF-8 with unicode there. Again this might result in a lot of broken code.
I share your worries (meanwhile :-)). Enforcing unicode is too strict. I think to relax the wrapper code so it can handle both unicode and non-unicode (for backward compabitlity)...possibly using some 'strict' flag that enforces the use of unicode...I just don't know yet how to add this in a same way. Andreas
Andreas Jung schrieb: ...
Now, if I have code that takes something from that request and displays it in a unicode page template, you'd have a problem, as you'd be mixing UTF-8 with unicode there. Again this might result in a lot of broken code.
I share your worries (meanwhile :-)). Enforcing unicode is too strict. I think to relax the wrapper code so it can handle both unicode and non-unicode (for backward compabitlity)...possibly using some 'strict' flag that enforces the use of unicode...I just don't know yet how to add this in a same way.
Maybe just have new uZPT with Unicode and leave the "old" ZPT allone? Maybe with limited ability to "add" old ZPT from ZMI or such. This would solve the backward-compatibility problems and would be a more smooth transition w/o the need of upgrade hacks and "strict" hacks (after all, we arent perl/php ;)) Maybe with a "make all my ZPT uZPT" or the like for the real desperate. ++Tino
Tino Wildenhain wrote at 2006-1-13 16:45 +0100:
... Maybe just have new uZPT with Unicode and leave the "old" ZPT allone? Maybe with limited ability to "add" old ZPT from ZMI or such.
This would solve the backward-compatibility problems and would be a more smooth transition w/o the need of upgrade hacks and "strict" hacks (after all, we arent perl/php ;))
I fear it is not that easy: Unless we set Python's "defaultencoding" to the site encoding (and we have such a thing), Python cannot mix Unicode and non-Unicode. Thus, your "old" ZPT's would need to use only other old ZPT's and "old" Python scripts and "old" methods (returning encoded texts) while "strict" ZPT's would need to use only new (strict) ZPT's, scripts and methods. Quite unfeasible... -- Dieter
Dieter Maurer schrieb:
Tino Wildenhain wrote at 2006-1-13 16:45 +0100:
... Maybe just have new uZPT with Unicode and leave the "old" ZPT allone? Maybe with limited ability to "add" old ZPT from ZMI or such.
This would solve the backward-compatibility problems and would be a more smooth transition w/o the need of upgrade hacks and "strict" hacks (after all, we arent perl/php ;))
I fear it is not that easy:
Unless we set Python's "defaultencoding" to the site encoding (and we have such a thing), Python cannot mix Unicode and non-Unicode.
Thus, your "old" ZPT's would need to use only other old ZPT's and "old" Python scripts and "old" methods (returning encoded texts) while "strict" ZPT's would need to use only new (strict) ZPT's, scripts and methods. Quite unfeasible...
Dont think so. The uZPTs would be aware of the fact of "unfriendly" environment. So handling of encoded templates can be done when they are used in a uZPT they would be promoted to unicode w/o touching the default python encoding. (If not unicode ... get encoding from template or site-default -> decode) Mixing all that capabilty with a single "switchable" ZPT implementation strikes me a lot harder to get right, useable and performant. But after all its just an idea. We can discuss it ;) ++Tino
Dieter Maurer wrote:
Tino Wildenhain wrote at 2006-1-13 16:45 +0100:
... Maybe just have new uZPT with Unicode and leave the "old" ZPT allone? Maybe with limited ability to "add" old ZPT from ZMI or such.
This would solve the backward-compatibility problems and would be a more smooth transition w/o the need of upgrade hacks and "strict" hacks (after all, we arent perl/php ;))
I fear it is not that easy:
Unless we set Python's "defaultencoding" to the site encoding (and we have such a thing), Python cannot mix Unicode and non-Unicode.
Thus, your "old" ZPT's would need to use only other old ZPT's and "old" Python scripts and "old" methods (returning encoded texts) while "strict" ZPT's would need to use only new (strict) ZPT's, scripts and methods. Quite unfeasible...
Right, and setting Python's default encoding is out of the question. Regards, Martijn
Martijn Faassen wrote at 2006-1-13 13:25 +0100:
... What about input? If I have an input form, browsers tend to submit in the encoding that the form as in, for instance UTF-8. This means I get UTF-8 strings into my request.
Of course, (textual) input should be converted to Unicode as well. Of course, the encoding used by the browser must be known for this. As the browser usually uses the encoding used by the page containing the form fixing a site encoding might solve this problem. -- Dieter
participants (7)
-
Andreas Jung -
Dieter Maurer -
Florent Guillaume -
Jim Fulton -
Martijn Faassen -
TAHARA Yusei -
Tino Wildenhain