[ZCM] [ZC] 1811/ 7 Comment "Unicode error in PageTemplate"

[Collector: Zope Bugs, Features, and Patches ...]

Issue #1811 Update (Comment) "Unicode error in PageTemplate"
 Status Pending, Zope/bug medium
To followup, visit:
  http://www.zope.org/Collectors/Zope/1811

==============================================================
= Comment - Entry #7 by don on Aug 12, 2005 6:29 am

Due to servers crash I had no possibility to install zope for testing, and now I've installed 2.8.1 and tested on it. 
The second error (with saving old page templates) was solved by changing method PageTemplate.write in PageTemplate.py
Now old page templates (created in zope 2.7) been saving normaly unless it contents python expressions with non-ascii (e.g. cyrillic) characters
    def write(self, text):
        assert type(text) in types.StringTypes
        if text[:len(self._error_start)] == self._error_start:
            errend = text.find('-->')
            if errend >= 0:
                text = text[errend + 4:]

        #Added by Don
        charset = getattr(self, 'management_page_charset', None)
        if charset and type(self._text) == types.StringType:
            try:
                unicode(self._text,'us-ascii')
            except UnicodeDecodeError:
                self._text = unicode(self._text, charset)
        #end Don's addition

        if self._text != text:
            self._text = text
        self._cook()

Now problem persists only when I try to save page template containing python expression with non-ascii characters
For example, I've created page template with one string:
<tal:block replace="python:request.get('Here word with cyrillic characters')" />
And when save it, got and error:
Compilation failed
exceptions.UnicodeEncodeError: 'ascii' codec can't encode characters in position 32-34: ordinal not in range(128)
I've commented some lines in method _cook() of PageTemplate class:
        #try:
        parser.parseString(self._text)
        self._v_program, self._v_macros = parser.getCode()
        #except:
        #    self._v_errors = ["Compilation failed",
        #                      "%s: %s" % sys.exc_info()[:2]]
And now I get  zope error with following traceback:

Module ZPublisher.Publish, line 113, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 40, in call_object
Module Products.PageTemplates.ZopePageTemplate, line 135, in pt_editAction
Module Products.PageTemplates.PageTemplate, line 66, in pt_edit
Module Products.PageTemplates.ZopePageTemplate, line 224, in write
Module Products.PageTemplates.PageTemplate, line 156, in write
Module Products.PageTemplates.PageTemplate, line 194, in _cook
Module TAL.HTMLTALParser, line 126, in parseString
Module TAL.HTMLParser, line 114, in feed
Module TAL.HTMLParser, line 159, in goahead
Module TAL.HTMLParser, line 349, in parse_endtag
Module TAL.HTMLTALParser, line 178, in handle_endtag
Module TAL.TALGenerator, line 801, in emitEndElement
Module TAL.TALGenerator, line 298, in emitCondition
Module TAL.TALGenerator, line 208, in compileExpression
Module Products.PageTemplates.TALES, line 135, in compile
Module Products.PageTemplates.ZRPythonExpr, line 35, in __init__
UnicodeEncodeError: 'ascii' codec can't encode characters in position 32-34: ordinal not in range(128)

________________________________________
= Comment - Entry #6 by efge on Aug 11, 2005 8:40 am

This is probably due to broken browsers that don't post back content with the encoding specified in the page. What browser are you using? It works for me here with Safari.

Maybe an accept-charset should be used on the input form for these broken browsers.
See http://www.w3.org/TR/html40/interact/forms.html#h-17.3 for the spec.


________________________________________
= Comment - Entry #5 by faduci on Aug 11, 2005 7:40 am

I have the same problems with German umlauts and the utf8 setting. If I set management_page_charset as a property of the root folder to 'utf8' and try to store a ZPT that contains umlauts, I get an error:

Error Type: UnicodeDecodeError
Error Value: 'ascii' codec can't decode byte 0xXX in position YYY: ordinal not in range(128)

If I try to save the exact same object in Zope 2.7.4 with the same setting for management_page_charset everything works fine (to my astonishment).

If I remove management_page_charset and force the browser to display the ZMI as utf8 before I paste the source of the ZPT, it works fine also.

So for me the problem seems to be that management_page_charset does not seem to work as it did in 2.7.4. 
________________________________________
= Comment - Entry #4 by anaber on Aug 3, 2005 7:44 am

I have exactly the same problem.
To fix it, I tried to insert "sys.setdefaultencoding('cp1251')" into the sitecustomize.py. Then I could upload file with russian characters in management console, but when I tried to view this one, I saw the '?' symbols instead of letters.
________________________________________
= Comment - Entry #3 by ajung on Jul 2, 2005 4:54 am

Could you try to figure out where the problem is? I've no idea how to reproduce this behaviour with utf8 or so...
________________________________________
= Edit - Entry #2 by ajung on Jul 2, 2005 4:53 am

 Changes: submitter email, importance (critical => medium)
________________________________________
= Request - Entry #1 by don on Jun 19, 2005 3:27 am

I have management_page_charset property with value "koi8-r". With zope 2.8 I can't create or edit Page Template if it contains russian characters in koi8-r encoding. When I try to save such Page Template, I'm getting an error:
Error Type: UnicodeDecodeError
Error Value: 'ascii' codec can't decode byte 0xf2 in position 199: ordinal not in range(128)

If it's new (created in zope 2.8) object, traceback is:
Module ZPublisher.Publish, line 113, in publish 
Module ZPublisher.mapply, line 88, in mapply 
Module ZPublisher.Publish, line 40, in call_object 
Module Products.PageTemplates.ZopePageTemplate, line 142, in pt_editAction 
Module Shared.DC.Scripts.Bindings, line 311, in __call__ 
Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec 
Module Products.PageTemplates.PageTemplateFile, line 110, in _exec 
Module Products.PageTemplates.PageTemplate, line 103, in pt_render
<PageTemplateFile at /rft/adv/ll/pt_editForm> 
Module StringIO, line 203, in getvalue 

If it's old (created in zope 2.7) object, traceback is:
Module ZPublisher.Publish, line 113, in publish 
Module ZPublisher.mapply, line 88, in mapply 
Module ZPublisher.Publish, line 40, in call_object 
Module Products.PageTemplates.ZopePageTemplate, line 135, in pt_editAction 
Module Products.PageTemplates.PageTemplate, line 66, in pt_edit 
Module Products.PageTemplates.ZopePageTemplate, line 224, in write 
Module Products.PageTemplates.PageTemplate, line 144, in write 

In zope 2.7 etherything was ok.
==============================================================