Re: [Zope] Re: problem with input elements and unicode/utf-8
Dieter, Andreas and Daryl Here comes a detailed description of a new Zope-installation I just made on a new Debian (testing) machine. Here are the steps to reproduce my problem. (I'm still not sure it is a problem other than me not understanding how this should work). Downloading Zope wget http://www.zope.org/Products/Zope/2.10.2/Zope-2.10.2.tgz tar -xzvf Zope-2.10.2.tgz cd Zope-2.10.2-final ./configure --prefix=/usr/local/zope make sudo make install sudo ./usr/local/zope/bin/mkzopeinstance.py /usr/local$ sudo chown -R zope.zope zope /usr/local$ sudo chown -R zope.zope zopeinst sudo jed /usr/local/zopeinst/etc/zope.conf <-- set effective-user zope leira:/usr/local/zopeinst/bin# sudo ./runzope Logging in to ZMI directly to port 8080 I add a new Page Template. If the page template contains only hei på deg it saves OK and runs OK. If it contains either of the tal-lines below I get the following error message: <div tal:content="python:unicode('hei på deg','utf8')" /> Compilation failed exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\xe5' in position 33: ordinal not in range(128) <div tal:content="python:u'hei på deg'" /> Compilation failed exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\xe5' in position 26: ordinal not in range(128) <div tal:content="python:'hei på deg'" /> Compilation failed exceptions.UnicodeEncodeError: 'ascii' codec can't encode character u'\xe5' in position 25: ordinal not in range(128) In this test I have used IE6 (and it is not passing HTTP_ACCEPT_CHARSET in the request). By the way: The character 'å' has value 0xe5 in iso-8859-1. Should the above tal-statements have worked? I have done no other configuration on the zope-instances than stated here but are consistant with my problems on another installation at work. Regards Jost --------------------------------- We won't tell. Get more on shows you hate to love (and love to hate): Yahoo! TV's Guilty Pleasures list.
--On 19. Februar 2007 12:29:34 -0800 Jostein Leira <jleira@yahoo.com> wrote:
If it contains either of the tal-lines below I get the following error message: <div tal:content="python:unicode('hei på deg','utf8')" />
I can reproduce this error. Obviously RestrictedPython.compile_restricted_eval() can't deal correctly with unicode strings. *SIGH* A-life-w/o-unicode-could-be-a-better-life, Andreas
--On 20. Februar 2007 17:52:52 +0100 Andreas Jung <lists@zopyx.com> wrote:
--On 19. Februar 2007 12:29:34 -0800 Jostein Leira <jleira@yahoo.com> wrote:
If it contains either of the tal-lines below I get the following error message: <div tal:content="python:unicode('hei på deg','utf8')" />
Please try out the following patch and report back: Index: ZRPythonExpr.py =================================================================== --- ZRPythonExpr.py (revision 72721) +++ ZRPythonExpr.py (working copy) @@ -29,7 +29,11 @@ def __init__(self, name, expr, engine): self.text = text = expr.strip().replace('\n', ' ') - code, err, warn, use = compile_restricted_eval(text, str(self)) + self.unicode = False + if isinstance(text, unicode): + text = text.encode('utf-8') + self.unicode = True + code, err, warn, use = compile_restricted_eval(text, self.__class__.__name__) if err: raise engine.getCompilerError()('Python expression error:\n%s' % '\n'.join(err)) @@ -40,7 +44,10 @@ __traceback_info__ = self.text vars = self._bind_used_names(econtext, {}) vars.update(self._globals) - return eval(self._code, vars, {}) + result = eval(self._code, vars, {}) + if self.unicode: + result = unicode(result, 'utf-8') + return result class _SecureModuleImporter: __allow_access_to_unprotected_subobjects__ = True -aj
--On 20. Februar 2007 18:52:51 +0100 Andreas Jung <lists@zopyx.com> wrote:
--On 20. Februar 2007 17:52:52 +0100 Andreas Jung <lists@zopyx.com> wrote:
--On 19. Februar 2007 12:29:34 -0800 Jostein Leira <jleira@yahoo.com> wrote:
If it contains either of the tal-lines below I get the following error message: <div tal:content="python:unicode('hei på deg','utf8')" />
This patch works better: Index: ZRPythonExpr.py =================================================================== --- ZRPythonExpr.py (revision 72721) +++ ZRPythonExpr.py (working copy) @@ -29,7 +29,9 @@ def __init__(self, name, expr, engine): self.text = text = expr.strip().replace('\n', ' ') - code, err, warn, use = compile_restricted_eval(text, str(self)) + if isinstance(text, unicode): + text = text.encode('utf-8') + code, err, warn, use = compile_restricted_eval(text, self.__class__.__name__) if err: raise engine.getCompilerError()('Python expression error:\n%s' % '\n'.join(err)) -aj
Andreas Jung wrote at 2007-2-20 17:52 +0100:
... I can reproduce this error. Obviously RestrictedPython.compile_restricted_eval() can't deal correctly with unicode strings.
Newer Python versions need to know the encoding of the source. In standard Python scripts, a coding comment in the form "# -*- coding ..." is used. For PageTemplates, the template would need to provide the encoding information in some way (though I do not know precisely how this should happen). -- Dieter
--On 21. Februar 2007 21:27:38 +0100 Dieter Maurer <dieter@handshake.de> wrote:
Andreas Jung wrote at 2007-2-20 17:52 +0100:
... I can reproduce this error. Obviously RestrictedPython.compile_restricted_eval() can't deal correctly with unicode strings.
Newer Python versions need to know the encoding of the source.
In standard Python scripts, a coding comment in the form "# -*- coding ..." is used.
This has absolutely nothing to do with the problem. Andreas
participants (3)
-
Andreas Jung -
Dieter Maurer -
Jostein Leira