[Zope] UnicodeDecodeError in Zope 2.10.4 (upgrade from 2.8.4)

[Maciej Wisniowski]

> Both
> <em tal:content="python: 'żółć'">template id</em>.
> and
> <em tal:content="structure python: 'żółć'">template id</em>
> is like playing with fire. Don't do it.
>
> What you've got there is unicode characters written down without any
> encoding information.
This is legacy code from zope 2.8.4 that I have to
deal with during migration. All new code is supposed to use
u'' strings.
In fact, above is only an example but in real case we
have properties set for folders, eg. 'title' that
contains national characters.

You can easily check it yourself if you want.
Just create a Folder and set it's 'title' property
to one that contains some unicode characters.
You'll not be even able to add ZPT object into
that folder I think (at last it is not possible for me).

>
> It will work if you set the internal ZPT encoding to be the same as it
> was entered into the template which I can see is not ASCII which is
> what your system defaults to.
What do you mean by internal ZPT encoding? Docs says:
"Starting Zope 2.10.2 the ZPT implementation uses unicode as internal
representation"
so there should be no encoding at this level, I think.

>
> This is internally what happens:
> >>> x="\xc5"
> >>> unicode(x)
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position
> 0: ordinal not in range(128)
> >>> import sys; sys.getdefaultencoding()
> 'ascii'
I already realized same thing and used sitecustomize.py
with sys.setdefaultencoding('utf-8') as a quick fix. This works
but is not nice solution. I rather expected Zope 2.10 resolver to
deal with this.

-- 
Maciej Wisniowski