[Zope] UnicodeDecodeError in Zope 2.10.4 (upgrade from 2.8.4)

[Maciej Wisniowski]

I've upgraded my installation from Zope2.8.4 to Zope 2.10.4
(by copying data.fs, Products/ etc.). I have publisher encoding
and management_page_charset set to utf-8. Also system
default encoding is utf-8.

Zope 2.10 is said to have better Unicode support with
UnicodeEncodeConflictResolver. It is but unfortunatelly in
some cases this is not enough. Seems that
code that is 'protected' by Resolver is not the only code
that may be affected by non unicode strings.
Simple example is with 'structure' keyword.

eg:
This works (resolver resolves conflict):
<em tal:content="python: 'żółć'">template id</em>.

This doesn't work:
<em tal:content="structure python: 'żółć'">template id</em>.

Also if you have Folder instance and set it's Title
to the string that contains some i18n characters
you're not able to even add page template
inside it.

Error traceback in both cases is same:

Error Type: UnicodeDecodeError
Error Value: 'ascii' codec can't decode byte 0xc5 in position 200:
ordinal not in range(128)

Traceback (innermost last):
Module ZPublisher.Publish, line 119, in publish
Module ZPublisher.mapply, line 88, in mapply
Module ZPublisher.Publish, line 42, in call_object
Module Shared.DC.Scripts.Bindings, line 313, in __call__
Module Shared.DC.Scripts.Bindings, line 350, in _bindAndExec
Module Products.PageTemplates.PageTemplateFile, line 129, in _exec
Module Products.PageTemplates.PageTemplate, line 89, in pt_render
Module zope.pagetemplate.pagetemplate, line 117, in pt_render
Module zope.tal.talinterpreter, line 271, in __call__
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 534, in do_optTag_tal
Module zope.tal.talinterpreter, line 516, in no_tag
Module zope.tal.talinterpreter, line 346, in interpret
Module zope.tal.talinterpreter, line 754, in do_insertStructure_tal
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc5 in position
200: ordinal not in range(128)

problem is with:
  text = unicode(structure)
at
Module zope.tal.talinterpreter, line 754, in do_insertStructure_tal

sitecustomize.py with setdefaultencoding('utf-8') solves this
but it is not nice solution. I wonder whether this should be
submitted as a bug, or maybe there is different solution that
I've missed?

For the record: one more thing that was wrong
for me during migration was 'expand' attribute of ZPT
that in case of few ZPT objects was set to true in
2.10.4 while oryginally in 2.8.4 it was false.
This caused that tales expressions disappeared
from ZPTs under 2.10.4. I've written a script that
explicitly set expand=0 to all ZPT instances in
Zope 2.8.4 and then, after migration, everything is OK.

-- 
Maciej Wisniowski