[Zope-dev] UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7

[Marius Gedminas]

On Tue, Aug 31, 2010 at 10:40:46AM +0100, Chris Withers wrote:
> After a recent upgraded to Zope 2.12, I'm now seeing errors like the 
> following when using IE (version 8) or Safari (version 4.0.3) to view 
> page templates. Chrome and Firefox work fine.
> 
> Traceback (innermost last):
>    Module ZPublisher.Publish, line 127, in publish
>    Module ZPublisher.mapply, line 77, in mapply
>    Module ZPublisher.Publish, line 47, in call_object
> <snip>
>    Module Shared.DC.Scripts.Bindings, line 324, in __call__
>    Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec
>    Module Products.PageTemplates.ZopePageTemplate, line 335, in _exec
>    Module Products.PageTemplates.ZopePageTemplate, line 432, in pt_render
>    Module Products.PageTemplates.PageTemplate, line 80, in pt_render
>    Module zope.pagetemplate.pagetemplate, line 115, in pt_render
>    Module zope.tal.talinterpreter, line 271, in __call__
> <snip>
>    Module zope.tal.talinterpreter, line 343, in interpret
>    Module zope.tal.talinterpreter, line 751, in do_insertStructure_tal
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 
> 366: ordinal not in range(128)
> 
> Has anyone seen this or anything similar before?

Yes.  Basically, every Zope site I've upgraded to 2.12 has suffered from
this.  Annoyingly this only happens with MSIE, so developers/admins tend
not to notice.

Clarifying the "only with MSIE" comment: this happens if and only if
the HTTP request doesn't explicitly include 'UTF-8' in Accept-Charset.
IIRC.

> Any solutions spring to mind?

Magically sprinkle .encode() / .decode() in TAL expressions/Python
scripts, where appropriate.

Most of the data stored in the ZODB is UTF-8 raw strings (things like
DTML Documents and DTML Methods don't support storing Unicode data).
Page templates now suddenly (since Zope 2.11) refuse to deal with raw
strings and insist on having Unicode.  Which means you need to do
explicit conversions not at your system boundaries, but inside your Zope
site, as you pass data between Python scripts and page templates.

Marius Gedminas
-- 
http://pov.lt/ -- Zope 3/BlueBream consulting and development
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 198 bytes
Desc: Digital signature
Url : http://mail.zope.org/pipermail/zope-dev/attachments/20100901/20a2439a/attachment.bin