[Zope] Zope 2.9 vs Zope 2.10 and Unicode problems

[Doyon, Jean-Francois]

Hello,

I just upgraded from Zope 2.9.4 to Zope 2.10 and am running into the common type of error:

UnicodeDecodeError: 'ascii' codec can't decode byte 0xe9 in position 295: ordinal not in range(128)

I've tracked it down to the fact that in Zope 2.10 it looks like the result of tag:replace tags (like the img/tag TAL), returns a unicode string:

u'<img src="http://<host>/jf293/canada_wordmark.gif" alt="" title="" height="21" width="86" />'

(In Zope 2.9.4 this was a normal string)

This creates a problem because the PT is in ISO-8859-1 and contains french accents in non unicode strings, such as in the <meta> tags.

<meta name="dc.creator" content="Gouvernement du Canada, Ressources Naturelles Canada, Secteur des sciences de la terre, Géomatique Canada - Centre canadien de télédétection, Division GéoAccès, Section de l'infrastructure de l'Atlas" /> 

Which looks like this by the time StringIO.getvalue() sees it:

<meta name="dc.creator" content="Gouvernement du Canada, Ressources Naturelles Canada, Secteur des sciences de la terre, G\xe9omatique Canada - Centre canadien de t\xe9l\xe9d\xe9tection, Division G\xe9oAcc\xe8s, Section de l\'infrastructure de l\'Atlas" />

So I now end up with the UnicodeDecodeError :(

How would I got about making sure that either my HTML ends up in unicode strings, or making sure the tal:replace (and possibly others?) don't use unicode strings?  Both these things are probably rooted deep in the zope 2 or zope 3 code?

My locale is set to en_CA and the default charset is iso-8859-1 ...

Any ideas?

Thanks in advance!

Jean-François Doyon
Internet Service Development and Systems Support / Spécialiste de dèveloppements internet et soutien technique
Canada Centre for Remote Sensing/Centre Canadien de télédétection
Natural Resources Canada/Ressources Naturelles Canada
http://atlas.gc.ca
Tel./Tél.: (613) 992-4902
Fax: (613) 947-2410