[Zope-CMF] [dev] i18n Message objects: encoding issues

[yuppie]

Hi!


A while ago I converted many message strings to i18n Message objects. 
Unfortunately I didn't think enough about side effects and now the CMF 
trunk (CMF 2.0) is in a rather broken state:

Message objects are unicode objects, and unless you use a Localizer or 
PTS that returns non-unicode strings as translation you get 
UnicodeDecodeErrors if your page template also contains non-ascii content.


I can't see a simple solution for this and any input is welcome. In 
general I see these options:


1.) Stick to non-unicode strings:

Avoiding everything that might return unicode strings. Reverting the 
i18n changes I made.

I don't like that option. Zope 3 uses unicode in many places and sooner 
or later we have to find a way to deal with unicode in page templates.


2.) Switch completely to unicode in CMF 2.0:

Sounds like a too big change, don't think we should try that.


3.) Use encoded strings and unicode side by side:

This requires to know the encoding of the encoded strings. 
'default_charset' should provide that information.


3.1.) Encode unicode strings before passing them to page templates:

The unicode strings are only returned by newly added parts of the code. 
So it might be easier to find all the places where a conversion is 
necessary. On the other hand Message objects are also unicode, so we 
would have to translate them before passing them to the templates.


3.2.) Decode non-unicode strings before passing them to page templates:

The RSS page works already that way. We need to touch a lot of skin code 
to implement this, but it looks more future proof to me. In Zope 2.10 
non-unicode input for page templates might become deprecated.


What do you think?

Cheers,

	Yuppie