[Zope-CMF] [dev] using unicode in CMF 2.0 - a proposal

[yuppie]

Hi!


A major goal of CMF 2.0 is to make it easier to use Zope3/Five features 
in CMF. One issue that is not resolved so far is the fact that the CMF 
uses encoded strings while Zope3/Five uses unicode strings. That calls 
for trouble, especially in page templates because they can't handle 
non-ascii encoded strings mixed with unicode.

One source of unicode strings are Message objects and translations. Both 
Localizer and Placeless Translation Service monkey patch page templates 
to work around the unicode problem. All unicode strings are encoded with 
the page encoding before joining them with other strings. This seems to 
work quite well, but it's nasty because of the monkey patching.


Proposal:

1.) All skin templates that have to deal with non-ascii content don't 
access the content objects directly. As it is already the case for most 
forms they should get their content from a skin script instead.

2.) The return value of all those scripts is converted using 
CMFDefault.utils.toUnicode and the 'default_charset' of the site.


This is a sledgehammer method and customized skins have to be changed in 
the same way. But after that we should be out of the woods:

- We can start to convert step by step other parts of the CMF to unicode 
and to use Zope3/Five components that return unicode.

- We should be prepared for the upcoming page template changes.


I know this has drawbacks, but I don't see a better solution. If there 
are no objections I'd like to work on this before the CMF 2.0 beta.


Cheers,

	Yuppie