[Zope-dev] IUnicodeEncodingConflictResolver moronosity

[Chris Withers]

On 21/09/2010 19:32, Hanno Schlichting wrote:
> Because there's not all that many people left using plain Zope 2 and
> upgrading to new versions.

Well, or they assume they must be doing something wrong because of this 
default.

> The rest is using Plone, which has its own version of this:
>
> def _unicode_replace(structure):
>      if isinstance(structure, str):
>          text = structure.decode('utf-8', 'replace')
>      else:
>          text = unicode(structure)
>      return text
>
> class UTF8EncodingConflictResolver(object):
>      implements(IUnicodeEncodingConflictResolver)
>
>      def resolve(self, context, text, expression):
>          return _unicode_replace(text)

Right, although the 'replace' is a lame cop out too ;-)

> We can do this, as we force manage_page_charset and
> default-zpublisher-encoding to be utf-8 and only support utf-8 and
> Unicode in the database.

Fundamentally, the developer must know what encoding is being used in 
their data. So, they should be able to register an appropriate decoder 
BUT...

> Zope 2 itself tries to support arbitrary encodings, at which point it
> can only guess badly or fail.

...having this default, which makes ABSOLUTELY NO SENSE in this 
universe, or likely any other, is fucking crazy.

So, would anyone object if I change the default, on both trunk and the 
2.12 branch to some variant of:

class BasicEncodingConflictResolver(object):
      implements(IUnicodeEncodingConflictResolver)

      def resolve(self, context, text, expression):
          logging.warn('You should register an '
                       'IUnicodeEncodingConflictResolver that matches '
                       'your content')
          return text.decode('ascii','replace')

...and delete that devil spawn turd that is PreferredCharsetResolver?

cheers,

Chris

-- 
Simplistix - Content Management, Batch Processing & Python Consulting
            - http://www.simplistix.co.uk