[Zope] SUMMARY: strange unicode behaviour, unicode and ZCTextIndex

[Toby Dickenson]

On Friday 25 July 2003 19:29, Dieter Maurer wrote:

>  > I find this inconvenient too, but that is the way that the Python
>  > language is defined. There is code inside Zope, and other libraries,
>  > that assumes Python behaves this way.
>
> I do not want to believe this. Can you give an example?

pDocumentTemplate contains this code below which relies on 'string'.join 
raising a UnicodeError exception if a list contains a mix of unicode strings 
and non-ascii plain strings:

def join_unicode(rendered):
    """join a list of plain strings into a single plain string,
    a list of unicode strings into a single unicode strings,
    or a list containing a mix into a single unicode string with
    the plain strings converted from latin-1
    """
    try:
        return ''.join(rendered)
    except UnicodeError:
        # A mix of unicode string and non-ascii plain strings.
        # Fix up the list, treating normal strings as latin-1
        rendered = list(rendered)
        for i in range(len(rendered)):
            if type(rendered[i]) is StringType:
                rendered[i] = unicode(rendered[i],'latin-1')
        return u''.join(rendered)

[note that actually Zope uses the "C optimised" cDocumentTemplate alternative, 
which contains equivalent logic]

> Due to this default encoding, I save me from myriads of
> encoding errors and make interactive debugging feasible.
> Surely, you will understand, that I do not want to add
> an "encode('iso-8859-1')" to any value I output with "print"
> during interactive debugging.

Yes. I work in a mostly utf-8 world, and I originally wanted Python's unicode 
support to work somewhat like you are using it when first pioneering unicode 
in Zope. Guido convinced me otherwise:

http://aspn.activestate.com/ASPN/Mail/Message/i18n-sig/581409
(last paragraph in particular)

> I did not met any library that has had problems with this --
> neither Zope nor any other Python library I am using.

I think that is similar to how many people used those pentiums with the fdiv 
bug without noticing a problem. 

I hope this helps,

-- 
Toby Dickenson - http://www.geminidataloggers.com/people/tdickenson

Want a job like mine?  http://www.geminidataloggers.com/jobs for Software
Engineering jobs at Gemini Data Loggers in Chichester, West Sussex, England