[Zope-dev] Re: Showstopper UnicodeDecodeError on Zope import???

[Nick Bower]

Wow Dieter - that's a really concise explanation.  It'll allow us to fix 
the product knowing this background.  Thanks!

Dieter Maurer wrote:
> Nick Bower wrote at 2004-10-8 16:41 +0200:
> 
>>...
>>Module Products.ZCTextIndex.Lexicon, line 69, in sourceToWordIds
>>Module Products.ZCTextIndex.Lexicon, line 135, in _getWordIdCreate
>>
>>UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 5: 
>>orginal not in range(128).
> 
> 
> In your lexicon operation a unicode and a non-unicode string is
> put together (this can happen internally during BTree traversal).
> 
> Whenever such a thing happens, Python tries to convert the
> non unicode to unicode -- using its default encoding.
> This fails as the non unicode string contains bytes not convertable
> this this encoding.
> 
> In a later message you reported that setting Python's default
> encoding to "utf-8" gave you an unexpected end exception.
> This means that your non unicode string is not utf-8 encoded.
> 
> 
> You should use as default encoding the encoding that is
> used for your non unicode strings.
> 
> If you do not know it, use an encoding that can map any 8 bit byte.
> Windows has a few of them (called "cpXXX" (for CodePage);
> I do not know the correct XXX).
>