Wow Dieter - that's a really concise explanation. It'll allow us to fix the product knowing this background. Thanks! Dieter Maurer wrote:
Nick Bower wrote at 2004-10-8 16:41 +0200:
... Module Products.ZCTextIndex.Lexicon, line 69, in sourceToWordIds Module Products.ZCTextIndex.Lexicon, line 135, in _getWordIdCreate
UnicodeDecodeError: 'ascii' codec can't decode byte 0xef in position 5: orginal not in range(128).
In your lexicon operation a unicode and a non-unicode string is put together (this can happen internally during BTree traversal).
Whenever such a thing happens, Python tries to convert the non unicode to unicode -- using its default encoding. This fails as the non unicode string contains bytes not convertable this this encoding.
In a later message you reported that setting Python's default encoding to "utf-8" gave you an unexpected end exception. This means that your non unicode string is not utf-8 encoded.
You should use as default encoding the encoding that is used for your non unicode strings.
If you do not know it, use an encoding that can map any 8 bit byte. Windows has a few of them (called "cpXXX" (for CodePage); I do not know the correct XXX).