[Zope] Searching Catalog for Unicode Values of FieldIndex

[Mohsen Moeeni]

I am disappointed to find out that FieldIndex does not support Unicode values.
While searching a catalog for a FieldIndex which has a value out of ASCII range,
one would encounter the following error:

'ascii' codec can't decode byte 0xd9 in position 0: ordinal not in range(128)

The interesting point is that the object has been indexed in the catalog with
the proper Unicode value but you can not search the index with this value.

I wonder why a very old message at:
http://mail.zope.org/pipermail/zope/2001-November/104636.html suggests
that FieldIndex supports Unicode.

I could also find a message reporting the same problem (*with full traceback*)
on the list. Andreas Jung propose to change the encoding of
Python to solve it:

http://mail.zope.org/pipermail/zope/2003-February/131252.html

While changing the default encoding can suppress the exception, the search
results are empty making the catalog useless. (I changed it to iso-8859-2).
Anyway I am not sure, I like to change the encoding of the python just to
get the desired value for FieldIndex.

I wonder if there is any chance to use FieldIndex with Unicode values.
Perhaps there are some other Index products which support Unicode values.
Or should one write his own Index if he wants a Unicode aware FieldIndex.

Cheers,
Mohsen,