[Zope] Searching Catalog for Unicode Values of FieldIndex

[Dieter Maurer]

Mohsen Moeeni wrote at 2004-8-17 19:12 +0430:
> ...
>> You can use a "Managable FieldIndex". It allows you to convert
>> all values to unicode with your chosen encoding (specifying the
>> encoding could be easier -- maybe in the next release).
>
>Okay I tried for some hours to make ManagableIndex work for me. Firstly,
>I read the documentation. I found out I have 3 points to do this conversion.
>None of them worked:
>
>* ``python: value.decode('utf-8')`` as the ValueProvider Normalizer raised no
> exception and I could find the object in catalog. However the object had
> no value under the interested index.

Apparently, you kept the "ignore exceptions" default in the 
Attribute lookup?

This means, exceptions are silently ignored.

Uncheck "ignore exceptions" to find out what goes wrong..

>* ``python: value.decode('utf-8')`` as Term Prenomalizer and Normalizer raised
>an ``AttributeError`` with value `` 'unicode' object has no attribute 'decode'``

This is a speaking error message, isn't it?

"value" is a unicode object and it does not have a "decode" attribute
(what you are looking for is "encode").

>I do not know why the Index term is converted to ``Unicode`` when it comes to
>Term Prenormalization. My code (which is an Archetypes product) does nothing
>to make this happen.
Archetypes ("StringField.set" to be precise) does this.


Keep your indexed values Unicode.
I recommend to use as prenormalizer:

  python: not isinstance(value, unicode) and unicode(value, yourCharset) or value

>I thought Term Prenomalizer would be the ideal place to make the
>conversion happen.
>Because, If I understand it right, the conversion will also apply to the terms
>provided for searching the index (``_apply_index`` method if I am right).

You are right. The Prenormalizer is executed before pattern expansion.


The next "ManagableIndex" release will provide automatical
unicode transformation with an explicit encoding argument.
However, it will happen too late for pattern expansion (as
it runs after the term normalizer).
The "Prenormalizer" is a better place.

-- 
Dieter