On Sun, 15 Aug 2004 22:35:00 +0200, Dieter Maurer <dieter@handshake.de> wrote:
Mohsen Moeeni wrote at 2004-8-14 14:19 +0430:
While searching a catalog for a FieldIndex which has a value out of ASCII range, one would encounter the following error:
'ascii' codec can't decode byte 0xd9 in position 0: ordinal not in range(128)
Look at the traceback. Always do this when you get errors!
I did. For others who might want to see the traceback I wrote an URL to another message on the list, because tracebacks make an email ugly :-)
Almost surely, your index contains a non ASCII and non unicode value. When the index lookup compares the keys, the non ASCII value is converted into unicode (as there is another unicode operand) and this fails.
I must admit, the word ``Unicode`` is not very thoughtful in my email. It's a normal but ``utf-8``-encoded string which contains non-ASCII characters. So you are right.
You can use a "Managable FieldIndex". It allows you to convert all values to unicode with your chosen encoding (specifying the encoding could be easier -- maybe in the next release).
Okay I tried for some hours to make ManagableIndex work for me. Firstly, I read the documentation. I found out I have 3 points to do this conversion. None of them worked: * ``python: value.decode('utf-8')`` as the ValueProvider Normalizer raised no exception and I could find the object in catalog. However the object had no value under the interested index. * ``python: value.decode('utf-8')`` as Term Prenomalizer and Normalizer raised an ``AttributeError`` with value `` 'unicode' object has no attribute 'decode'`` upon creating the object instance. See end of the email for the traceback. I do not know why the Index term is converted to ``Unicode`` when it comes to Term Prenormalization. My code (which is an Archetypes product) does nothing to make this happen. So I wonder what I am missing. I thought Term Prenomalizer would be the ideal place to make the conversion happen. Because, If I understand it right, the conversion will also apply to the terms provided for searching the index (``_apply_index`` method if I am right). Cheers, Mohsen, ------ 2004-08-17T18:40:24 ERROR(200) SiteError http://mysite.com/plone/createObject Traceback (most recent call last): File "/usr/local/zope270/lib/python/ZPublisher/Publish.py", line 100, in publish request, bind=1) File "/usr/local/zope270/lib/python/ZPublisher/mapply.py", line 88, in mapply if debug is not None: return debug(object,args,context) File "/usr/local/zope270/lib/python/ZPublisher/Publish.py", line 40, in call_object result=apply(object,args) # Type s<cr> to step into published object. File "/usr/local/zope270/lib/python/Products/CMFFormController/FSControllerPythonScript.py", line 88, in __call__ result = FSControllerPythonScript.inheritedAttribute('__call__')(self, *args, **kwargs) File "/usr/local/zope270/lib/python/Products/CMFFormController/Script.py", line 141, in __call__ return BaseFSPythonScript.__call__(self, *args, **kw) File "/usr/local/zope270/lib/python/Products/CMFCore/FSPythonScript.py", line 104, in __call__ return Script.__call__(self, *args, **kw) File "/usr/local/zope270/lib/python/Shared/DC/Scripts/Bindings.py", line 306, in __call__ return self._bindAndExec(args, kw, None) File "/usr/local/zope270/lib/python/Shared/DC/Scripts/Bindings.py", line 343, in _bindAndExec return self._exec(bound_data, args, kw) File "/usr/local/zope270/lib/python/Products/CMFCore/FSPythonScript.py", line 160, in _exec result = apply(f, args, kw) File "Script (Python)", line 16, in createObject File "/usr/local/zope270/lib/python/Products/CMFCore/PortalFolder.py", line 363, in invokeFactory , kw File "/usr/local/zope270/lib/python/Products/CMFCore/TypesTool.py", line 709, in constructContent ob = apply(info.constructInstance, (container, id) + args, kw) File "/usr/local/zope270/lib/python/Products/CMFCore/TypesTool.py", line 401, in constructInstance return self._finishConstruction(ob) File "/usr/local/zope270/lib/python/Products/CMFCore/TypesTool.py", line 299, in _finishConstruction ob.reindexObject(idxs=['portal_type', 'Type']) File "/usr/local/zope270/lib/python/Products/Archetypes/CatalogMultiplex.py", line 60, in reindexObject c.catalog_object(self, self.__url(), idxs=lst) File "/usr/local/zope270/lib/python/Products/ZCatalog/ZCatalog.py", line 513, in catalog_object update_metadata=update_metadata) File "/usr/local/zope270/lib/python/Products/ZCatalog/Catalog.py", line 381, in catalogObject blah = x.index_object(index, object, threshold) File "/usr/local/zope270/lib/python/Products/ManagableIndex/ManagableIndex.py", line 203, in index_object val= self._evaluate(obj) File "/usr/local/zope270/lib/python/Products/ManagableIndex/ManagableIndex.py", line 441, in _evaluate return self._standardizeValue(v,object) File "/usr/local/zope270/lib/python/Products/ManagableIndex/ManagableIndex.py", line 445, in _standardizeValue return self._standardizeTerm(value,object,1) File "/usr/local/zope270/lib/python/Products/ManagableIndex/ManagableIndex.py", line 401, in _standardizeTerm value = self._prenormalizeTerm(value, object) File "/usr/local/zope270/lib/python/Products/ManagableIndex/ManagableIndex.py", line 424, in _prenormalizeTerm return normalizer._normalize(value, object) File "/usr/local/zope270/lib/python/Products/ManagableIndex/Evaluation.py", line 130, in _normalize return evaluator._evaluate(value,object) File "/usr/local/zope270/lib/python/Products/ManagableIndex/Evaluation.py", line 86, in _evaluate v= EvalAndCall.inheritedAttribute('_evaluate')(self,value,object) File "/usr/local/zope270/lib/python/Products/ManagableIndex/Evaluation.py", line 58, in _evaluate return expr(context) File "/usr/local/zope270/lib/python/Products/PageTemplates/ZRPythonExpr.py", line 47, in __call__ return eval(code, g, {}) File "Python expression "value.decode('utf-8')"", line 1, in <expression> AttributeError: 'unicode' object has no attribute 'decode'