[Zope] 1- Unicode support for TextIndexNG, 2- Umlauts nor supported by QueryParser?

[Andreas Jung]

TextIndexNG is not yet fully unicodeaware so I recommend avoid
using the UnicodeSplitter. To can use the standard ZopeSplitter
instead when you are using ISO-8859-1 or -15. But you must
have configured your locale settings.

-aj
----- Original Message -----
From: "Juergen R. Plasser / Hexagon" <plasser@hexagon.at>
To: <zope@zope.org>
Sent: Tuesday, June 11, 2002 10:57
Subject: [Zope] 1- Unicode support for TextIndexNG, 2- Umlauts nor supported
by QueryParser?


1 - When I try to set Unicode support for an TextIndexNG index I get the
following error when indexing: TypeError: object of type 'module' is not
callable

Full trace:

Traceback (innermost last):
  File /usr/local/zope/2-5-0/lib/python/ZPublisher/Publish.py, line 150, in
publish_module
  File /usr/local/zope/2-5-0/lib/python/ZPublisher/Publish.py, line 114, in
publish
  File /usr/local/zope/2-5-0/lib/python/Zope/__init__.py, line 159, in
zpublisher_exception_hook
    (Object: )
  File /usr/local/zope/2-5-0/lib/python/ZPublisher/Publish.py, line 98, in
publish
  File /usr/local/zope/2-5-0/lib/python/ZPublisher/mapply.py, line 88, in
mapply
    (Object: manage_reindexIndex)
  File /usr/local/zope/2-5-0/lib/python/ZPublisher/Publish.py, line 39, in
call_object
    (Object: manage_reindexIndex)
  File /usr/local/zope/2-5-0/lib/python/Products/ZCatalog/ZCatalog.py, line
455, in manage_reindexIndex
    (Object: Catalog)
  File /usr/local/zope/2-5-0/lib/python/Products/ZCatalog/ZCatalog.py, line
441, in reindexIndex
    (Object: Catalog)
  File /usr/local/zope/2-5-0/lib/python/Products/ZCatalog/ZCatalog.py, line
480, in catalog_object
    (Object: Catalog)
  File /usr/local/zope/2-5-0/lib/python/Products/ZCatalog/Catalog.py, line
367, in catalogObject
  File
/usr/local/zope/2-5-0/lib/python/Products/TextIndexNG/TextIndexNG.py, line
305, in index_object
    (Object: Ort)
TypeError: object of type 'module' is not callable

2 -  When I try to search a TextIndexNG field with umlauts (here the german

ü) I get the following error:

Error Type: QueryParserError
Error Value: Illegal character 'ü'

Traceback (innermost last):
  File /usr/local/zope/2-5-0/lib/python/ZPublisher/Publish.py, line 150, in
publish_module
  File /usr/local/zope/2-5-0/lib/python/ZPublisher/Publish.py, line 114, in
publish
  File /usr/local/zope/2-5-0/lib/python/Zope/__init__.py, line 159, in
zpublisher_exception_hook
    (Object: Catalog)
  File /usr/local/zope/2-5-0/lib/python/ZPublisher/Publish.py, line 98, in
publish
  File /usr/local/zope/2-5-0/lib/python/ZPublisher/mapply.py, line 88, in
mapply
    (Object: report_html)
  File /usr/local/zope/2-5-0/lib/python/ZPublisher/Publish.py, line 39, in
call_object
    (Object: report_html)
  File /usr/local/zope/2-5-0/lib/python/OFS/DTMLMethod.py, line 127, in
__call__
    (Object: report_html)
  File /usr/local/zope/2-5-0/lib/python/DocumentTemplate/DT_String.py, line
473, in __call__
    (Object: report_html)
  File /usr/local/zope/2-5-0/lib/python/DocumentTemplate/DT_In.py, line
437, in renderwb
    (Object: Catalog)
  File /usr/local/zope/2-5-0/lib/python/Products/ZCatalog/ZCatalog.py, line
579, in searchResults
    (Object: Catalog)
  File /usr/local/zope/2-5-0/lib/python/Products/ZCatalog/Catalog.py, line
576, in searchResults
  File /usr/local/zope/2-5-0/lib/python/Products/ZCatalog/Catalog.py, line
470, in _indexedSearch
  File
/usr/local/zope/2-5-0/lib/python/Products/TextIndexNG/TextIndexNG.py, line
398, in _apply_index
    (Object: Ort)
  File
/usr/local/zope/2-5-0/lib/python/Products/TextIndexNG/TextIndexNG.py, line
425, in query
    (Object: Ort)
  File /usr/local/zope/2-5-0/lib/python/Products/TextIndexNG/BaseParser.py,
line 29, in __call__
  File
/usr/local/zope/2-5-0/lib/python/Products/TextIndexNG/parsers/PyQueryParser
/PyQueryParser.py, line 147, in parse
  File
/usr/local/zope/2-5-0/lib/python/Products/TextIndexNG/parsers/PyQueryParser
/yacc.py, line 195, in parse
  File
/usr/local/zope/2-5-0/lib/python/Products/TextIndexNG/parsers/PyQueryParser
/lex.py, line 320, in realtoken
  File
/usr/local/zope/2-5-0/lib/python/Products/TextIndexNG/parsers/PyQueryParser
/PyQueryParser.py, line 60, in t_error
QueryParserError: (see above)

The index field is configured like this:

Stemmer disabled
Splitter ZopeSplitter
Splitter: casefolding enabled
Splitter: index numbers disabled
Splitter: index single characters disabled
Splitter: max. length of splitted words 64
Default query parser PyQueryParser
Compressed lists disabled
Use converters disabled
Near distance  5
Similarity search disabled

Any hints? What can I do to force indexing of umlauts?`

Regards,
Juergen


_______________________________________________
Zope maillist  -  Zope@zope.org
http://lists.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 http://lists.zope.org/mailman/listinfo/zope-announce
 http://lists.zope.org/mailman/listinfo/zope-dev )