[ZCM] [ZC] 1343/ 5 Accept "Multilingual search gives a wrong results."

[Collector: Zope Bugs, Features, and Patches ...]

Issue #1343 Update (Accept) "Multilingual search gives a wrong results."
 Status Accepted, Catalog/bug medium
To followup, visit:
  http://collector.zope.org/Zope/1343

==============================================================
= Accept - Entry #5 by ajung on May 28, 2004 9:20 am

 Status: Pending => Accepted

 Supporters added: ajung

Searching for  руль

has the following effect in _apply_index():

The repr(query_string) is: '\xd1\x80\xd1\x83\xd0\xbb\xd1\x8c'
which looks like valid utf-8 (assuming that this is the utf8
representation of the query.

The QueryParser spits out: PhraseNode(['\xd1', '\xd1', '\xd0', '\xd1'])
which is completely wrong.

However I have little knowledge of the QueryParser maybe
someone from the ZCTextIndex should look at this.



________________________________________
= Comment - Entry #4 by anaiw on May 19, 2004 4:45 am

To be shorter, the following piece of text in utf-8 encoding. 
---
решают проблему
---
returned as a result for "руль" word.
I think, the problem is in splitter...
________________________________________
= Comment - Entry #3 by anaiw on May 19, 2004 4:27 am

sorry I forgot. 
Create a new Plone site or use existing one. 
Then add a document, written in Russian with following text  inside: 
---
<p>Городские операторы 
беспроводных и проводных сетей решают проблему "последней мили", 
осуществляя доступ к магистральным операторам абонентов, для которых 
кабельный доступ невозможен или невыгоден.</p>
---
Call this document 'test' for example and with the 'test' id. (it's doesn't matter, actually)
 
Then use Plone search to find Russian words "руль" or "удар", for instance. (These text hasn't these words inside)
It will give you as a result this document. 
I think, it will find give you this document in search results with another Russian words to search for. These words where given just for instance. 
Documents with the other pieces of text will be given as search results for other words. 
________________________________________
= Comment - Entry #2 by ajung on May 19, 2004 3:41 am

Please give us a reasonable description how to reproduce this error.


________________________________________
= Request - Entry #1 by anaiw on May 19, 2004 3:34 am

Under Zope 2.7 I have Plone 2.0-Final installed and my site is based on it. As I know Plone use Zope search engine so I write here. 
The problem is that when I try to search on Russian the word that is definitely doesn't present on my site I got some documents in search results. I use ZCTextIndex type index to index searchable text.  Though search works fine for english. It happens not only for exact word, it happens for different words in Russian which aren't appear as in searchable text as in metadata. But it searches fine the words which are really present on the site.
I have tried different options for ZCTextIndex Lexicon - no effect. 

==============================================================