On Thu, 24 May 2001, Casey Duncan wrote:

Works great for me. Perhaps you are using a Vocabulary that has Globbing turned off?

I'm not sure, how do I check?

This query works:

wil?car*

This doesn't:

(wil?car* or something else) and (word1 and word2)

I can't see that the query-parsers in UnTextIndex.py transforms them differently, but I might be missing something obvious.

On Thu, 24 May 2001, Erik Enge wrote:

This query works:

wil?car*

This doesn't:

(wil?car* or something else) and (word1 and word2)

If the first works, then you are using a globbing vocabulary. The second one should work, but maybe there is a bug. Or perhaps your search criteria is so strict that you are getting no results.

I can't see that the query-parsers in UnTextIndex.py transforms them differently, but I might be missing something obvious.

There's _nothing_ obvious in that particular chunk of code.

-Michel

On Thu, 24 May 2001, Erik Enge wrote:

Good, then it's just not me. Is the overall design philosophy for ZCatalog/Catalog/SearchIndex documented anywhere? (By the way, from lib/python/SearchIndex/TextIndex.py, what is sws and cv3?)

I'm trying to get a knot of knowledge into my head by studying the SearchIndex modules. I'm writing more or less what's happening out on http://wiki.async.com.br/index.php?SearchIndex but I'm not moving so fast, as the code isn't too trivial.

If you want to stop by and help, by all means. :-)

Take care, -- //\ Christian Reis, Senior Engineer, Async Open Source, Brazil ~/~ http://async.com.br/~kiko/ | [+55 16] 274 4311

On Thu, 24 May 2001, Michel Pelletier wrote:

I don't think you are using a globbing vocabulary.

But globbing works for other queries. In the same catalog.

If you are not using a glob vocab, I suspect it stripped out the ? and is hitting on 'eri'. Do you have that word anywhere?

I tried searching for:

eri

and that gave me four results. No globbing, then?

Then again, where did you get these objects? If you were looking at the wrong point in the code, the wildcards may not have been expanded yet.

Could be it...

...

[['erik', '...', '...', '...', 'enge']]

Where do you see this?

I can't reproduce it right now, I'll let you know if I see it again.

Make sure you are using a globbing vocab. Note that you can't change a catalog's vocabulary once the catalog is made, so you have to make a new catalog.

How can I change it for a new one?

On Tue, 29 May 2001, Erik Enge wrote:

On Thu, 24 May 2001, Michel Pelletier wrote:

the unsplitted, unl and unq are my debug flags, but you can see what happens: without parens the '*' has it's desired effect, with, it doesn't.

Got a clue? Is this my bug, or ZCatalog's?

Must be ZCatalog's. I'm guessing the paren matching takes a different code path that doesn't expand wildcards.

-Michel

On Wed, 30 May 2001, Erik Enge wrote:

I'm going bug hunting...

I'm back :)

I think I found the bug. In lib/python/SearchIndex/GlobbingLexicon.py in the query_hook() method. It seems to say that: "if I can't find a '*' or a '?' in the word, then go to else-clause", where the else-clause says sodd off.

Since it iterates over the query, 'word' is actually a list if you use parens in your query, and you won't find any wildcards there. I think.

Add a dash of recursiveness, and it seems to be solved (for me):

def erik_hook(self, q): "doc string" words = [] for w in q: if ( (self.multi_wc in w) or (self.single_wc in w) ): wids = self.get(w) for wid in wids: if words: words.append(Or) words.append(wid) else: words.append(self.erik_hook(w)) return words or ['']

def query_hook(self, q): """expand wildcards""" words = [] for w in q: if ( (self.multi_wc in w) or (self.single_wc in w) ): wids = self.get(w) for wid in wids: if words: words.append(Or) words.append(wid) else: words.append(self.erik_hook(w))

Not really tested, but it seems to work. This might have been resolved in CVS, I don't know, should I post it as a bug?

On Wed, 30 May 2001, Chris McDonough wrote:

Thanks for tracking this down...

No worries :)

If you're so inclined, please put this in the Collector [...] so it doesn't get dropped on the floor.

Done: URL:http://classic.zope.org:8080/Collector/2262/view/

On Wed, 30 May 2001, Andre Schubert wrote:

Is it right, the the browser send a request and got the response when the site is completly rendered( all queries executed ) ? If yes, how can i directly write to the client. First all headers, and after every query send the data to the client, if i can do this, then there will be no timeout.

RESPONSE.write

--RDM

On Thu, 31 May 2001, Andre Schubert wrote:

I tested with lynx. If i type http://somewhere.com/foo/test i got no response because timeout, this means, that RESPONSE.setStatus and the first RESPONSE.write are sent back to the client if the body processing is done, but i would send every command as it is processed back to the client. Or is it my Zope( 2.2.4 ) on Immunix 6.2 RedHat.

I presume you tested it first without the large processing to make sure the method was otherwise working.

I haven't used RESPONSE.write myself. I know that others on this list have, so hopefully someone will chime in with a working example or a debuggin suggestion. Of course, it's always possible that streaming got broken at some point; I'm not sure that it gets used by very many people so breakage may take a while to get noticed...

--RDM

Erik Enge wrote:

On Wed, 30 May 2001, Erik Enge wrote:

...

I'm going bug hunting...

I'm back :)

I think I found the bug. In lib/python/SearchIndex/GlobbingLexicon.py in the query_hook() method. It seems to say that: "if I can't find a '*' or a '?' in the word, then go to else-clause", where the else-clause says sodd off.

Since it iterates over the query, 'word' is actually a list if you use parens in your query, and you won't find any wildcards there. I think.

Add a dash of recursiveness, and it seems to be solved (for me):

def erik_hook(self, q):
    "doc string"
    words = []
    for w in q:
        if ( (self.multi_wc in w) or
             (self.single_wc in w) ):
            wids = self.get(w)
            for wid in wids:
                if words:
                    words.append(Or)
                words.append(wid)
        else:
            words.append(self.erik_hook(w))
    return words or ['']

def query_hook(self, q):
    """expand wildcards"""
    words = []
    for w in q:
        if ( (self.multi_wc in w) or
             (self.single_wc in w) ):
            wids = self.get(w)
            for wid in wids:
                if words:
                    words.append(Or)
                words.append(wid)
        else:
            words.append(self.erik_hook(w))

Not really tested, but it seems to work. This might have been resolved in CVS, I don't know, should I post it as a bug?

Erik,

I'm afraid that your patch does not solve all the problems you mentioned in an earlier mail.

You are right that the implementation of query_hook in Zope 2.3.2 and 2.4.0b1 cannot handle words with wildcards in nested lists, but your patch will lead to endless recursion, if you enter the most simple query: just one word without wildcards. In this case, "if ( (self.multi_wc in w)..." evaluates to false, hence self.erik_hook is call for this word, where "if ( (self.multi_wc in w)..." is again false, and erik_hook is called again...

The statement "q = parse(s)" in UnTextIndex.query (and PositionIndex.query) before the call to query_hook can return nested lists, so query_hook must be aware of this.

This can be done with:

def query_hook(self, q): """expand wildcards""" words = [] for w in q: if type(w) is type([]): words.append(self.query_hook(w)) else: if ( (self.multi_wc in w) or (self.single_wc in w) ): wids = self.get(w) for wid in wids: if words: words.append(Or) words.append(wid) else: words.append(w) # if words is empty, return something that will make textindex's # __getitem__ return an empty result list return words or ['']

You also mentioned the strange results of queries like "eri* and enge". These are caused by another bug in query_hook:

The results from the wildcard expansion are simply inserted into the result list. Example: "ab* and xyz" may be expended by query_hook into

['aba', 'or', 'abb', 'or', 'abc', 'and', 'xyz']

Since UnTextIndex.evaluate looks first for 'and' operators, this is eqivalent to

['aba', 'or', 'abb', 'or', ['abc', 'and', 'xyz']]

(The funny (or confusing) side effect is that "ab* and xyz" may return different results compared with "xyz and ab*", because "aba and xyz" probably gives results different from those for "abc and xyz".)

but we need a result like

[['aba', 'or', 'abb', 'or', 'abc'], 'and', 'xyz']

This version of query_hook below fixes the problem:

def query_hook(self, q): """expand wildcards""" words = [] for w in q: if type(w) is type(''): if ( (self.multi_wc in w) or (self.single_wc in w) ): wids = self.get(w) alternatives = [] for wid in wids: if alternatives: alternatives.append(Or) alternatives.append(wid) words.append(alternatives or ['']) else: words.append(w) else: words.append(self.query_hook(w)) # if words is empty, return something that will make textindex's # __getitem__ return an empty result list return words or ['']

You also mentioned the parse result

['abc', '...', '...', '...', 'def']

which you could not reproduce. Playing with Catalogs, I accidentally produced the corresponding query: '"abc ... def"', i.e., the double quotation marks are part of the query string. UnTextIndex.quotes splits the string between two quotation marks at word boundaries and inserts '...' between all words:

for i in range(1,len(splitted),2): # split the quoted region into words splitted[i] = filter(None, split(splitted[i]))

# put the Proxmity operator in between quoted words for j in range(1, len(splitted[i])): splitted[i][j : j] = [ Near ]

I think that UnTextIndex.quotes should remove all query operators contained in the search string, before the '...' operator is inserted, otherwise the search results will be quite fancy.

Abel