[Zope] zcatalog -- returning context of hits on fulltext

[R. David Murray]

On Mon, 14 Aug 2000, Jimmie Houchin wrote:
> I may be clueless and out of my league here and I haven't read the
> sources so I don't know... Well enough of a disclaimer. :)

I *have* read the ZCatalog/SearchIndex sources, but I don't understand
this part of it yet (or really that much of it at all!).  I think
we're getting into zope-dev terratory here...

> Is there anything in there which can provide the seek or byte position
> of the hit within text object? If so, it shouldn't be too difficult to
> read X bytes before and after the position and thereby provide what your
> looking for.

The standard TextIndex implementation records a notion of "position" for
each occurence of each word indexed.  I *think* this position is a word
count position, but I'm not sure.  Part of the code references a
'row', but it isn't at all clear that that has any relationship to
a source record.  If it is a word count, the other thing you'd need to
check would be whether it is a word count before or after splitter
activity.  I think it's the latter, which makes things more complicated.
Or just means you have to use more fuzz in your context <grin>.

> This would be nice to have out of the box.

The TextIndex 'position' information is intended to be used for
the 'near' operator (...) (so you can search on multiple words
"close" to each other for some definition of close).  You could
also use it to enforce word order (Maybe the "" operator does
that?).  Currently I think the result of applying the near operator
is used to adjust the "weight" of the index match, which affects
the order of the results returned.  (I haven't tested to see if
any of this works!)

So, the basic information you are looking for is there in some sense
to establish the position, but you'd still have to retrieve the
original sentences from the object itself, or from a full-text
metadata field.  Both of these are going to be memory intensive
operations.  If you index based on, say, individual lines, you'd
loose some of the the benefits of the near operator, though.  So
I'd say indexing based on paragraphs would probably be your best
approach.  This would also help mask position errors introduced if
the word count is indeed post-splitter.  Of course, you'll have to
decend to python to get access to the methods that will return the
actual position information.  But at least the code to record it
is already there.

Take a look at lib/python/SearchIndex/TextIndex.py for source
enlightenment.

--RDM