[Zope-dev] ZCatalog "Key Error" question

[zope-dev at rosenthaltechnologies.com]

Thank you so much for following-up with me, here.  I think you're right
about this having to do with the indexes on the ZCatalog.

The catalog has:
1.  Seven "FieldIndex" indexes.
2.  A "DateIndex" index
3.  Three "ZCTextIndex" indexes

I actually create these from python in a function that accepts the
"catalog" as a parameter during the Product Initialization process.

I have a function "catalog_add" that creates and initializes the a
ZCatalog when this product is created and initialized, so let me
actually post the sourcecode here for that function and the functions it
calls.  It's fairly short.  There are two functions of note
"catalog_add" and "addIndexes".  I'd actually be grateful if you would
take a look and let me know if you see anything.  In particular, am I
using an index that you know to have problems, or that I'm using
improperly.

class Newsdesk(Folder,Persistent,Implicit):
        meta_type = "Newsdesk"
        # ...
        def catalog_add(self):
                "setup the catalog"
                self.manage_addProduct["ZCatalog"].manage_addZCatalog("Catalog",
                "Default Catalog")
                catalog = self._getOb('Catalog')
                self.addLexicon(catalog)
                self.addIndexes(catalog)
                self.addMetas(catalog)
                catalog = self._getOb('Catalog')
                return catalog

        def addIndexes(self, cat):
                cat.addIndex('date_found', 'FieldIndex', 'date_found')
                cat.addIndex('deleted_p', 'FieldIndex', 'deleted_p')
                cat.addIndex('uri', 'FieldIndex', 'uri')
                cat.addIndex('categories_field', 'FieldIndex',
                'categories')
                cat.addIndex('categorized_p', 'FieldIndex',
                'categorized_p')
                cat.addIndex('approved_p', 'FieldIndex', 'approved_p')
                cat.addIndex('search_id', 'FieldIndex', 'search_id')
                extras = Empty()
                extras.doc_attr = 'title'
                extras.index_type = 'Okapi BM25 Rank'
                extras.lexicon_id = 'Lexicon'
                cat.addIndex('title', 'ZCTextIndex', extras)

                extras2 = Empty()
                extras2.doc_attr = 'source_name'
                extras2.index_type = 'Okapi BM25 Rank'
                extras2.lexicon_id = 'Lexicon'
                cat.addIndex('source_name', 'ZCTextIndex', extras2)

                extras3 = Empty()
                extras3.doc_attr = 'categories'
                extras3.index_type = 'Okapi BM25 Rank'
                extras3.lexicon_id = 'Lexicon'
                cat.addIndex('categories', 'ZCTextIndex', extras3)


                cat.addIndex('bobobase_modification_time', 'DateIndex')

        def addMetas(self, cat):
                "create the metadata columns"
                cat.manage_addColumn("categorized_p")
                cat.manage_addColumn("deleted_p")
                cat.manage_addColumn("search_id")
                cat.manage_addColumn("categories")
                cat.manage_addColumn("date_found")
                cat.manage_addColumn("source_name")
                cat.manage_addColumn("title")
                cat.manage_addColumn("uri")

        def addLexicon(self, cat):
                elem = []
                wordSplitter = Empty()
                wordSplitter.group = 'Word Splitter'
                wordSplitter.name = 'HTML aware splitter'
                caseNormalizer = Empty()
                caseNormalizer.group = 'Case Normalizer'
                caseNormalizer.name = 'Case Normalizer'
                stopWords = Empty()
                stopWords.group = 'Stop Words'
                stopWords.name = 'Remove listed and single char words'
                elem.append(wordSplitter)
                elem.append(caseNormalizer)
                elem.append(stopWords)
                cat.manage_addProduct['ZCTextIndex'].manage_addLexicon('Lexicon',
                'Default Lexicon', elem)



On Sun, 3 Sep 2006 20:11:33 +0200, "Dieter Maurer" <dieter at handshake.de>
said:
> Dieter Maurer wrote at 2006-9-3 19:22 +0200:
> >zope-dev at rosenthaltechnologies.com wrote at 2006-9-1 13:29 -0400:
> >>
> >>Traceback for my Key Error (Dieter, thank you for alerting me on where
> >>to find this):
> >>
> >>Traceback (innermost last):
> >> ...
> >>   - Physical Path: /nephron_links/news_dtml
> >>  Module DocumentTemplate.DT_String, line 476, in __call__
> >>  Module DocumentTemplate.DT_In, line 465, in renderwb
> >>  Module DocumentTemplate.DT_In, line 747, in sort_sequence
> >>  Module Products.ZCatalog.Lazy, line 158, in __getitem__
> >>  Module Products.ZCatalog.Catalog, line 520, in getScoredResult
> >>KeyError: -464571725
> >
> >This does not look like a catalog corruption.
> >
> >Instead, it looks as if the catalog changed since the search was
> >done. Are you caching your search results?
> 
> 
> Sorry! I was wrong!
> 
>   It has nothing to do with a catalog modification after the search.
> 
> Looks indeed as if an index reported a document which is no longer
> indexed. Looks like an index bug. Which (types of) indexes are involved
> in your failing queries?
> 
> 
> 
> -- 
> Dieter
--------------------
Brian Rosenthal
General Partner, RoboCommerce, LLC
brian at robocommerce.com
http://www.robocommerce.com
800-644-7626