[Zope-dev] KeywordIndex errors: This should not happen.

[Jean Jordaan]

Hi all

Our app has been hitting these errors for months:

2002-09-30T11:06:27 ERROR(200) KeywordIndex unindex_object could not remove 
documentId -444153566 from index ComponentVersions.  This should not happen.
Traceback (innermost last):
   File 
/usr/local/zope/2-5-1/lib/python/Products/PluginIndexes/common/UnIndex.py, line 
168, in removeForwardIndexEntry
     (Object: ComponentVersions)
KeyError: -444153566

Here is the relevant piece Zope code::

     def removeForwardIndexEntry(self, entry, documentId):
         """Take the entry provided and remove any reference to documentId
         in its entry in the index."""
         global _marker
         indexRow = self._index.get(entry, _marker)
         if indexRow is not _marker:
             try:
                 indexRow.remove(documentId)
                 if not indexRow:
                     del self._index[entry]
                     try: self.__len__.change(-1)
                     except AttributeError: pass # pre-BTrees-module instance
             except AttributeError:
                 # index row is an int
                 del self._index[entry]
                 try: self.__len__.change(-1)
                 except AttributeError: pass # pre-BTrees-module instance
             except:
                 import pdb; pdb.set_trace() #DBG

I inserted that 'set_trace' just before the error is logged. At this
point, the app state looks like this::

     (Pdb) self
     <KeywordIndex instance at 9544940>
     (Pdb) p self.id
     'ComponentVersions'
     (Pdb) entry
     '44.131'
     (Pdb) p indexRow
     <IITreeSet object at 0x1355a3d0>
     (Pdb) len(indexRow.keys())
     566
     (Pdb) indexRow.remove(documentId)
     *** KeyError: -444153566

     (Pdb) [self._index.keys()[n] for n in range(0,69)]
     ['', ' 2.13', '1', '1.14', '1.22', '1.25', '1.3', '1.6', '1.8.7',
     '1.9.8', '17.9', '2.0.6', '2.1', '2.15', '2.2.1', '2.2.3m', '2.2.7',
     '2.2.7m', '2.3.4m', '2.4', '2.4.2m', '2.4.3m', '2.4.4m', '2.4.5',
     '2.5', '2.6.6m', '2.7.7m', '2.8.5m', '2.8.6m', '2.8.7m', '2.9',
     '20.13-37', '22.13-42', '22.42-47', '22.5-42', '24.12-49',
     '24.13-49', '24.19-49', '4.4', '4.9', '40.54e', '41.184', '42.95',
     '43.11', '43.96', '43.96 ', '44.120', '44.131', '45.11', '7.3',
     '7.7', '8.0', '8.1', '8.2', '8.4', '8.4.1', '8.5.9', '8.7.2m',
     '8.7.6m', '8.8.9m', '8.9.6m', '8.9.7m', '9.0', '9.12', '9.2', '9.3',
     '9.4   ', '9.8', 'x']
     (Pdb) [indexRow.keys()[n] for n in range(0, len(indexRow.keys()))]
     [-1948802264, -1018656130, -444153568, -111852827, -32258423,
     -32258421, -32258398, -32258397, -32258396, 226891955, (and so on)

     (Pdb) catalog = self.restrictedTraverse('Catalog')
     (Pdb) catalog.getpath(-444153566)
     '/Licenses/LicenseDocs/License_377'
     (Pdb) catalog.getIndexDataForRID(-444153566)['ComponentVersions']
     ['2.4', '8.9.7m', '44.131', '44.131', '2.4.3m', '4.9', '9.12',
     '24.13-49', '1.25', '2.15', '9.3', '2.8.6m']

Rephrasing that, I'd say: we're trying to remove the RID -444153566
from the list returned by: indexRow.keys()['44.131']

That RID is absent from that list, which raises a KeyError that
"should not happen". There *is* an entry in the Catalog for that RID,
which *has* '44.131' in the list under the 'ComponentVersions' key in
the dict returned by getIndexDataForRID.

Did the RID get removed from the KeywordIndex list of RIDs prematurely?
Or did it never get inserted? Can anyone shed any enlightenment? Where
else should I be looking?

-- 
Jean Jordaan
Upfront Systems                         http://www.upfrontsystems.co.za