KeywordIndex errors: This should not happen.
Hi all Our app has been hitting these errors for months: 2002-09-30T11:06:27 ERROR(200) KeywordIndex unindex_object could not remove documentId -444153566 from index ComponentVersions. This should not happen. Traceback (innermost last): File /usr/local/zope/2-5-1/lib/python/Products/PluginIndexes/common/UnIndex.py, line 168, in removeForwardIndexEntry (Object: ComponentVersions) KeyError: -444153566 Here is the relevant piece Zope code:: def removeForwardIndexEntry(self, entry, documentId): """Take the entry provided and remove any reference to documentId in its entry in the index.""" global _marker indexRow = self._index.get(entry, _marker) if indexRow is not _marker: try: indexRow.remove(documentId) if not indexRow: del self._index[entry] try: self.__len__.change(-1) except AttributeError: pass # pre-BTrees-module instance except AttributeError: # index row is an int del self._index[entry] try: self.__len__.change(-1) except AttributeError: pass # pre-BTrees-module instance except: import pdb; pdb.set_trace() #DBG I inserted that 'set_trace' just before the error is logged. At this point, the app state looks like this:: (Pdb) self <KeywordIndex instance at 9544940> (Pdb) p self.id 'ComponentVersions' (Pdb) entry '44.131' (Pdb) p indexRow <IITreeSet object at 0x1355a3d0> (Pdb) len(indexRow.keys()) 566 (Pdb) indexRow.remove(documentId) *** KeyError: -444153566 (Pdb) [self._index.keys()[n] for n in range(0,69)] ['', ' 2.13', '1', '1.14', '1.22', '1.25', '1.3', '1.6', '1.8.7', '1.9.8', '17.9', '2.0.6', '2.1', '2.15', '2.2.1', '2.2.3m', '2.2.7', '2.2.7m', '2.3.4m', '2.4', '2.4.2m', '2.4.3m', '2.4.4m', '2.4.5', '2.5', '2.6.6m', '2.7.7m', '2.8.5m', '2.8.6m', '2.8.7m', '2.9', '20.13-37', '22.13-42', '22.42-47', '22.5-42', '24.12-49', '24.13-49', '24.19-49', '4.4', '4.9', '40.54e', '41.184', '42.95', '43.11', '43.96', '43.96 ', '44.120', '44.131', '45.11', '7.3', '7.7', '8.0', '8.1', '8.2', '8.4', '8.4.1', '8.5.9', '8.7.2m', '8.7.6m', '8.8.9m', '8.9.6m', '8.9.7m', '9.0', '9.12', '9.2', '9.3', '9.4 ', '9.8', 'x'] (Pdb) [indexRow.keys()[n] for n in range(0, len(indexRow.keys()))] [-1948802264, -1018656130, -444153568, -111852827, -32258423, -32258421, -32258398, -32258397, -32258396, 226891955, (and so on) (Pdb) catalog = self.restrictedTraverse('Catalog') (Pdb) catalog.getpath(-444153566) '/Licenses/LicenseDocs/License_377' (Pdb) catalog.getIndexDataForRID(-444153566)['ComponentVersions'] ['2.4', '8.9.7m', '44.131', '44.131', '2.4.3m', '4.9', '9.12', '24.13-49', '1.25', '2.15', '9.3', '2.8.6m'] Rephrasing that, I'd say: we're trying to remove the RID -444153566 from the list returned by: indexRow.keys()['44.131'] That RID is absent from that list, which raises a KeyError that "should not happen". There *is* an entry in the Catalog for that RID, which *has* '44.131' in the list under the 'ComponentVersions' key in the dict returned by getIndexDataForRID. Did the RID get removed from the KeywordIndex list of RIDs prematurely? Or did it never get inserted? Can anyone shed any enlightenment? Where else should I be looking? -- Jean Jordaan Upfront Systems http://www.upfrontsystems.co.za
Don't know if this will help but these errors specifically occur with 'lines' properties with Keyword Indexes on them. What is also significant is that there are legitimate duplicates in the lines properties and that they are duplicated on the index data as well. This doesn't feel right. I would have thought that only unique values should be stored in the index data. -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za On Mon, 30 Sep 2002 14:17:17 +0200 Jean Jordaan <jean@upfrontsystems.co.za> wrote:
Hi all
Our app has been hitting these errors for months:
2002-09-30T11:06:27 ERROR(200) KeywordIndex unindex_object could not remove documentId -444153566 from index ComponentVersions. This should not happen. Traceback (innermost last): File /usr/local/zope/2-5-1/lib/python/Products/PluginIndexes/common/UnIndex.py, line 168, in removeForwardIndexEntry (Object: ComponentVersions) KeyError: -444153566
On Monday 30 September 2002 08:36 am, Roché Compaan wrote:
Don't know if this will help but these errors specifically occur with 'lines' properties with Keyword Indexes on them.
What is also significant is that there are legitimate duplicates in the lines properties and that they are duplicated on the index data as well. This doesn't feel right. I would have thought that only unique values should be stored in the index data.
If you feel this is a bug, you should file a collector issue for it. -Casey
If you feel this is a bug, you should file a collector issue for it.
We don't have a deep enough understanding of the Catalog to diagnose :/ We're not sure if it's a Catalog issue, or something we're doing wrong; i.e. we don't know how the Catalog gets into that state. -- Jean Jordaan Upfront Systems http://www.upfrontsystems.co.za
On Mon, 30 Sep 2002 09:50:56 -0400 Casey Duncan <casey@zope.com> wrote:
On Monday 30 September 2002 08:36 am, Roché Compaan wrote:
Don't know if this will help but these errors specifically occur with 'lines' properties with Keyword Indexes on them.
What is also significant is that there are legitimate duplicates in the lines properties and that they are duplicated on the index data as well. This doesn't feel right. I would have thought that only unique values should be stored in the index data.
If you feel this is a bug, you should file a collector issue for it.
I'm not sure yet ... -- Roché Compaan Upfront Systems http://www.upfrontsystems.co.za
Toby Dickenson wrote:
Our app has been hitting these errors for months:
Have you tried it in 2.6? A couple of bug in this area have been fixed since 2.5.x
Yeah, this'd be my suggestion having suffered more terminally from exactly these bugs :-S cheers, Chris
Have you tried it in 2.6?
Yeah, this'd be my suggestion having suffered more terminally from exactly these bugs :-S
Hmm! That's interesting. I was worried that we might be causing it by tickling the Catalog the wrong way .. Did you ever get to the bottom of the bugs? Do they have collector issues, perhaps? -- Jean Jordaan Upfront Systems http://www.upfrontsystems.co.za
On Tue, Oct 01, 2002 at 08:37:16AM +0200, Jean Jordaan wrote:
Have you tried it in 2.6?
Yeah, this'd be my suggestion having suffered more terminally from exactly these bugs :-S
Hmm! That's interesting. I was worried that we might be causing it by tickling the Catalog the wrong way .. Did you ever get to the bottom of the bugs? Do they have collector issues, perhaps?
It would be fine if we could get a reproducable testcase for this problem. I have seen this problem multiple times in the past I have not been able to track it down to a simple testcase. Andreas
It would be fine if we could get a reproducable testcase for this problem.
Absolutely. When the error we see occurs, the RID that is looked for has already gone missing. It looks like the real error happens silently sometime before, and not in all cases: the majority of the instances of the affected class are cataloged OK. -- Jean Jordaan Upfront Systems http://www.upfrontsystems.co.za
Jean Jordaan wrote:
Hmm! That's interesting. I was worried that we might be causing it by tickling the Catalog the wrong way .. Did you ever get to the bottom of the bugs? Do they have collector issues, perhaps?
They were bugs in Zope's BTree implementation which were fixed for 2.6. cheers, Chris
participants (6)
-
Casey Duncan -
Chris Withers -
Jean Jordaan -
Roch� Compaan -
Toby Dickenson -
zopyx＠t-online.de