Index errors revisited: KeywordIndex unindex_object could not remove documentId -883140122 from index ComponentVersions. This should not happen.
Hi all We've posted about this before, but never managed to pin down the issue nicely. It doesn't interfere with our application, as far as we could tell, so we've let it slide. Here's another attempt at formulating the problem. We run a script which iterates over 719 <ZClass License> instances. For each instance, either one or two <ZClass Reminder> instances are created and edited. The script completes, logging all the way: 2003-02-27T09:43:44 INFO(0) MyDebugLog recreate_reminders> INFO: creating reminders for License_99 Then, Zope starts spitting out 657 errors like the following: 2003-02-27T09:43:49 ERROR(200) KeywordIndex unindex_object could not remove documentId -883140119 from index ComponentVersions. This should not happen. Traceback (innermost last): File /usr/local/zope/2-5-1/lib/python/Products/PluginIndexes/common/UnIndex.py, line 168, in removeForwardIndexEntry (Object: ComponentVersions) KeyError: -883140119 Most of these errors are identical. There are 172 unique errors, the rest are all retries. The 172 keys are not in the catalog. The only indexes involved are ComponentVersions and NodeNames, both 'lines' properties of <ZClass Reminder>. Does anyone know how to cause this, so that I can stop doing it? I'm trying to narrow it down again .. (Zope 2.5.1 (source release, python 2.1, linux2), python 2.1.3, linux2) -- Jean Jordaan http://www.upfrontsystems.co.za
In article <3E5E247B.6050607@upfrontsystems.co.za> you write:
We run a script which iterates over 719 <ZClass License> instances. For each instance, either one or two <ZClass Reminder> instances are created and edited. The script completes, logging all the way:
2003-02-27T09:43:44 INFO(0) MyDebugLog recreate_reminders> INFO: creating reminders for License_99
Then, Zope starts spitting out 657 errors like the following:
2003-02-27T09:43:49 ERROR(200) KeywordIndex unindex_object could not remove documentId -883140119 from index ComponentVersions. This should not happen. Traceback (innermost last): File /usr/local/zope/2-5-1/lib/python/Products/PluginIndexes/common/UnIndex.py, line 168, in removeForwardIndexEntry (Object: ComponentVersions) KeyError: -883140119
Most of these errors are identical. There are 172 unique errors, the rest are all retries. The 172 keys are not in the catalog. The only indexes involved are ComponentVersions and NodeNames, both 'lines' properties of <ZClass Reminder>.
Does anyone know how to cause this, so that I can stop doing it? I'm trying to narrow it down again ..
(Zope 2.5.1 (source release, python 2.1, linux2), python 2.1.3, linux2)
It would be interesting to test it with Zope 2.6.1... I'd hate to debug something that has already been fixed. To debug it, I'd log a full backtrace by patching UnIndex.py line 168 when the KeyError is raised. But I don't know enough about the catalog to debug deeper. An invariant is being broken somewhere. Maybe because of a too-greedy try/except: ? Which reminds me, there is such a greedy one in manage_beforeDelete, it could be your culprit. BTW are you using CMF? Florent -- Florent Guillaume, Nuxeo (Paris, France) +33 1 40 33 79 87 http://nuxeo.com mailto:fg@nuxeo.com
Jean Jordaan wrote at 2003-2-27 16:45 +0200:
We've posted about this before, but never managed to pin down the issue nicely. It doesn't interfere with our application, as far as we could tell, so we've let it slide. Here's another attempt at formulating the problem.
The reason is very simple: In order to unindex objects later, Zope's indexes remember the values an object has been indexed by. "KeywordIndex" is a bit stupid about this. If the sequence contains duplicates, it does not remove them. Later, when the object is unindexed, it tries to remove the object from the list for each term in the sequence. In case of a duplicate, this goes well when it is encountered for the first time but, when it is encountered again, the object can (of course) not be removed... The fix: fix "KeywordIndex" such that is removes duplicates from "_unindex[documentId]". Dieter
Dieter:
The reason is very simple:
Indeed! You make it clear as day :)
The fix: fix "KeywordIndex" such that is removes duplicates from "_unindex[documentId]".
We're looking to see if we can fix it for us .. -- Jean Jordaan http://www.upfrontsystems.co.za
Jean Jordaan wrote:
Dieter:
The reason is very simple:
Indeed! You make it clear as day :)
The fix: fix "KeywordIndex" such that is removes duplicates from "_unindex[documentId]".
We're looking to see if we can fix it for us ..
Is this fixed in 2.6.x? If not, please file a collector issue when you have a patch and I'll make sure it's applied on the 2.6 branch as well as the HEAD. cheers, Chris
Yo Chris
If not, please file a collector issue when you have a patch and I'll make sure it's applied on the 2.6 branch as well as the HEAD.
http://collector.zope.org/Zope/828 -- Jean Jordaan http://www.upfrontsystems.co.za
participants (4)
-
Chris Withers -
Dieter Maurer -
Florent Guillaume -
Jean Jordaan