[Zope-dev] RE: [ZODB-Dev] [ATT] BTrees conflict resolution leads to indexinconsistencies

[Dieter Maurer]

Tim Peters wrote at 2005-3-22 23:45 -0500:
>[Dieter Maurer]
>> Zope indexes use code like this to remove document ids from document
>> lists:
>>
>> 	 def removeForwardIndexEntry(self, entry, documentId):
>> 	     indexRow = self._index[entry]
>> 	     indexRow.remove(documentId)
>> 	     if not indexRow:
>> 	         del self._index[entry]
>>
>> If a concurrent transaction adds a document id to the same "indexRow",
>> then this document id can be lost because it is inserted into a document
>> list discarded by the other transaction.
>>
>> Under normal conditions, a "ConflictError" would result because two
>> transactions modify the same object (the document list). However, the
>> BTrees conflict resolution prevents the "ConflictError" and allows the
>> loss.
>
>This must have been a horror to track down -- thank you for the effort!

It was not that bad.

We have had 2 instances of index corruption within the last
3 years -- without any indication how the indexes could have become
corrupt.

Then, we got another index corruption last week and finally
two yesterday.

All corruptions affected index entries with very short
document lists (workflow state index entries for short living
states). Consistency checks could not find inconsistencies in
other more populated index entries.
That provided the essential clue. Manaual concurrent insertion
and deletion finally reproduced the inconsistency.

-- 
Dieter