[Zope-dev] [ATT] BTrees conflict resolution leads to index inconsistencies

[Dieter Maurer]

[Crosspost] Please followup to "zodb-dev at zope.org".

Zope indexes use code like this to remove document ids from
document lists:

	 def removeForwardIndexEntry(self, entry, documentId):
	     indexRow = self._index[entry]
	     indexRow.remove(documentId)
	     if not indexRow:
	         del self._index[entry]

If a concurrent transaction adds a document id to the same
"indexRow", then this document id can be lost because
it is inserted into a document list discarded by the other
transaction.

Under normal conditions, a "ConflictError" would result
because two transactions modify the same object
(the document list). However, the BTrees conflict resolution
prevents the "ConflictError" and allows the loss.


The result is an index inconsistency: the forward index ("_index")
entry and the backward one ("_unindex") not longer correspond.
Further operations lead to "unable to remove documentId
from document list" exceptions.


I suggest to tighten BTrees conflict resolution not to allow
concurrent insertions when a bucket was emptied.


-- 
Dieter