BTrees strangeness (was [Zope-dev] Zope 2.X BIG Session problems - blocker - our site dies - need help of experience Zope developer, please)

[Dieter Maurer]

Chris McDonough wrote at 2004-3-3 04:55 -0500:
>(boldly crossposting this to zodb-dev, please respond on one list or the
>other but not both)
>
>That error *appears* to be caused by reaching a state that is impossible
>to reach.  The code in question is:
>
>        for key in list(self._data.keys(None, max_ts)):
>            assert(key <= max_ts)
>            STRICT and _assert(self._data.has_key(key))
>            for v in self._data[key].values():
>                to_notify.append(v)
>            del self._data[key]
>
>The line that says "for v in self._data[key].values()" is the line that
>throws the KeyError.   But it should be impossible for the code to throw
>a KeyError for the expression "self._data[key]" because the "keys"
>method of the _data IOBTree just told us that the key named by "key" was
>one of its keys via the range search; it should be an invariant.

If we had a low conflict connection, I would understand how
this could happen:

  All BTree leaves are chained together. The "keys" method
  uses this chain to enumerate the keys.

  "[]" on the other hand, uses the tree structure to
  locate its key.

  Assume that a parallel thread removes a key and commits
  the transaction while we are in the for loop above.
  We may read the old state (with the later deleted key)
  for "keys". During the "for", we receive invalidations
  for the nodes affected by the deletion.
  When we access "[key]" we may try to load a tree node
  which is not yet in the ZODB cache and meanwhile
  invalidated. When we suppress the resulting
  "ReadConflictError", we may not find "key" (as it
  is by now deleted).


In my "Transience" implementation, I ignore this exceptional
case. I do use a low conflict connection and have to
be prepared for this situation. Furthermore, the
situation is not problematic: I want to determine
sessions that should be deleted. Someone else
already did it -- this is fine. No need to do it twice...

-- 
Dieter