[Zope-dev] ZCatalog "Key Error" question

[Dieter Maurer]

zope-dev at rosenthaltechnologies.com wrote at 2006-9-3 18:06 -0400:
>
>Thank you so much for following-up with me, here.  I think you're right
>about this having to do with the indexes on the ZCatalog.
>
>The catalog has:
>1.  Seven "FieldIndex" indexes.
>2.  A "DateIndex" index
>3.  Three "ZCTextIndex" indexes

I have not seen reports that these indexes cause inconsistencies
of the type you have described.


A long time ago, there have been problems in the "BTrees"
conflict resolution which caused catalog inconsistencies
(of a different kind). They affected document lists of length one
in the index. When the last remaining document was removed
and a new one concurrently inserted, the insertion could be lost.
But this was fixed a long time ago.


Apart from bugs, inconsistencies are often caused by
"try: ... except: ..." clauses. If the "try" clause
modifies the persistent state and then is aborted by an exception,
it is vital that the exception reaches the ZPublisher such that
it aborts the transaction. If the "except" prevents this, then
the ZPublisher will commit and you are likely to get inconsistencies.


In older Zope versions (before Zope 2.8.1), such a dangerous
"try: ... except: ..." has been in
"OFS.ObjectManager.ObjectManager.manage_beforeDelete".
In more modern versions, the code has been slightly cleaned
up but it still is error prone.
Skim your log files whether you see "manage_beforeDelete () threw"
ERROR log entries.


You can get more hints towards the problem by examining the
query that results in your exception:

  If this query just consists of a subquery to a single
  index, then you know the problematic index.

  If it combines several indexes (by an "and"), then
  all of these indexes would need to be inconsistent (which
  is possible but far less likely than a single index).
  In this case, one would expect that not the index
  but the catalog primary data got corrupted.

  In the latter case, the problem would need to
  happen during initial indexing or final unindexing.
  Studying the code may exclude some possibilities.



-- 
Dieter