[Zope] POSKeyError

[Tim Peters]

[Kevin Carlson]
> Whenever I try to reindex an index in the portal_catalog or delete the
> index, add it back and then reindex, I get the first error below.  I ran
> fstest.py and there were no errors in the Data.fs file yet.  I also ran
> fsrecover.py and there were zero bytes removed from the Data.fs file.

Which means the low-level structure of your Data.fs file is fine;
roughly speaking, it's like a sentence that's grammatical but doesn't
completely make sense.  Please read this for background info; you'll
find it worth your time:

    http://zope.org/Wikis/ZODB/FileStorageBackup

If fstest.py has no complaints, it's unlikely fsrecover.py will change
anything.  Those two go together.

> Also, When I run fsrefs.py, I get the second bit of output below.

Which I'm moving up to here:

> fsrefs.py output
> 
> oid 0xC8DF8L BTrees.OOBTree.OOBucket
> last updated: 2005-01-25 09:42:25.200913, tid=0x35AC9C66B861E00L
> refers to invalid object:
>        oid 0xCC49AL missing: 'Products.CMFDefault.File.File'
> 
> oid 0xC8E1FL BTrees.OOBTree.OOBucket
> last updated: 2005-01-25 09:42:25.200913, tid=0x35AC9C66B861E00L
> refers to invalid object:
>        oid 0xCC49CL missing: 'BTrees.IOBTree.IOBTree'
> 
> oid 0xD297EL BTrees.OOBTree.OOBucket
> last updated: 2005-02-10 04:08:47.313263, tid=0x35B2278C9DEB333L
> refers to invalid object:
>        oid 0xCC49AL missing: 'Products.CMFDefault.File.File'

So two objects have vanished from this database, with object ids
(oids) 0xCC49A and 0xCC49C, and in total they're referenced _from_
three other objects.

As a result, any attempt to reference oids 0xCC49A or 0xCC49C will
raise POSKeyError -- it's akin to the KeyError Python raises if you
try to look up a key that doesn't exist in a Python dict.  Here a few
objects refer to other objects that don't actually exist.

There's isn't a wholly safe automated way to "fix that".
...

> Exception Type  POSKeyError
> Exception Value         00000000000cc49a

Note that oid 0xcc49a was identified by fsrefs.py as one of the two
oids (the missing object of type Products.CMFDefault.File.File) that
just don't exist in this database.  One way or another, the database
has to be changed to stop referring to this (non-existent) object,
else you'll continue to get a POSKeyError whenever an attempt to
reference it is made.  fsrecover.py can't do anything about this
(which should be clearer if you read the page linked to above).

The script mentioned in this recent message may (or may not) be useful to you:

    http://mail.zope.org/pipermail/zodb-dev/2005-January/008415.html

Note that, as it says, running it is guaranteed to cause data loss --
the resulting Data.fs may or may not be usable (it should be
well-formed (fstest.py shouldn't have any complaints), and fsrefs.py
should stop complaining too, but the script may delete objects
containing data you care about).

BTW, which version of Zope are you using here?  Major fixes went into
Zope 2.7.3 + ZODB 3.2.4 to repair bugs that could cooperate to create
POSKeyError situations (both transient and permanent).  Zope 2.7.4 +
ZODB 3.2.5 are the current stable releases.  There are no known Zope
or ZODB bugs that can cause POSKeyError situations in these recent
versions.