[ZODB-Dev] Recovering from a POSkeyerror

[Tim Peters]

[Paulo Eduardo Neves]
> Hi, I'm somewhat desperade, maybe someone here can help me.

I hope so (but it probably won't be me).

This is worth reading:

    http://zope.org/Wikis/ZODB/FileStorageBackup

> I'm trying to recover from a POSkeyerror in a ZClass. I'm still using
> Zope 2.6.4
>
> I have the ZClass instances working fine, but when I go to this URL in
> the management web interface:
>
> /Control_Panel/Products/Comentario/ComentarioClass/
>
> I get the POSkeyerror
>
> I've read the tips in http://www.zopelabs.com/cookbook/1054240694
>
> The problem is that I can't find the problem object.
>
> obj =
root.unrestrictedTraverse("Control_Panel/Products/Comentario/ComentarioClass
")
>
> obj.objectItems() is empty if I try with the parent folder, I can access
> all objects without an POSKeyError
>
> Can anyone help me to find the corrupted object?

There's a long comment at the bottom of the page you referenced, starting
with "If you don't have a clue where to start looking for the broken
object(s) ...".  Maybe that will help.

> It looks like there's some kind of ZClass magic in there.

The number of people who *could* help gets cut by a factor of ten with each
new sentence <wink>.

> When I run fsfrefs.py I get the following output:
>
> oid 0x324ff failed to load
> oid 0x32500 failed to load
> oid 0x32506 failed to load
> oid 0x32507 failed to load 
> oid 0x3250a failed to load
> oid 0x3250b failed to load
> oid 0x32510 failed to load
> oid 0x32511 failed to load
> oid 0x32514 failed to load
> oid 0x32515 failed to load
> oid 0x32516 failed to load
> oid 0x32517 failed to load
> oid 0x32518 failed to load 
> oid 0x32519 failed to load
> oid 0x3251a failed to load
> oid 0x3251b failed to load
> oid 0x3251c failed to load
> oid 0x3251d failed to load
> oid 0x3251e failed to load
> oid 0x330df failed to load

Run fsrefs.py with -v and it will show tracebacks associated with the load
failures (see the link I gave above).  Download ZODB 3.2.4 (the current
stable release) and use its fsrefs.py, and I bet a bunch of those complaints
will go away (the reasons why were discussed here (on zodb-dev) over the
last week -- look at the archives).

Note that fsrefs.py doesn't know anything about versions, so if your
POSKeyError is coming from an object in a version, fsrefs.py won't be able
to find the problem.

The POSKeyError exception you got should have contained the oid (object id)
of the missing object.  You can use fsdump.py (see the link I gave above) to
get a semi-readable text dump of your .fs file, and search for that oid in
the dump for more clues.

> It looks like I have the problem in more than one object.:-(

Not necessarily.  The version of fsrefs.py you're using can produce the
messages you're seeing for objects whose creation has merely been undone,
which isn't actually a problem in isolation.  The current ZODB's fsrefs.py
is better-behaved in this case.

The real mystery here so far is why you're *only* seeing "failed to load"
messages.  That's not the same as a POSKeyError, although it would be much
easier (for me <wink>) if you tried a current fsrefs.py than to try to
explain all that can go wrong with the fsrefs.py you're using.

The only likely scenario I have in mind is that one of your active objects
refers to an object whose creation has been undone.  If so, a current
fsrefs.py should produce a much clearer message about that.

> I also don't know how to discover the parent folder of an object having
> the oid.
>
> Would upgrade to 2.7 help me?

It won't fix any pre-existing .fs problems, no.  There have been many
bugfixes to Zope, ZODB and ZEO since Zope 2.6.4, and upgrading to a current
release would prevent some kinds of problems you're vulnerable to so long as
you stick with 2.6.4 (note that Zope 2.7.4 final should be released this
weekend).