[Zope] unpickle error on Data.fs pack

[Tim Peters]

[Gerhard Schmidt]
> since three days we have problems when packing the Data.fs.
>
> 2006-01-25T03:40:42 ERROR(200) zrpc:7266 Error raised in delayed method
> Traceback (most recent call last):
>   File "/usr/local/www/Zope/lib/python/ZEO/StorageServer.py", line 991, in run
>     result = self._method(*self._args)
>   File "/usr/local/www/Zope/lib/python/ZEO/StorageServer.py", line 315, in _pack_impl
>     self.storage.pack(time, referencesf)
>   File "/usr/local/www/Zope/lib/python/ZODB/FileStorage.py", line 1582, in pack
>     opos = p.pack()
>   File "/usr/local/www/Zope/lib/python/ZODB/fspack.py", line 700, in pack
>     self.gc.findReachable()
>   File "/usr/local/www/Zope/lib/python/ZODB/fspack.py", line 456, in findReachable
>     self.findReachableAtPacktime([z64])
>   File "/usr/local/www/Zope/lib/python/ZODB/fspack.py", line 531, in findReachableAtP
> acktime
>     todo.extend(self.findrefs(pos))
>   File "/usr/local/www/Zope/lib/python/ZODB/fspack.py", line 604, in findrefs
>     return referencesf(self._file.read(dh.plen))
>   File "/usr/local/www/Zope/lib/python/ZODB/referencesf.py", line 38, in referencesf
>     raise ValueError, 'Error unpickling %r' % p
> ValueError: Error unpickling '((U\x0eBTrees.OIBTreeq\x01U\x08OIBucketq\x02tq\x03Nt.((
> U\x05nchenq\x04J\xc6{a\xfeU\x0fnchen/ottobrunnq\x05J\xbd\xeby\xcfU\x04ndigq\x06J\n\xf
> 0}QU\x05ndnisq\x07J\xd9\xdc\xbfIU\x02neq\x08J1!\x15\xe9U\x05nebenq\tJT]4\xc0U\x03netq
> \nJ\xf3cU\xb6U\x04net/q\x0bJ\nM\xe5\xd6U\x07networkq\x0cJ\xf5\x85!\xe5U\tnetzartigq\r
> J\xd4\xf9\x906U\x03neuq\x0eJv\xd7>\xe9U\x04neueq\x0fJW\xedD\xd8U\x05neuenq\x10J0>!\x0
> 7U\x05neuesq\x11J\xb9\xa5\xb4sU\x08neuestenq\x12JW2\xcc-U\x07nftigenq\x13J\xd5"i>U\x0
> 3ngeq\x14J%\xa9X\x10U\x06ngerenq\x15J>\x1d\x14YU\x04ngigq\x16J\xc4\xe6\xe5\xd4U\x06ng
> igenq\x17J}\xbd\xffpU\nngigkeitenq\x18J)]\x06IU\x05nichtq\x19J\x0bgy>U\x07nkungenq\x1
> aJC4\xf7\x10U\x04nnenq\x1bJU\xc4bFU\x04nochq\x1cJ\xb4\xf6\xcdUU\x07norbertq\x1dJ-\xf3
> \xd7\x8fU\x06normenq\x1eJ[\x84\xd4\xaeU\x07normungq\x1fJ\xf4\xe9\xfc\xfcU\x08notebook
> q J\xf7\xf2\x9e\xf9U\x0fnotebookeinsatzq!J`\x8fRiU\tnotebooksq"J\xba>\xecvU\x12notebo
> okverwendungq#J_R\x10\x9aU\x02nrq$J\xfcSg\xddU\x05nscheq%J-\x88\xf8\xccU\x06nstigeq&J
> \xa35\x0e\xcdU\x04nterq\'J\xb1\x94\x9b\xeeU\nnumerischeq(Jf\\n\xfeQ

Not that it will help much, but the \x03 on the start of the next line
is the immediate cause of the error (it's in a _position_ where "a
pickle opcode" is expected, but 0x03 isn't a legitimate pickle
opcode).

> \x03nurq)J\xd2\x95
> C\xf4U\x0cnutzbringendq*J\xd3\x84\x84\xeaU\x06nutzenq+J\xa7^\x86IU\tnutzungsmq,J\xc1\
> xca\xb9LU\x02obq-J\xbe\xb7e\x94U\x06objectq.J\xaaP\x14\xf9U\x04oderq/J\\0\xc2(U\x05od
> imaq0J\xcd\xf9\x0f:t(U\x08\x00\x00\x00\x00\x00%\xa5\xdaq1(U\x0eBTrees.OIBTreeq2U\x08O
> IBucketq3ttq4Qtq5.'
> ------
>
> I've tried to recover the data.fs with fsrecover but it returns
> without error and the error remains.

This is corruption _inside_ an object pickle.  fsrecover can't do
anything about that.  See

    http://zope.org/Wikis/ZODB/FileStorageBackup

for background info.

> fsrefs.py terminates with an error.

Yes, fsrefs can't load the pickle either.

> Any idea how to fix the Data.fs.

Sorry, not easily, no.  You need to exploit your application-specific
knowledge about the importance of this specific OIBucket, and couple
that with fiddly knowledge about how pickle works.

> The System is still up an running and no error shown so far.

Maybe you could locate the object in question from an fsdump (see the
link above), and delete the whole object.