[ZODB-Dev] zeopack error

[Kaweh Kazemi]

I did the change and here we go:

[INFO] 2012-01-31T12:47:22: (::ffff:127.0.0.1:36857) pack(time=1328010442.395493) started...
[ERROR] 2012-01-31T14:06:53: (32702) Error raised in delayed method
Traceback (most recent call last):
  File "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZEO/StorageServer.py", line 1377, in run
    result = self._method(*self._args)
  File "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZEO/StorageServer.py", line 343, in _pack_impl
    self.storage.pack(time, referencesf)
  File "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/FileStorage/FileStorage.py", line 1078, in pack
    pack_result = self.packer(self, referencesf, stop, gc)
  File "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/FileStorage/FileStorage.py", line 1034, in packer
    opos = p.pack()
  File "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/FileStorage/fspack.py", line 397, in pack
    self.gc.findReachable()
  File "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/FileStorage/fspack.py", line 190, in findReachable
    self.findReachableAtPacktime([z64])
  File "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/FileStorage/fspack.py", line 275, in findReachableAtPacktime
    for oid in self.findrefs(pos):
  File "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/FileStorage/fspack.py", line 328, in findrefs
    return self.referencesf(self._file.read(dh.plen))
  File "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/serialize.py", line 630, in referencesf
    u.noload()
TypeError: unhashable type: 'list'
[INFO] 2012-01-31T14:06:53: (::ffff:127.0.0.1:36857) disconnected


I still don't know what to do beyond that.

I have one assumption - seeing this traceback - but I can't prove if it's correct: one "old" object (which is candidate to be removed) in the users storage is referencing an object in a different storage (we are using multiple databases) which has been packed away previously. Generally that shouldn't happen, but maybe we have wrongly deleted objects from the second storage and packed it previously. 

And I'm still stuck how to continue beyond that.

PS: packing works if I disable gc, but I would rather remove the problematic objects/transactions and get rid of old stuff to get a clean storage again.

Kind regards,
Kaweh


On 31.01.2012, at 12:35, Martijn Pieters wrote:

> On Tue, Jan 31, 2012 at 12:23, Kaweh Kazemi <kaweh at me.com> wrote:
>> [ERROR] 2012-01-31T09:20:25: (22455) Error raised in delayed method
>> None
> 
> Ugh, a quick look at the code reveals that this must've been raised by
> a SlowMethodThread handler, and it indeed doesn't provide the
> exception info in the log because that's already been cleared.
> 
> However, the method that does the logging does get passed the
> exception info (it is after all, partially passed back to the packing
> script). Could you open ZEO/zrpc/connection.py and edit line 54, the
> context looks like this:
> 
>    def error(self, exc_info):
>        self.sent = 'error'
>        log("Error raised in delayed method", logging.ERROR, exc_info=True)
>        self.conn.return_error(self.msgid, *exc_info[:2])
> 
> If you do not know where that file is located, search for the ZODB3
> egg path in your bin/zeo file, that's the directory where you'll find
> the ZEO code. Change "exc_info=True" to "exc_info=exc_info", so the
> error method looks like this:
> 
>    def error(self, exc_info):
>        self.sent = 'error'
>        log("Error raised in delayed method", logging.ERROR, exc_info=exc_info)
>        self.conn.return_error(self.msgid, *exc_info[:2])
> 
> Restart your ZEO server and attempt to pack again. Now it should log
> the exception instead of None.
> 
> -- 
> Martijn Pieters