[ZODB-Dev] zeopack error

[Jim Fulton]

I'm sorry I haven't had time to look at this. Still don't really.

Thanks Marius!!!

On Wed, Feb 8, 2012 at 6:48 PM, Marius Gedminas <marius at gedmin.as> wrote:
> On Thu, Feb 09, 2012 at 01:25:48AM +0200, Marius Gedminas wrote:
>> On Wed, Feb 08, 2012 at 01:24:55PM +0100, Kaweh Kazemi wrote:
>> > Recap: last week I examined problems I had packing our 4GB users
>> > storage.
> ...
>>     >>> unp = pickle.Unpickler(f)
>>     >>> unp.persistent_load = lambda oid: '<persistent reference %r>' % oid
>>     >>> pprint.pprint(unp.load())
>>     {'data': {"<persistent reference ['m', ('game', '\\x00\\x00\\x00\\x00\\x00\\x00\\tT', <class '__main__.Tool'>)]>": 1,
>>               "<persistent reference ['m', ('game', '\\x00\\x00\\x00\\x00\\x00\\x00\\x12\\x03', <class '__main__.EnergyPack'>)]>": 1}}

Note the reference to __main__. This is almost certainly the root problem.
Classes shouldn't be defined in __main__ (except when experimenting).

At one time, I thought pickle disallowed pickling classes from __main__.
ZODB probably should. It's a bug magnet.


>>
>> Those look like cross-database references to me.
>>
>> The original error (aaaugh Mutt makes it hard for me to look upthread
>> while I'm writing a response) was something about non-hashable lists?
>> Looks like a piece of code is trying to put persistent references into a
>> dict, which can't possibly work in all cases.
> ...
>> > During my checks I realized that running the pack in a Python 2.7
>> > environment (using the same ZODB version - 3.10.3) works fine, the
>> > pack reduces our 4GB storage to 1GB. But our production server uses
>> > Python 2.6 (same ZODB3.10.3) which yields the problem (though the test
>> > had been done on OS X 10.7.3 - 64bit, and the production server is
>> > Debian Squeeze 32bit).
>>
>> I've no idea why running the same ZODB version on Python 2.7 instead of
>> 2.6 would make this error go away.
>
> Duh!  The code that fails is in the standard library -- in the cPickle
> module:
>
>> > Traceback (most recent call last):
> ...
>> >   File "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/FileStorage/fspack.py", line 328, in findrefs
>> >     return self.referencesf(self._file.read(dh.plen))
>> >   File "/usr/local/lib/python2.6/dist-packages/ZODB3-3.10.3-py2.6-linux-i686.egg/ZODB/serialize.py", line 630, in referencesf
>> >     u.noload()
>> > TypeError: unhashable type: 'list'
>
> Since the bug is in the stdlib, it's not surprising that the newer
> stdlib cPickle from Python 2.7 fixes it.

I suspect a bug in the application (defining persistent classes in __main__)
is the root problem that's aggravated by the cPickle problem.

> That was fun.

I'm glad. :)

Thanks.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton