[ZODB-Dev] Old transaction records with "bad" pickle data preventing packing
Jim Fulton
jim at zope.com
Mon Jun 14 09:10:42 EDT 2010
On Mon, Jun 14, 2010 at 3:43 AM, Jens Vagelpohl <jens at dataflake.org> wrote:
> Hi all,
>
> In a system using Zope 2.11 and ZODB 3.7.0b3 I'm having a problem with
> old transaction records containing pickle data that fails on
> cPickle.Unpickler.noload as employed by ZODB.serialize.referencesf, I
> get an AttributeError from inside the cPickle module:
It would be interesting to see the traceback.
> def referencesf(p, oids=None):
> """Return a list of object ids found in a pickle
>
> A list may be passed in, in which case, information is
> appended to it.
>
> Weak references are not included.
> """
>
> refs = []
> u = cPickle.Unpickler(cStringIO.StringIO(p))
> u.persistent_load = refs
> u.noload()
> u.noload()
>
> # Now we have a list of referencs. Need to convert to list of
> # oids:
> <snip>
>
> The noload failure means I cannot pack the database right now. Since
> only historical transactions contain the problematic attribute and I
> don't care about those anymore
Note that:
- in ZODB 3.9, there's an option top pack without doing GC, and
- ZODB 3.9 ZEO servers can be used with older clients.
> I have introduced a simple change locally
> that ignores the failure:
>
> def referencesf(p, oids=None):
> """Return a list of object ids found in a pickle
>
> A list may be passed in, in which case, information is
> appended to it.
>
> Weak references are not included.
> """
>
> refs = []
> u = cPickle.Unpickler(cStringIO.StringIO(p))
> u.persistent_load = refs
> try:
> u.noload()
> u.noload()
> except AttributeError:
> pass
>
> # Now we have a list of referencs. Need to convert to list of
> # oids:
> <snip>
That's a bad idea.
> With this change I can pack the database and a few simple check
> (fsrefs.py etc) indicate the database is consistent after the pack.
Cool. You got lucky. :)
> My question: Is there any risk associated with ignoring the failure
> during packing?
Yes. You might ignore references to objects causing them to be
removed while still used.
> The latest object versions for the persistent objects in
> question do not have the problematic attribute anymore, so the latest
> records are "safe" and will not cause the AttributeError, anyway.
Right, it worked out for your DB.
Jim
--
Jim Fulton
More information about the ZODB-Dev
mailing list