[ZODB-Dev] Old transaction records with "bad" pickle data preventing packing

[Jens Vagelpohl]

Hi all,

In a system using Zope 2.11 and ZODB 3.7.0b3 I'm having a problem with
old transaction records containing pickle data that fails on
cPickle.Unpickler.noload as employed by ZODB.serialize.referencesf, I
get an AttributeError from inside the cPickle module:

def referencesf(p, oids=None):
    """Return a list of object ids found in a pickle

    A list may be passed in, in which case, information is
    appended to it.

    Weak references are not included.
    """

    refs = []
    u = cPickle.Unpickler(cStringIO.StringIO(p))
    u.persistent_load = refs
    u.noload()
    u.noload()

    # Now we have a list of referencs.  Need to convert to list of
    # oids:
<snip>

The noload failure means I cannot pack the database right now. Since
only historical transactions contain the problematic attribute and I
don't care about those anymore I have introduced a simple change locally
that ignores the failure:

def referencesf(p, oids=None):
    """Return a list of object ids found in a pickle

    A list may be passed in, in which case, information is
    appended to it.

    Weak references are not included.
    """

    refs = []
    u = cPickle.Unpickler(cStringIO.StringIO(p))
    u.persistent_load = refs
    try:
        u.noload()
        u.noload()
    except AttributeError:
        pass

    # Now we have a list of referencs.  Need to convert to list of
    # oids:
<snip>

With this change I can pack the database and a few simple check
(fsrefs.py etc) indicate the database is consistent after the pack.

My question: Is there any risk associated with ignoring the failure
during packing? The latest object versions for the persistent objects in
question do not have the problematic attribute anymore, so the latest
records are "safe" and will not cause the AttributeError, anyway.

jens