[Zope-dev] misterious btree issue

6 Nov 2008

      Hello,

I've run into a misterious issue while evolving generations from an
old DB.

Quick fact is that it seems like a BTree kept an object reference to
an object which was deleted from it. I'll try to explain/dig into
below.

BTrees source is of revision 75706, running on win32, compiled with
mingw32.

The evolution is about totally removing a utility because it's source
will be ripped out in the next generation. It's in z3c.vocabulary.
The utility contains objects in BtreeContainers recursively.

Problem was that the loop at #kill ALL should have removed all object
references from IntIds.
For the first look it removed all, the loop at #clean intids never
found one (toremove was always empty).
All seemed fine, until the next generation came along and
tried to touch all objects in ZODB. (ZODB was packed first)
And *failed* on an instance of the utility. (source was gone)
Debugging that found utility with
http://www.zopelabs.com/cookbook/1114086617 clearly showed that 
it is still referenced by the IntIds.ids OIBTree, though impossible to
reach via IntIds.ids.keys() or IntIds.ids.values().

What helped was to recreate both IntId BTrees from scratch, like at
#recreate intid trees. That and packing made the utility finally go
away.

Anybody noticed something like this?

--------------------------------------
def evolve(context):
    """Evolve the ZODB.

    - remove IVocabularyManager utilities

    """
    root = getRootFolder(context)

    storage = context.connection._storage

    for site in findObjectsProviding(root, interfaces.IRecruiterSite):
        originalSite = hooks.getSite()
        hooks.setSite(site)

        sm = site.getSiteManager()
        if sm['default'].has_key('vocabularyManager'):
            vm = sm['default']['vocabularyManager']

            ##kill ALL
            #remove all items from the utility to remove them from IntIds
            for k,v in list(vm.items()):
                for kk, vv in list(v.items()):
                    for kkk, vvv in list(vv.items()):
                        del vv[kkk]
                    del v[kk]
                del vm[k]

            sm.unregisterUtility(vm, IVocabularyManager, '')
            del sm['default']['vocabularyManager']

        if sm['default'].has_key('ids'):
            #clean intids
            iidutil = sm['default']['ids']
            toremove = []
            for id in iidutil:
                obj = iidutil.getObject(id)
                if not belowsite(obj, site):
                    toremove.append(obj)

                if obj.__module__.startswith("z3c.vocabulary"):
                    toremove.append(obj)

            for k, id in iidutil.ids.items():
                obj = iidutil.getObject(id)
                if obj.__module__.startswith("z3c.vocabulary"):
                    toremove.append(obj)
                if k.object.__module__.startswith("z3c.vocabulary"):
                    toremove.append(k.object)

            for obj in toremove:
                iidutil.unregister(obj)

            #recreate intid trees:
            #because it seems like sometimes some buckets or whatever
            #reference objects once they are long gone
            ids = iidutil.ids
            refs = iidutil.refs
            iidutil.ids = iidutil.family.OI.BTree()
            iidutil.refs = iidutil.family.IO.BTree()

            for k,v in ids.items():
                iidutil.ids[k] = v

            for k,v in refs.items():
                iidutil.refs[k] = v

-----------------------
#next generation touching all objects
def evolve(context):
    """Evolve the ZODB.
    """
    storage = context.connection._storage

    next_oid = None
    while True:
        oid, tid, data, next_oid = storage.record_iternext(next_oid)

        modname, classname = get_pickle_metadata(data)

        obj = context.connection.get(oid)
        # Make sure that we tell all objects that they have been changed. Who
        # cares whether it is true! :-)
        obj._p_activate()
        obj._p_changed = True

        if next_oid is None:
            break

-- 
Best regards,
 Adam GROSZER                          mailto:agroszer@gmail.com
--
Quote of the day:
When a deep injury is done us, we never recover until we forgive. 
- Alan Paton