Hello, I've run into a misterious issue while evolving generations from an old DB. Quick fact is that it seems like a BTree kept an object reference to an object which was deleted from it. I'll try to explain/dig into below. BTrees source is of revision 75706, running on win32, compiled with mingw32. The evolution is about totally removing a utility because it's source will be ripped out in the next generation. It's in z3c.vocabulary. The utility contains objects in BtreeContainers recursively. Problem was that the loop at #kill ALL should have removed all object references from IntIds. For the first look it removed all, the loop at #clean intids never found one (toremove was always empty). All seemed fine, until the next generation came along and tried to touch all objects in ZODB. (ZODB was packed first) And *failed* on an instance of the utility. (source was gone) Debugging that found utility with http://www.zopelabs.com/cookbook/1114086617 clearly showed that it is still referenced by the IntIds.ids OIBTree, though impossible to reach via IntIds.ids.keys() or IntIds.ids.values(). What helped was to recreate both IntId BTrees from scratch, like at #recreate intid trees. That and packing made the utility finally go away. Anybody noticed something like this? -------------------------------------- def evolve(context): """Evolve the ZODB. - remove IVocabularyManager utilities """ root = getRootFolder(context) storage = context.connection._storage for site in findObjectsProviding(root, interfaces.IRecruiterSite): originalSite = hooks.getSite() hooks.setSite(site) sm = site.getSiteManager() if sm['default'].has_key('vocabularyManager'): vm = sm['default']['vocabularyManager'] ##kill ALL #remove all items from the utility to remove them from IntIds for k,v in list(vm.items()): for kk, vv in list(v.items()): for kkk, vvv in list(vv.items()): del vv[kkk] del v[kk] del vm[k] sm.unregisterUtility(vm, IVocabularyManager, '') del sm['default']['vocabularyManager'] if sm['default'].has_key('ids'): #clean intids iidutil = sm['default']['ids'] toremove = [] for id in iidutil: obj = iidutil.getObject(id) if not belowsite(obj, site): toremove.append(obj) if obj.__module__.startswith("z3c.vocabulary"): toremove.append(obj) for k, id in iidutil.ids.items(): obj = iidutil.getObject(id) if obj.__module__.startswith("z3c.vocabulary"): toremove.append(obj) if k.object.__module__.startswith("z3c.vocabulary"): toremove.append(k.object) for obj in toremove: iidutil.unregister(obj) #recreate intid trees: #because it seems like sometimes some buckets or whatever #reference objects once they are long gone ids = iidutil.ids refs = iidutil.refs iidutil.ids = iidutil.family.OI.BTree() iidutil.refs = iidutil.family.IO.BTree() for k,v in ids.items(): iidutil.ids[k] = v for k,v in refs.items(): iidutil.refs[k] = v ----------------------- #next generation touching all objects def evolve(context): """Evolve the ZODB. """ storage = context.connection._storage next_oid = None while True: oid, tid, data, next_oid = storage.record_iternext(next_oid) modname, classname = get_pickle_metadata(data) obj = context.connection.get(oid) # Make sure that we tell all objects that they have been changed. Who # cares whether it is true! :-) obj._p_activate() obj._p_changed = True if next_oid is None: break -- Best regards, Adam GROSZER mailto:agroszer@gmail.com -- Quote of the day: When a deep injury is done us, we never recover until we forgive. - Alan Paton
Hi Adam Adam GROSZER wrote:
Hello,
I've run into a misterious issue while evolving generations from an old DB.
Quick fact is that it seems like a BTree kept an object reference to an object which was deleted from it. [snip]
Yes, the BTree implementations we use may keep a reference to a deleted key. I think I have seen this documented somewhere but don't have the time to look it up now. It saves some bookkeeping to do it that way, and I think it also reduces the risk of conflict errors.
What helped was to recreate both IntId BTrees from scratch, like at #recreate intid trees. That and packing made the utility finally go away.
Anybody noticed something like this Yes, I ran into this a few weeks ago when I needed to delete the code for an obsolete content type. IIRC you only really need to recreate the mapping from keyreferences to intids (the ids attribute of the default IIntIds utility).
Hope this helps - Jacob
On Nov 6, 2008, at 8:14 AM, Jacob Holm wrote:
Hi Adam
Adam GROSZER wrote:
Hello,
I've run into a misterious issue while evolving generations from an old DB.
Quick fact is that it seems like a BTree kept an object reference to an object which was deleted from it. [snip]
Yes, the BTree implementations we use may keep a reference to a deleted key. I think I have seen this documented somewhere but don't have the time to look it up now. It saves some bookkeeping to do it that way, and I think it also reduces the risk of conflict errors.
In particular, it might have a reference to the key in an internal BTree node. (I don't know this to be true, but I wouldn't be surprised if it was true.) If this is the case, we can definitely fix it. Jim -- Jim Fulton Zope Corporation
[Jim Fulton]
In particular, it might have a reference to the key in an internal BTree node. (I don't know this to be true, but I wouldn't be surprised if it was true.)
It was true when I was working on BTrees ... here, from http://svn.zope.org/ZODB/trunk/src/BTrees/Development.txt?rev=85198&view=mar... ... + When a key is deleted, it's physically removed from the bucket it's in, but this doesn't propagate back up the tree: since keys in interior nodes only serve to guide searches, it's OK-- and saves time --to leave "stale" keys in interior nodes. ...
If this is the case, we can definitely fix it.
Right, just delete that paragraph ;-)
On Nov 6, 2008, at 11:32 AM, Tim Peters wrote:
[Jim Fulton]
In particular, it might have a reference to the key in an internal BTree node. (I don't know this to be true, but I wouldn't be surprised if it was true.)
It was true when I was working on BTrees ... here, from
http://svn.zope.org/ZODB/trunk/src/BTrees/Development.txt?rev=85198&view=mar...
... + When a key is deleted, it's physically removed from the bucket it's in, but this doesn't propagate back up the tree: since keys in interior nodes only serve to guide searches, it's OK-- and saves time --to leave "stale" keys in interior nodes. ...
If this is the case, we can definitely fix it.
Right, just delete that paragraph ;-)
:) It would be great if someone (Adam?) created a launchpad issue to fix this. If I get into the BTree code, which I may do to make sizes setable via class attributes, I'll look at fixing this. Jim -- Jim Fulton Zope Corporation
Hello Jim, Does this fit? https://bugs.launchpad.net/zope3/+bug/294788 Thursday, November 6, 2008, 6:14:34 PM, you wrote: JF> On Nov 6, 2008, at 11:32 AM, Tim Peters wrote:
[Jim Fulton]
In particular, it might have a reference to the key in an internal BTree node. (I don't know this to be true, but I wouldn't be surprised if it was true.)
It was true when I was working on BTrees ... here, from
http://svn.zope.org/ZODB/trunk/src/BTrees/Development.txt?rev=85198&view=mar...
... + When a key is deleted, it's physically removed from the bucket it's in, but this doesn't propagate back up the tree: since keys in interior nodes only serve to guide searches, it's OK-- and saves time --to leave "stale" keys in interior nodes. ...
If this is the case, we can definitely fix it.
Right, just delete that paragraph ;-)
JF> :) JF> It would be great if someone (Adam?) created a launchpad issue to fix JF> this. JF> If I get into the BTree code, which I may do to make sizes setable via JF> class attributes, I'll look at fixing this. JF> Jim JF> -- JF> Jim Fulton JF> Zope Corporation -- Best regards, Adam GROSZER mailto:agroszer@gmail.com -- Quote of the day: When the wind is great, bow before it; when the wind is heavy, yield to it.
On Nov 6, 2008, at 1:06 PM, Adam GROSZER wrote:
Hello Jim,
Does this fit?
Yup. Thanks! Jim -- Jim Fulton Zope Corporation
participants (4)
-
Adam GROSZER -
Jacob Holm -
Jim Fulton -
Tim Peters