[ZODB-Dev] new FileStorage pack and POSKeyError

[Jeremy Hylton]

I've just checked in a new pack implementation for FileStorage.  It's a
backport of the new pack we developed for ZODB4 a few weeks ago.  The
new pack fixes several bugs, including the one reported by Dieter Maurer
(http://collector.zope.org/Zope/875).

One of the bugs that is fixed may explain the mystery POSKeyErrors that
several people have seen.  I'll explain the details in a minute.

Since this is a totally new pack implementation, there's still a good
chance that it has bugs.  We've expanded the pack test suite and code
coverage looks good, but that only goes so far.  I'd be very interested
to hear reports from people who can make a copy of a production Data.fs
and pack it with the new code.

The POSKeyError may be caused by a bug related to undo.  An undo
operation can undo the creation of an object.  In that case, the object
still exists in the database, but it is no longer reachable.  If the
undo-create operation is later undone, then the object will become
reachable again.  If a pack occurs and the pack time falls between an
undo-create and an undo-undo-create, then the old pack implementation
would fail to copy the created object.  After the pack, the parent
object will have a reference to the created object, but the database
won't have it.  Thus, you'd get a POSKeyError and no evidence that the
referenced object ever existed.

The bug is fixed in the new pack implementation.  It scans from the
packtime to the end of the database, looking for undo records and adding
the objects to which they refer to the rootset of garbage collection.

The new code will be available in ZODB 3.2.  I plan for a ZODB 3.2
release that will be compatible with Zope 2.6 and Python 2.1.  So Zope
2.6 users can get the fixed FileStorage without waiting for Zope 2.7.

Jeremy