[Zope] ZODB Key error in root Zope database

[Brian Brinegar]

We've got a bit of a situation...

We run Zope 2.11 with a main ZODB and a few dozen separate file storages
for sub-sites.

Occasionally, we've encountered the problem where a cut-and-paste occurs
between storages and upon a pack results in a key error. Normally, we
avoid this because the FileStorages represent individual sites.

Last night something happened where a script was cut-and-pasted from one
of the sub-storages into the main, root-level, database.

Upon a restart we encountered a key error:

  File "/var/local/zope/Zope-2.11.4-1/lib/python/ZODB/Connection.py",
line 361, in get_connection
    new_con = self._db.databases[database_name].open(
KeyError: 'site1'

Normally, we've encountered these errors when a database is packed and
the solution is to revert to a pre-pack database and remove or fix the
offending object.

In this case, we can't even start Zope or mount the storage in a script
to remove the offending items.

Before I go on, I found that ZODB 3.9 provides options to disallow
cross-database references and we'll work to get to this version during
our next upgrade cycle.

In the mean time we've reverted to a previous backup prior to the
offending object being placed in the root database. This resulted in
about 200 MB of data loss. I would like a way to mount the corrupted
database and attempt to determine exactly what was lost.

What we've tried so far has gone like this:

from ZODB import FileStorage, DB
from OFS.Folder import Folder

storage = FileStorage.FileStorage('BadData.fs')
db = DB(storage)
conn = db.open()
root = conn.root()
app = root['Application']

app.manage_deleteObjects(['OffendingObject'])

# This throws the same exception so we tried working around the
# mechanics of manage_deleteObjects with this.

app._objects = tuple([i for i in app._objects if i['id'] !=
'OffendingObject'])

Same exception. Can't access the database. We've tried using _setOb to
replace the object.

Any suggestions? It seems as though it needs to be able to access
_objects before it mounts the mounted databases.


Thanks,

-- 
Brian Brinegar
Web Services Coordinator
Engineering Computer Network