Richard Taylor wrote:

Today I had to role back two days of transactions from my production site because when I packed the database I was informed of a CorruptTransactionError.

Did anything else happen previous to this? Did you run out of space or anything like that? You should have been able to use Data.fs.old, which is a copy of the database before the pack to restore the data. Or was the error in there too? I'd be interested in looking at the Data.fs file before the pack to try to figure out what might have gone wrong. (If you send my your Data.fs file, please remember to send it to me privately and to zip or compress it. :)

We are using Zope for an internal knowledge management application where anyone in the organization can add objects. So I have no way of know what was added after the fateful transaction and no way of getting any of it back.

Bummer!

Indeed.

I think this raises a few questions about ZDB:

1) We need some tools for selectively removing bad transactions rather than just truncating Data.fs back to the last good one and loosing everything that comes after it.

Zope 2.2 has just such a tool. In the ZODB directory, there is a Python script, fsrecover.py which simply calls the recover function in the FileStorage module. This will work with any 2.x databases. It scans from both the beginning and the end of the file until it finds a corrupted section and then removes the corrupted portion from the file. You utility modifies the file in-place, so you need to shut-down the site, or work on a copy when you use it.

2) We could do with a tools that can verify the ZODB offline. This could then be run at regular intervals (maybe once an hour from cron) so that corruptions can be picked up earlier.

You could use a little Python script that did something like: import ZODB.FileStorage file_name='../../var/Data.fs' file=open(file_name, 'r+b') index={} vindex={} tindex=[] ZODB.FileStorage.read_index( file, file_name, index, vindex, tindex) This basically reads the FileStorage index as would normally be done during startup.

3) Some way to find out what was added after a corrupt transaction is needed so that at least I could see who I need to ask to re-add their stuff.

The fsrecover script should avoid the need for this. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.