Shawn Murphy wrote:
Yes....
We experienced a nasty confluence of suprises: - CanOfRaid is imperfect, - reiserfsck is imperfect, - our backups were imperfect [We already knew we were imperfect.]
:]
The result is that all we have left of about 6 weeks of several peoples development effort is two Data.fs scraps recovered from a reiser fs by one of the reiser developers. We also have an intact 2 month old Data.fs which we would like to update with whatever we can retrieve from these two scraps.
Fragment#1 is 187MB where the first 147MB is nothing but nulls and the remainder is stuff that satisfies a tranalyzer rigged to not gag because of the missing leading 'FS21'. The offsets within Fragment#1 are all correct. That is, each transaction at the location in the file it thinks it ought to be. Fragment#2 is 600KB and has about 500K of nulls at the beginning. I am not yet sure if the offsets are 'proper' in it. I presume that these nulls are artifacts of the restoration process from the reiserfs.
So my question is: What is the best strategy for getting the Folders, DTML Methods, DTML Documents and ZSQL Methods contained in these Data.fs fragments safely back into production?
Just to get the ball rolling....
I have a done a evening's worth of experimenting with tranalyzer to evaluate the state of these fragments and explore strategies for getting their contents back into service.
Have you tried the FileStorage recovery tool, lib/python/ZODB/fsrecover.py? There's a reasonable chance that if you add the missing "FS21" at the beginning and use fsrecover: cd lib/python ZODB/fsrecover.py file_name you will be able to recover the good data. (Note that it works in place, so make a copy.) If this doesn't work, you might be able to figure something out by reading the function that this script calls.
It seems that it will not work to simply append these zope transaction records to the end of a Data.fs because these transaction records have offset data built into them (the backpointer).
Right. There can be multiple backpointers.
Some form of any of the following could get the job done, but which one? Is there some better approach? An existing tool? Guru wisdom appreciated...
Plan 1 ====== Simply massage the data so it has the right offset values and then tack it on the end of the old Data.fs.
You may be able to get away with concatinating your old file and the damaaged one and then using fsrecover top remove the nulls and tack the new data onto the old, adjusting the offsets.
Cons: What will happen with transactions which are edits of missing objects?
Edits aren't a problem. It's references to missing objects that will be problematic.
Plan 2 ====== Have the data drive a program which interacts with a running ZODB and replays the transactions. Several ways to do this come to mind... a) write an http client to perform the operations against a normal, running Zope b) write a tool which uses FileStorage.py to read the frag and write to the clean Data.fs
Take a look at the FileStorage iterator in Zope 2.3 and CVS and at the BaseStorage copy method.
Plan 3 ====== Make a human readable form suitable for cut and pasting into zope via the regular management screens.
Eek. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org