[Zope-dev] Re: Q: how to retrieve transactions from a Data.fs fragment?

[Jim Fulton]

Shawn Murphy wrote:
> 
> Yes....
> 
> We experienced a nasty confluence of suprises:
>  - CanOfRaid is imperfect,
>  - reiserfsck is imperfect,
>  - our backups were imperfect
> [We already knew we were imperfect.]

:]

> The result is that all we have left of about 6 weeks of several peoples
> development effort is two Data.fs scraps recovered from a reiser fs by one
> of the reiser developers.  We also have an intact 2 month old Data.fs
> which we would like to update with whatever we can retrieve from these two
> scraps.
> 
> Fragment#1 is 187MB where the first 147MB is nothing but nulls and the
> remainder is stuff that satisfies a tranalyzer rigged to not gag because
> of the missing leading 'FS21'.  The offsets within Fragment#1 are all
> correct.  That is, each transaction at the location in the file it thinks
> it ought to be.  Fragment#2 is 600KB and has about 500K of nulls at the
> beginning.  I am not yet sure if the offsets are 'proper' in it. I presume
> that these nulls are artifacts of the restoration process from the
> reiserfs.
> 
> So my question is:  What is the best strategy for getting the Folders,
> DTML Methods, DTML Documents and ZSQL Methods contained in these Data.fs
> fragments safely back into production?
> 
> Just to get the ball rolling....
> 
> I have a done a evening's worth of experimenting with tranalyzer to
> evaluate the state of these fragments and explore strategies for getting
> their contents back into service.

Have you tried the FileStorage recovery tool, lib/python/ZODB/fsrecover.py?

There's a reasonable chance that if you add the missing "FS21" at the
beginning and use fsrecover:

  cd lib/python
  ZODB/fsrecover.py file_name

you will be able to recover the good data. (Note that it
works in place, so make a copy.)

If this doesn't work, you might be able to figure something
out by reading the function that this script calls. 

> It seems that it will not work to simply append these zope transaction
> records to the end of a Data.fs because these transaction records have
> offset data built into them (the backpointer). 

Right. There can be multiple backpointers.

> Some form of any of the
> following could get the job done, but which one?  Is there some better
> approach?  An existing tool?  Guru wisdom appreciated...
> 
> Plan 1
> ======
> Simply massage the data so it has the right offset values and then
> tack it on the end of the old Data.fs.

You may be able to get away with concatinating your old file
and the damaaged one and then using fsrecover top remove the
nulls and tack the new data onto the old, adjusting the offsets.
 
> Cons:
>   What will happen with transactions which are edits of missing objects?

Edits aren't a problem. It's references to missing objects that will
be problematic.
 
> Plan 2
> ======
> Have the data drive a program which interacts with a running ZODB and
> replays the transactions.  Several ways to do this come to mind...
>   a) write  an http client to perform the operations against a normal,
>      running Zope
>   b) write a tool which uses FileStorage.py to read the frag and write to
>      the clean Data.fs

Take a look at the FileStorage iterator in Zope 2.3 and CVS and at the
BaseStorage copy method.
 
> Plan 3
> ======
> Make a human readable form suitable for cut and pasting into zope via
> the regular management screens.

Eek.
 
Jim

--
Jim Fulton           mailto:jim@digicool.com   Python Powered!        
Technical Director   (888) 344-4332            http://www.python.org  
Digital Creations    http://www.digicool.com   http://www.zope.org