[Zope] corrupted Data.fs (fwd)

[Jim Fulton]

oliver wrote:
> 
> Hello Jim,
> 
> Chris Withers told me to contact you for a tough problem with our Data.fs
> I don't know if you saw my email to the Zope list so I attached it to this
> one.

It was a good idea to send me a private email. :)

> One funny thing is that the invalid record number has changed somehow
> and is now 7472388L
> I tried truncating the Data.fs (which is now 579088915 bytes) to various
> values to see what happens (e.g. dd if=Data.fs.corr of=Data.fs bs=1M
> count=50) but that yielded the same results.
> Is there a way to fix my database?

This is very odd. I'm willing to look at this. 

I'm certainly interested in seeing data files that fsrecover
doesn't work on, so that I can make it work better.
I've heard of a case like this before. fsrecover only works on
bad transaction records, not bad data records. I've seen one or
two cases where data records were damaged but transaction records
weren't, but I've never had an actual data.fs file with this problem
to look at. I actually worked up a patch at one point, but was 
uncomfortable incoporating it without test data.

It would probably be a good idea to give me a login (ssh) to a machine
where the data file is, rather than trying to send it to me.
Alternatively, you could put a compressed copy on a CD and
FedEx it to me to look at tomorrow. Obviously, I need the 
original corrupted file.

Do you have a backup? There are certainly some hardware
failures that will render the file unusable. This is 
true for all databases.

Jim

> thanks for your help
> -Oliver
> 
> ---------- Forwarded message ----------
> Date: Thu, 06 Sep 2001 13:47:53 +0100
> From: Chris Withers <chrisw@nipltd.com>
> To: oliver <oliver@firstfloor.org>
> Cc: zope@zope.org
> Subject: Re: [Zope] corrupted Data.fs
> 
> oliver wrote:
> >
> > line 1455, in shift_transactions_forward
> >     if plen: p=read(plen)
> > OverflowError: long int too long to convert
> 
> This is a 'bad plen' according to Jim. FSRecover can't handle it :-(
> 
> Bu^heg Jim Fulton, he can help...
> 
> Chris
> 
>   ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
> 
> From oliver@firstfloor.org Thu Sep  6 15:55:09 2001
> Date: Thu, 6 Sep 2001 14:34:44 +0200 (CEST)
> From: oliver <oliver@firstfloor.org>
> To: zope@zope.org
> Subject: corrupted Data.fs
> 
> Hi,
> 
> I am running Zope 2.3.0 on Linux and after a power failure the db is
> corrupted:
> 
>  File "/home/ofrommel/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py", line
> 218, in panic
>     raise CorruptedTransactionError, message
> ZODB.FileStorage.CorruptedTransactionError:
> /home/ofrommel/Zope/var/Data.fs data record exceeds transaction record at
> 35816171L
> 
> now when I try to fix it with fsrecover I get the following error message:
> 
> [root@m945 var]# python ../lib/python/ZODB/fsrecover.py  Data.fs
> Traceback (innermost last):
>   File "../lib/python/ZODB/fsrecover.py", line 94, in ?
>     FileStorage.recover(sys.argv[1])
>   File "/home/ofrommel/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py",
> line 1529, in recover
>     npos = shift_transactions_forward(
>   File "/home/ofrommel/Zope-2.3.0-src/lib/python/ZODB/FileStorage.py",
> line 1455, in shift_transactions_forward
>     if plen: p=read(plen)
> OverflowError: long int too long to convert
> 
> how can I fix the database now?
> 
> thanks for your help
> -oliver

--
Jim Fulton           mailto:jim@zope.com       Python Powered!        
CTO                  (888) 344-4332            http://www.python.org  
Zope Corporation     http://www.zope.com       http://www.zope.org