var/Data.fs: any way to regenerate from scratch?
We had a particularly impressive server crash a couple of weeks back, where our RAID controller blew up ... the result was some pretty impressive data corruption ... since the crash, we've been having a problem with Zope 2.5.1, particularly the ability to manage the site. When trying to Add Folder, it results in an error of: Traceback (innermost last): File /usr/local/www/Zope/lib/python/ZPublisher/Publish.py, line 150, in publish_module File /usr/local/www/Zope/lib/python/ZPublisher/Publish.py, line 127, in publish File /usr/local/www/Zope/lib/python/ZPublisher/Publish.py, line 127, in publish File /usr/local/www/Zope/lib/python/ZPublisher/Publish.py, line 127, in publish File /usr/local/www/Zope/lib/python/ZPublisher/Publish.py, line 122, in publish File /usr/local/www/Zope/lib/python/Zope/__init__.py, line 126, in zpublisher_exception_hook File /usr/local/www/Zope/lib/python/ZPublisher/Publish.py, line 102, in publish File /usr/local/www/Zope/lib/python/Zope/__init__.py, line 173, in commit File /usr/local/www/Zope/lib/python/ZODB/Transaction.py, line 234, in commit File /usr/local/www/Zope/lib/python/ZODB/Connection.py, line 348, in commit (Info: (('OFS.Folder', 'Folder'), '\x00\x00\x00\x00\x00\x003\x97', '')) File /usr/local/www/Zope/lib/python/ZODB/FileStorage.py, line 668, in store (Object: /usr/local/www/Zope/var/Data.fs) ConflictError: database conflict error (oid 0000000000003397, serial was 034b1b936fdaccb3, now 0000000000000000) I tried to run the fsrecover.py script on the Data file (a copy of it), and it closes with: exceptions.ValueError: unpack list of wrong size So I'm figuring that the Data.fs must be one of the casualties of the crash ... Now, Zope is a new one for me, and really have no idea of what the Data.fs is even used for ... is it something that I can just delete and Zope will just start building it from scratch? Or is it something that I can regenerate based on information from the file system? I found the following link: http://www.zope.org/Members/itamar/CorruptedZODB And tried it, but it made no difference ... from what I read in the URL, it sounds like I'm basically removing transactions, one at a time, from Data.fs ... what does that affect (ie. what am I losing for each transaction I remove)? how far back can/should I remove? Pointers, as well as advice, is greatly appreciated ... Thanks ..
I tried to run the fsrecover.py script on the Data file (a copy of it), and it closes with:
exceptions.ValueError: unpack list of wrong size
So I'm figuring that the Data.fs must be one of the casualties of the crash ...
Now, Zope is a new one for me, and really have no idea of what the Data.fs is even used for ... is it something that I can just delete and Zope will just start building it from scratch? Or is it something that I can regenerate based on information from the file system?
I found the following link:
http://www.zope.org/Members/itamar/CorruptedZODB
And tried it, but it made no difference ... from what I read in the URL, it sounds like I'm basically removing transactions, one at a time, from Data.fs ... what does that affect (ie. what am I losing for each transaction I remove)? how far back can/should I remove?
The Data.fs contains your entire Zope installation. Everything else is replacable. The effects of removing transactions from your ZODB are as you might expect: your newer data goes away one by one. In general you don't have much choice as to how much you lose before things get fixed. There was (is) a very recent thread on this list about recovering data from a corrupt Data.fs with a lot of suggestions. That's probably more instructive than any single answer I can give. But -- the simple things to try are: recover from backup fsrecover.py export/import to a new installation You might look into DirectoryStorage for future use: I know I am. --jcc
On Wed, 26 Mar 2003, J Cameron Cooper wrote:
I tried to run the fsrecover.py script on the Data file (a copy of it), and it closes with:
exceptions.ValueError: unpack list of wrong size
So I'm figuring that the Data.fs must be one of the casualties of the crash ...
Now, Zope is a new one for me, and really have no idea of what the Data.fs is even used for ... is it something that I can just delete and Zope will just start building it from scratch? Or is it something that I can regenerate based on information from the file system?
I found the following link:
http://www.zope.org/Members/itamar/CorruptedZODB
And tried it, but it made no difference ... from what I read in the URL, it sounds like I'm basically removing transactions, one at a time, from Data.fs ... what does that affect (ie. what am I losing for each transaction I remove)? how far back can/should I remove?
The Data.fs contains your entire Zope installation. Everything else is replacable.
'K, what is the MySQL backend used for? (sorry, forgot to mention in my original post)
The ZODB which is stored in your Data.fs is a OODBMS. Whomever is using your Zope instance is probably storing all of his python scripts, html, dtml, zpt's, etc in the ZODB (Data.fs) but storing actual business data in a MySQL installation. (ie, the scripts and pages used to show user data are in Data.fs, the user data itself is probably in the MySQL instance). ps. Still at Acadia Marc? ;) - Rocky Burt
--------- Original Message --------- From: Marc G. Fournier [mailto:scrappy@hub.org] Sent: March 26, 2003 16:36:55 -0400 To: J Cameron Cooper, zope@zope.org Subject: Re: [Zope] var/Data.fs: any way to regenerate from scratch?
On Wed, 26 Mar 2003, J Cameron Cooper wrote:
I tried to run the fsrecover.py script on the Data file (a copy of it), and it closes with:
exceptions.ValueError: unpack list of wrong size
So I'm figuring that the Data.fs must be one of the casualties of the crash ...
Now, Zope is a new one for me, and really have no idea of what the Data.fs is even used for ... is it something that I can just delete and Zope will just start building it from scratch? Or is it something that I can regenerate based on information from the file system?
I found the following link:
http://www.zope.org/Members/itamar/CorruptedZODB
And tried it, but it made no difference ... from what I read in the URL, it sounds like I'm basically removing transactions, one at a time, from Data.fs ... what does that affect (ie. what am I losing for each transaction I remove)? how far back can/should I remove?
The Data.fs contains your entire Zope installation. Everything else is replacable.
'K, what is the MySQL backend used for? (sorry, forgot to mention in my original post)
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
On Wed, 26 Mar 2003, Rocky Burt wrote:
The ZODB which is stored in your Data.fs is a OODBMS. Whomever is using your Zope instance is probably storing all of his python scripts, html, dtml, zpt's, etc in the ZODB (Data.fs) but storing actual business data in a MySQL installation. (ie, the scripts and pages used to show user data are in Data.fs, the user data itself is probably in the MySQL instance).
Damn, that's what I feared :( This is going to be a nightmare :( Okay, next question ... as far as I can tell, the site itself is working great, from the users perspective, its just when we try to go into the 'manage' interface to add new folders and such that we have the problem ... we have yet to find a 'bad page' in the users front end ... if the Data.fs was corrupted, based on how you describe how it works above, shouldn't the users front end also be broken?
ps. Still at Acadia Marc? ;)
Yup ...
Damn, that's what I feared :( This is going to be a nightmare :(
Okay, next question ... as far as I can tell, the site itself is working great, from the users perspective, its just when we try to go into the 'manage' interface to add new folders and such that we have the problem ... we have yet to find a 'bad page' in the users front end ... if the Data.fs was corrupted, based on how you describe how it works above, shouldn't the users front end also be broken?
Depending on in what fashion your Data.fs is corrupted... could be that only certain area's are broken. Could be anything in the ZMI (Zope Manage Interface) triggering it. I have no experience with trying to fix a corrupted Data.fs. But one idea might be to transfer as much out of that Data.fs into another Data.fs as possible. And then start redoing missing pieces in the second Data.fs (you can transfer content from one Zope to another Zope with tools like ZSyncer). - Rocky
Damn, that's what I feared :( This is going to be a nightmare :(
Okay, next question ... as far as I can tell, the site itself is working great, from the users perspective, its just when we try to go into the 'manage' interface to add new folders and such that we have the problem ... we have yet to find a 'bad page' in the users front end ... if the Data.fs was corrupted, based on how you describe how it works above, shouldn't the users front end also be broken?
Depends on how things are broken. I suspect that the data structures for appending transactions are borked. (The unpack list error seems to support that.) In cases of Data.fs corruption, this is not an uncommon symptom. The good news is that you can still read things. At worst, you end up doing a bunch of copy/pasting of source between browser windows. If you can install ZSyncer, go for it. Else, the Zope export/import facilites should help a lot. --jcc
On Wednesday 26 March 2003 2:43 am, Marc G. Fournier wrote:
ConflictError: database conflict error (
This is the oid of your object that has a problem. I guess this is the oid assigned to your new new folder.
oid 0000000000003397,
This is the serial number of the revision of this object that was loaded at the start of this transaction. All zeros means it want loaded - it was newly created.
serial now 0000000000000000)
This is the serial number of the most recent version in the storage. But the object is newly created.... it shouldnt be in there! It looks like FileStorage has assigned the new object an oid that is already in use, but then noticed its mistake when comitting the transaction.
serial was 034b1b936fdaccb3,
I dont see why this would be caused by a corrupt data.fs, so your data might still be intact. Some things to try..... 0. If you havent already, take a backup of data.fs and data.fs.index 1. Are you using ZEO? if so, which version? try without. 2. Are there any unusual messages in the stupid log file that might give a clue about why it has assigned a bogus oid? 3. Try shutting down zope/zeo, deleting data.fs.index, and restarting. This file is a cache used to improve startup time. This first restart will be a little slower than normal because FileStorage is rebuilding its index - and hopefully this will fix your problem. -- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson
Woo hoo!! Removing the index file appears to have done it ... have to do some more testing, but I could actually create a folder!! :) Thanks :) On Thu, 27 Mar 2003, Toby Dickenson wrote:
On Wednesday 26 March 2003 2:43 am, Marc G. Fournier wrote:
ConflictError: database conflict error (
This is the oid of your object that has a problem. I guess this is the oid assigned to your new new folder.
oid 0000000000003397,
This is the serial number of the revision of this object that was loaded at the start of this transaction. All zeros means it want loaded - it was newly created.
serial now 0000000000000000)
This is the serial number of the most recent version in the storage. But the object is newly created.... it shouldnt be in there! It looks like FileStorage has assigned the new object an oid that is already in use, but then noticed its mistake when comitting the transaction.
serial was 034b1b936fdaccb3,
I dont see why this would be caused by a corrupt data.fs, so your data might still be intact. Some things to try.....
0. If you havent already, take a backup of data.fs and data.fs.index
1. Are you using ZEO? if so, which version? try without.
2. Are there any unusual messages in the stupid log file that might give a clue about why it has assigned a bogus oid?
3. Try shutting down zope/zeo, deleting data.fs.index, and restarting. This file is a cache used to improve startup time. This first restart will be a little slower than normal because FileStorage is rebuilding its index - and hopefully this will fix your problem.
-- Toby Dickenson http://www.geminidataloggers.com/people/tdickenson
Woo hoo!! Removing the index file appears to have done it ... have to do some more testing, but I could actually create a folder!! :)
Hum. That sounds about right, actually. A bad index would explain why object pointers would be way off. Still doesn't satisfy me as to the error on fsrecover, but, hey, if it works I'm not going to argue. (Does fsrecover use the index (through the ZODB libs) if it exists?) --jcc
participants (4)
-
J Cameron Cooper -
Marc G. Fournier -
Rocky Burt -
Toby Dickenson