recover from POSKeyError after corruption by pack()ing bug
Hi! I have a serious problem: I think about a week ago my database was damaged by this nasty pack()ing bug. Unfortunately this error was not noticeable until my zope restart yesterday. :-( The consequence is a POSKeyError (POSKeyError: 0000000000000009) which prevents zope from starting. I tried to repair the database with fsrecover.py (from zope-2.6.3 and zope-2.6.4rc1), and it deleted some invalid transactions but it did not solve the problem. Then I tried the instructions on http://www.zopelabs.com/cookbook/1054240694, but had no success at all because I got again a POSKeyError. *g* I used fsrefs.py on the Data.fs and the first 2 error it found was: --- cut here --- oid 0x4 AccessControl.User.UserFolder last updated: 2004-01-16 14:08:02.781121, tid=0x3526BB00BDDB9CCL refers to invalid object: oid 0x9 missing: 'Persistence.PersistentMapping' oid 0xf17 AccessControl.User.UserFolder last updated: 2004-01-16 14:08:14.036884, tid=0x3526BB03BE40544L refers to invalid object: oid 0x1162 missing: 'Persistence.PersistentMapping' --- cut here --- After these 2 errors follow about 24Kb of errors or references to missing objects or objects that could not be loaded. It seems to me, that some very basic objects were destroyed a week ago. Of course all backups during the last week contain the damaged database. :-( It is really a pity, that the consequences of this database damage shows not up until the next zope restart... Is there any possibility to recover the database or to "extract" the transactions of the last 8 days and to prevent the complete loss of data in that time? Has anybody an idea, which I could try? Bye, Dominik P.S. I hope I got the suitable mailing list, I was uncertain if zope.general or zope.zobd fits better...
I have had this problem and worked through it, through a ton of trial and error. Basically, using all the tricks with that Zope labs how-to and with fsrecover I found that there was a bad object. Once I deleted that object the errors went away. I am sorry there is not a lot of help on this one, it is just a strange problem. Jake -- http://www.ZopeZone.com Dominik Schäfer said:
Hi!
I have a serious problem: I think about a week ago my database was damaged by this nasty pack()ing bug. Unfortunately this error was not noticeable until my zope restart yesterday. :-( The consequence is a POSKeyError (POSKeyError: 0000000000000009) which prevents zope from starting. I tried to repair the database with fsrecover.py (from zope-2.6.3 and zope-2.6.4rc1), and it deleted some invalid transactions but it did not solve the problem. Then I tried the instructions on http://www.zopelabs.com/cookbook/1054240694, but had no success at all because I got again a POSKeyError. *g* I used fsrefs.py on the Data.fs and the first 2 error it found was: --- cut here --- oid 0x4 AccessControl.User.UserFolder last updated: 2004-01-16 14:08:02.781121, tid=0x3526BB00BDDB9CCL refers to invalid object: oid 0x9 missing: 'Persistence.PersistentMapping'
oid 0xf17 AccessControl.User.UserFolder last updated: 2004-01-16 14:08:14.036884, tid=0x3526BB03BE40544L refers to invalid object: oid 0x1162 missing: 'Persistence.PersistentMapping' --- cut here --- After these 2 errors follow about 24Kb of errors or references to missing objects or objects that could not be loaded. It seems to me, that some very basic objects were destroyed a week ago. Of course all backups during the last week contain the damaged database. :-( It is really a pity, that the consequences of this database damage shows not up until the next zope restart... Is there any possibility to recover the database or to "extract" the transactions of the last 8 days and to prevent the complete loss of data in that time? Has anybody an idea, which I could try?
Bye, Dominik
P.S. I hope I got the suitable mailing list, I was uncertain if zope.general or zope.zobd fits better...
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Dominik Schäfer wrote at 2004-1-26 21:44 +0100:
I have a serious problem: I think about a week ago my database was damaged by this nasty pack()ing bug. Unfortunately this error was not noticeable until my zope restart yesterday. :-(
Hopefully, you did not pack again in the meantime. In this case, you get at least the state from the pack in "Data.fs.old".
... Is there any possibility to recover the database
Pack has physically removed a lot of transaction and object records. There is no way to reconstruct them from this (corrupted) file.
or to "extract" the transactions of the last 8 days and to prevent the complete loss of data in that time? Has anybody an idea, which I could try?
This is possible but it will not be easy: The idea is that emulate packing in some way. Packing essentially consists of 2 phases. In the first, the packing, phase transactions records are copied into the pack file. During this copying, object records for no longer reachable objects or no longer current are dropped. If a transaction happens to get empty, it is dropped as a whole. This phase ends when the packing time is reached. In the second, the copying, phase, transaction records after the packing time are copied to the pack file. In this process, no object records or transactions are deleted. However, pointers are adjusted. The idea to recover your data would be to emulate packing by processing your "Data.fs.old" in the packing phase and all new transaction records in the copying phase. This will require some tweaking as in your case, both segments reside in different files (while usually, they come from the same file).
P.S. I hope I got the suitable mailing list, I was uncertain if zope.general or zope.zobd fits better...
zodb-dev@zope.org might have been even better... -- Dieter
Hi!
I have a serious problem: I think about a week ago my database was damaged by this nasty pack()ing bug. Unfortunately this error was not noticeable until my zope restart yesterday. :-( Hopefully, you did not pack again in the meantime. In this case, you get at least the state from the pack in "Data.fs.old". In fact, I did. Every night with a cronjob which does packing (days=7) and backups the database after packing... This is my usual procedure and it was also done last week because I didn't about the problem. It is a pity, that Zope doesn't check if the result of the packing process is OK. Perhaps I should set up a separate zope instance which is used to check the backup of the database after it is created and will issue some error mail if zope doesn't even start correctly with this backup. But I think that would not help very much, because not all errors in Data.fs result in Zope not starting... :-/
Is there any possibility to recover the database Pack has physically removed a lot of transaction and object records. There is no way to reconstruct them from this (corrupted) file. Ok, I expected and feared this... ;-)
or to "extract" the transactions of the last 8 days and to prevent the complete loss of data in that time? Has anybody an idea, which I could try? This is possible but it will not be easy: The idea is that emulate packing in some way.
Packing essentially consists of 2 phases. [...] The idea to recover your data would be to emulate packing by processing your "Data.fs.old" in the packing phase and all new transaction records in the copying phase.
This will require some tweaking as in your case, both segments reside in different files (while usually, they come from the same file). OK, so I would have to write a program which does this? Uuhh... I think I have a very basic idea, how Zope works and a some intermediate experience in Python, but I don't believe, that I could actually write a program, which performs this process, you described... Anyway, because I packed the database in the meantime (several times), this procedure would not work here, doesn't it? Seems, that I have to live with the data loss. :-( But nonetheless: Thank you for your efforts. ;-)
Bye, Dominik
Dominik Schäfer wrote at 2004-1-28 10:59 +0100:
... recovering lost data due to packing bug ...
The idea is that emulate packing in some way.
Packing essentially consists of 2 phases. [...] The idea to recover your data would be to emulate packing by processing your "Data.fs.old" in the packing phase and all new transaction records in the copying phase.
This will require some tweaking as in your case, both segments reside in different files (while usually, they come from the same file). ... Anyway, because I packed the database in the meantime (several times), this procedure would not work here, doesn't it?
You definitely need a backup copy of an uncorrupted database. Had you not packed in the meantime, "Data.fs.old" would have been such a backup. The packing in between also makes it a bit more difficult to determine which transactions to copy. It would still be possible though (if you have a backup copy of an uncorrupted database). -- Dieter
On Tuesday 27 January 2004 19:30, Dieter Maurer wrote:
The idea to recover your data would be to emulate packing by processing your "Data.fs.old" in the packing phase
That half is easy - its a file copy
and all new transaction records in the copying phase.
I think you can do that with copyTransactionFrom, but you would need to implement some filter for the storage iterator to prevent it copying *all* transactions. You just need the new ones.
This will require some tweaking as in your case,
Indeed -- Toby Dickenson
participants (4)
-
Dieter Maurer -
Dominik Schäfer -
Jake -
Toby Dickenson