alan runyan wrote:
I hope you dont mind, but im cc'ing the list-- ppl need to see this
one of the scariest things i can imagine is a customer calling up saying the entire *monolithic* database is corrupt. the entire workday would grind to a halt and someone would have to jump through lots of hoops to just get the database (zope filesystem) back in order :(((( this is unacceptable (period) for some of the uses i had in store for Zope.
The risk here is identical to a corrupted ext2 filesystem, which is monolithic. This is also not much different that what other relational databases provide, ie, your data in a prorietary format in one file. Some even use their own partition! This is not to say that your comment is not valid. But many of your worries are mitigated every day for many 'monolithic' systems buy ensuring clean system shutdowns, redundant information storage (like RAID), the use of journaling filesystems, and the *regular backing up of data*. Further Data.fs files don't just up and corrupt themselves, just like ext2 filesystems do not randomly corrupt themselves. ZODB has evolved over years and is quite stable. Something has to be done to have put the Data.fs file in a corrupt state, such as the partition running out of space (Note Zope is smart enough to detect incomplete transaction due to insufficient space on the next startup) or unclean system shutdowns (where the file buffers do not get flushed). Once again these are risks that are mitigated by various means such as those above (and perhaps some duct-tape on the power switch).
is zope going to support multiple databases?
Yes.
jim f. said the ZODB is a SPOF -- are there any plans on fixing this?
No, it's not broken. It is no more so a single point of failure than any other non-redundant component of your system.
is there anyway to 'block all writes' while backing up the ZODB?
Yes, block all requests, or at least those requests that you know will write data.
w/o taking the entire website down/mostly reads anyways?
No, just like backing up any system, there is a give an take here. If you want to back up a UNIX box but still work on it, then you run the risk of getting a backup that is not entirely current. If you want to guarentee currency, then boot down to single user mode. Same situation in Zope, if you want absolute currency, then you must do the equivelent of bring Zope to 'single user mode', ie, don't let anyone log in. Note that a Zope backup is still VALID if users are still writing data to it, it just might not be absolutly CURRENT.
is there a way you can 'scan' to see if the ZODB is intact? i.e. having a process scan the ZODB and return whether or not it is valid? what was the 'offending' transaction?
It is true that Data.fs corruption is difficult to diagnose and fix, and that ext2 has a fancy tool (fsck) to help you with your linux filesystem and Zope should also provide such a tool. Ty Sarna's 'Tranalyzer' is a good place to start. -Michel
thanks ~alan
----- Original Message ----- From: "M. Adam Kendall" <mak@kha0s.org> To: "alan runyan" <runyaga@thisbox.com> Sent: Sunday, February 20, 2000 1:52 AM Subject: RE: data.fs corruption
Of course, you probably aren't going to like this answer, but I spent about 4.5 hours with the Tranalyzer product, and a good hex editor hacking away at my Data.fs. The good news: I recovered all my data. The bad news: I have alot less hair now..
I basically used Tranalyzer to figure out where it blew up, and then from the comments in FileStorage.py in the Zope source I was able to figure out the file format for the Data.fs. At least I knew how many bytes record/field lengths were supposed to be. Then using the hex editor I manipulated the Data.fs until I had ripped out the offending transaction record.
Worked like a charm but was a very long drawn out process.. I would say that Digital Creations needs to provide a tool for easy data recover from corrupt Data.fs.
Adam
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )