Hi all. We're experiencing nightly corruption of a Data.fs in an intranet application. We've removed all the packing/backup stuff, we've moved from zope 2.6.4 to zope 2.7.4, but we still have data corruption in the morning. Here is the error from the event log: 2005-02-22T09:21:51 ERROR(200) ZODB Couldn't load state for 0x3b3939 Traceback (most recent call last): File "/home/zope/zope-2.7.4/lib/python/ZODB/Connection.py", line 597, in setstate p, serial = self._storage.load(oid, self._version) File "/home/zope/zope-2.7.4/lib/python/ZODB/FileStorage.py", line 689, in load return self._load(oid, version, self._index, self._file) File "/home/zope/zope-2.7.4/lib/python/ZODB/FileStorage.py", line 663, in _load doid, serial, prev, tloc, vlen, plen = unpack(DATA_HDR, h) error: unpack str size does not match format ------ 2005-02-22T09:22:07 ERROR(200) ZODB Couldn't load state for 0x3214dc Traceback (most recent call last): File "/home/zope/zope-2.7.4/lib/python/ZODB/Connection.py", line 597, in setstate p, serial = self._storage.load(oid, self._version) File "/home/zope/zope-2.7.4/lib/python/ZODB/FileStorage.py", line 689, in load return self._load(oid, version, self._index, self._file) File "/home/zope/zope-2.7.4/lib/python/ZODB/FileStorage.py", line 663, in _load doid, serial, prev, tloc, vlen, plen = unpack(DATA_HDR, h) error: unpack str size does not match format ------ 2005-02-22T09:23:01 ERROR(200) ZODB Couldn't load state for 0x0eed Traceback (most recent call last): File "/home/zope/zope-2.7.4/lib/python/ZODB/Connection.py", line 597, in setstate p, serial = self._storage.load(oid, self._version) File "/home/zope/zope-2.7.4/lib/python/ZODB/FileStorage.py", line 689, in load return self._load(oid, version, self._index, self._file) File "/home/zope/zope-2.7.4/lib/python/ZODB/FileStorage.py", line 665, in _load raise CorruptedDataError, h CorruptedDataError: ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ I'm quite confident the problem is outside of Zope, but I would like to have some pointer on where to investigate in order to undestand the problem. Regards Marco
Ciao marco, there is a command-line utility in lib/python/ZODB/fsrecover.py that will repair damaged Data.fs files. First back up your Data.fs then
python lib/python/ZODB/fsrecover.py var/Data.fs
If the problem persist, you can try http://www.zope.org/Members/tsarna/Tranalyzer Regards. Massimiliano --------------------------------------------------------------------------- HT&T Consulting --------------------------------------------------------------------------- NOTICE: This e-mail message and all attachments transmitted with it may contain legally privileged and confidential information intended solely for the use of the addressee. If the reader of this message is not the intended recipient, you are hereby notified that any reading, dissemination, distribution, copying, or other use of this message or its attachments, hyperlinks, or any other files of any kind is strictly prohibited. If you have received this message in error, please notify the sender immediately by telephone (+39 587 757360) or by a reply to this electronic mail message and delete this message and all copies and backups thereof. --------------------------------------------------------------------------- -----Messaggio originale----- Da: zope-bounces@zope.org [mailto:zope-bounces@zope.org] Per conto di Marco Bizzarri Inviato: martedì 22 febbraio 2005 9.56 A: Zope Mailing List Oggetto: [Zope] Corrupted data.fs Hi all. We're experiencing nightly corruption of a Data.fs in an intranet application. We've removed all the packing/backup stuff, we've moved from zope 2.6.4 to zope 2.7.4, but we still have data corruption in the morning. Here is the error from the event log: 2005-02-22T09:21:51 ERROR(200) ZODB Couldn't load state for 0x3b3939 Traceback (most recent call last): File "/home/zope/zope-2.7.4/lib/python/ZODB/Connection.py", line 597, in setstate p, serial = self._storage.load(oid, self._version) File "/home/zope/zope-2.7.4/lib/python/ZODB/FileStorage.py", line 689, in load return self._load(oid, version, self._index, self._file) File "/home/zope/zope-2.7.4/lib/python/ZODB/FileStorage.py", line 663, in _load doid, serial, prev, tloc, vlen, plen = unpack(DATA_HDR, h) error: unpack str size does not match format ------ 2005-02-22T09:22:07 ERROR(200) ZODB Couldn't load state for 0x3214dc Traceback (most recent call last): File "/home/zope/zope-2.7.4/lib/python/ZODB/Connection.py", line 597, in setstate p, serial = self._storage.load(oid, self._version) File "/home/zope/zope-2.7.4/lib/python/ZODB/FileStorage.py", line 689, in load return self._load(oid, version, self._index, self._file) File "/home/zope/zope-2.7.4/lib/python/ZODB/FileStorage.py", line 663, in _load doid, serial, prev, tloc, vlen, plen = unpack(DATA_HDR, h) error: unpack str size does not match format ------ 2005-02-22T09:23:01 ERROR(200) ZODB Couldn't load state for 0x0eed Traceback (most recent call last): File "/home/zope/zope-2.7.4/lib/python/ZODB/Connection.py", line 597, in setstate p, serial = self._storage.load(oid, self._version) File "/home/zope/zope-2.7.4/lib/python/ZODB/FileStorage.py", line 689, in load return self._load(oid, version, self._index, self._file) File "/home/zope/zope-2.7.4/lib/python/ZODB/FileStorage.py", line 665, in _load raise CorruptedDataError, h CorruptedDataError: ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@ ^@^@^@^@ I'm quite confident the problem is outside of Zope, but I would like to have some pointer on where to investigate in order to undestand the problem. Regards Marco _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
On Tue, Feb 22, 2005 at 12:24:15PM +0100, trashMan wrote:
Ciao marco,
there is a command-line utility in lib/python/ZODB/fsrecover.py that will repair damaged Data.fs files.
First back up your Data.fs then
python lib/python/ZODB/fsrecover.py var/Data.fs
I just tried this and get lots of errors... such as This Python has API version 1012, module xxxx has version 1011. What to do?
Regards.
Massimiliano
--------------------------------------------------------------------------- HT&T Consulting ---------------------------------------------------------------------------
-- John
John Poltorak wrote:
On Tue, Feb 22, 2005 at 12:24:15PM +0100, trashMan wrote:
Ciao marco,
there is a command-line utility in lib/python/ZODB/fsrecover.py that will repair damaged Data.fs files.
First back up your Data.fs then
python lib/python/ZODB/fsrecover.py var/Data.fs
I just tried this and get lots of errors... such as
This Python has API version 1012, module xxxx has version 1011.
What to do?
It looks like you're using different releases of python from the one you've compiled Zope. Try using the same version as the one which is used for Zope. Regards Marco
On Tue, Feb 22, 2005 at 04:06:18PM +0100, Marco Bizzarri wrote:
John Poltorak wrote:
On Tue, Feb 22, 2005 at 12:24:15PM +0100, trashMan wrote:
Ciao marco,
there is a command-line utility in lib/python/ZODB/fsrecover.py that will repair damaged Data.fs files.
First back up your Data.fs then
python lib/python/ZODB/fsrecover.py var/Data.fs
I just tried this and get lots of errors... such as
This Python has API version 1012, module xxxx has version 1011.
What to do?
It looks like you're using different releases of python from the one you've compiled Zope.
Try using the same version as the one which is used for Zope.
How do I tell which version of Python was used to build Zope? And what do I do if I want to upgrade python? Do I also need to rebuild Zope?
Regards Marco
-- John
On Tue, Feb 22, 2005 at 03:11:35PM +0000, John Poltorak wrote: [snip]
How do I tell which version of Python was used to build Zope? And what do I do if I want to upgrade python? Do I also need to rebuild Zope?
Look at the scripts in the "bin" sub-directory of your Zope instance. In my case, both runzope and zopectl contain this: PYTHON="/usr/bin/python" So, run that executable. In my case, when I do, I see the following: ~/a1/Python/Zope/Instance7/bin [5] /usr/bin/python Python 2.3.4 (#2, Feb 2 2005, 11:10:56) [GCC 3.3.4 (Debian 1:3.3.4-9ubuntu5)] on linux2 Type "help", "copyright", "credits" or "license" for more information. This is important on my machine, because I have installed both Python 2.3 and 2.4 (2.4 is in /usr/local). Dave -- Dave Kuhlman http://www.rexx.com/~dkuhlman
[Marco Bizzarri]
We're experiencing nightly corruption of a Data.fs in an intranet application.
We've removed all the packing/backup stuff, we've moved from zope 2.6.4 to zope 2.7.4, but we still have data corruption in the morning.
Here is the error from the event log: ... error: unpack str size does not match format ... CorruptedDataError: ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
Note that those are the two symptoms described here, under "FileStorage corruption": http://zope.org/Wikis/ZODB/FileStorageBackup
I'm quite confident the problem is outside of Zope, but I would like to have some pointer on where to investigate in order to undestand the problem.
All we know about those is explained on the page referenced above. Best advice is to try running on different hardware. If that makes the problem go away (historically likely), running HW and I/O diagnostic/stress tests appropriate for your HW and OS may identify the failing component(s). The most recent of these I know of where the cause was definitely found turned out to be a RAID controller that failed only under high load (but normally worked fine, making the problem extra-hard to find). If you determine a cause, please let us know what it was.
participants (5)
-
Dave Kuhlman -
John Poltorak -
Marco Bizzarri -
Tim Peters -
trashMan