[ZODB-Dev] Re: Database Corruption.

[Ramon Aseniero]

Hi Tim,

No I don't truncate the data.fs at a random time and size.

I only time I truncate it is when the database gets corrupted and the
site crashes, but I followed this instructions
http://www.zope.org/Members/itamar/CorruptedZODB for truncating the
data.fs

Below are more log messages from event.log


------------------------------------------------------------------------
----
------
2004-06-28T18:02:42 ERROR(200) ZODB Couldn't load state for
'\x00\x00\x00\x00\x00\x03\x02L'
Traceback (innermost last):
  File /usr/lib/zope/lib/python/ZODB/Connection.py, line 551, in
setstate
  File /usr/lib/zope/lib/python/ZODB/FileStorage.py, line 645, in load
    (Object: /village/zope/default/var/Data.fs)
  File /usr/lib/zope/lib/python/ZODB/FileStorage.py, line 623, in _load
    (Object: /village/zope/default/var/Data.fs)
error: unpack str size does not match format
------
2004-06-28T18:07:19 ERROR(200) ZODB Couldn't load state for
'\x00\x00\x00\x00\x00\x02y\x89'
Traceback (innermost last):
  File /usr/lib/zope/lib/python/ZODB/Connection.py, line 551, in
setstate
  File /usr/lib/zope/lib/python/ZODB/FileStorage.py, line 645, in load
    (Object: /village/zope/default/var/Data.fs)
  File /usr/lib/zope/lib/python/ZODB/FileStorage.py, line 625, in _load
    (Object: /village/zope/default/var/Data.fs)
CorruptedDataError:
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@
------
2004-06-28T18:09:06 ERROR(200) ZODB Couldn't load state for
'\x00\x00\x00\x00\x00\x02y\x89'
Traceback (innermost last):
  File /usr/lib/zope/lib/python/ZODB/Connection.py, line 551, in
setstate
  File /usr/lib/zope/lib/python/ZODB/FileStorage.py, line 645, in load
    (Object: /village/zope/default/var/Data.fs)
  File /usr/lib/zope/lib/python/ZODB/FileStorage.py, line 625, in _load
    (Object: /village/zope/default/var/Data.fs)
CorruptedDataError:
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@
------
2004-06-28T18:10:04 ERROR(200) ZODB Couldn't load state for
'\x00\x00\x00\x00\x00\x02\xfd\x1d'
Traceback (innermost last):
  File /usr/lib/zope/lib/python/ZODB/Connection.py, line 551, in
setstate
  File /usr/lib/zope/lib/python/ZODB/FileStorage.py, line 645, in load
    (Object: /village/zope/default/var/Data.fs)
  File /usr/lib/zope/lib/python/ZODB/FileStorage.py, line 625, in _load
    (Object: /village/zope/default/var/Data.fs)
CorruptedDataError:
^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@
------------------------------------------------------------------------
----

Thanks,
Ramon

-----Original Message-----
From: Tim Peters [mailto:tim at zope.com] 
Sent: Wednesday, June 30, 2004 11:30 AM
To: jim at zope.com; 'Ramon Aseniero'
Cc: zodb-dev at zope.org
Subject: RE: [ZODB-Dev] Re: Database Corruption.

[Ramon Aseniero]
> 2004-06-28T18:02:30 ERROR(200) ZODB Couldn't load state for
> '\x00\x00\x00\x00\x00\x02y\x89' Traceback (innermost last):
>   File /usr/lib/zope/lib/python/ZODB/Connection.py, line 551, in
setstate
>   File /usr/lib/zope/lib/python/ZODB/FileStorage.py, line 645, in load
>     (Object: /village/zope/default/var/Data.fs)
>   File /usr/lib/zope/lib/python/ZODB/FileStorage.py, line 623, in
_load
>     (Object: /village/zope/default/var/Data.fs)
> error: unpack str size does not match format

This is certainly peculiar.  The failing unpack is the last line here:

    def _load(self, oid, version, _index, file):
        try:
            pos = _index[oid]
        except KeyError:
            raise POSKeyError(oid)
        file.seek(pos)
        read = file.read
        h = read(DATA_HDR_LEN)
        doid, serial, prev, tloc, vlen, plen = unpack(DATA_HDR, h)

The only ways I see that we *could* get a "str size does not match
format"
error are:  (a) I/O on this box is plain broken; or, (b) the object is
at
the very end of a "truncated" .fs file, so that fewer than DATA_HDR_LEN
bytes remain to be read.

I don't recall ever seeing a report of this failure mode before.

Ramon, I assume you don't truncate your Data.fs file to random sizes at
random times <wink>.