[ZODB-Dev] Re: Database Corruption.

[Tim Peters]

[Ramon Aseniero]
> No I don't truncate the data.fs at a random time and size.

Yes, I was kidding about that.

> I only time I truncate it is when the database gets corrupted and the
> site crashes,

How frequently does that occur?  How long has it been going on?  Some time
with Google suggests you've been seeing it for at least 5 weeks, but haven't
been forthcoming about details that might help to solve it.

> but I followed this instructions
> http://www.zope.org/Members/itamar/CorruptedZODB for truncating the
> data.fs

Meaning you run fsrecover.py?  Or meaning something other/more than that?
Do the problems go away then for some time?  If so, for how long?

> Below are more log messages from event.log
...
> CorruptedDataError:
> ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@
...
> CorruptedDataError:
> ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@
...
> CorruptedDataError:
> ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@
^@^@^@^@^@^@

Has that been the most common symptom of corruption in your experience?

That output *suggests* you have unexpected blocks of NUL (0) bytes inside
your Data.fs file.  As Jeremy said, we have seen reports of that before, but
rarely, and they were never traced to a ZODB bug.  Some causes that are
known are outside of ZODB's control, and are sketched in this new message
thread:

    http://mail.zope.org/pipermail/zodb-dev/2004-July/007575.html

So please read that.

Since the most likely cause of parts of a FileStorage getting overwritten by
NUL bytes is currently believed to be HW or system SW problems (see the
thread referenced above), would it be possible for you to *try* running your
site on a different physical machine?

Also, as Jim asked before, can we get a copy of your .fs and .index files?

Finally, are you running any Zope products or Python extension modules that
aren't included with the core Zope distribution?  If so, what are they?