[ZODB-Dev] URGENT: ZODB down - Important Software Application at CERN

[Pedro Ferreira]

Hello,

We're using ZODB for the Indico Project (at CERN), since 2004, without
any kind of problem. However, today, our database went down and we can't
find a way to recover it. This is a major issue, since we have ~4000
users depending on this application, and we're simply not able to access
the data in any way.
Around 00:30 tonight the database went down, and since, all the
connections are refused. We tried to restart the database, but the
script seems to hang, while trying to create the index:

-rw-r--r--   1 root  root  6396734704 May 25 13:21 dataout.fs
-rw-r--r--   1 root  root         173 May 25 12:21 dataout.fs.index
-rw-r--r--   1 root  root   229755165 May 25 13:22
dataout.fs.index.index_tmp
-rw-r--r--   1 root  root           7 May 25 12:21 dataout.fs.lock
-rw-r--r--   1 root  root    70956364 May 25 13:21 dataout.fs.tmp

We tried to do fsrecovery, but it says "0 bytes removed during
recovery", and the result ends up being the same. We tried it in
different machines, with no success. In one of them, after a while
trying to create the index, a Python exception was thrown, saying
"maximum recursion depth exceeded".
We're using 3.4 in production, but we've tried with 3.9 and 3.9b0 as
well, with no success.
We're getting kind of desperate, since the same seems to happen with
yesterday's backup, and trying to restore previous backups with repozo
raises a CRC Error.
Has anyone ever experienced this? Any clues on how to solve this problem?
We'd really appreciate you could help us out, since this is becoming a
big issue here at CERN (a lot of people's work depends on this).

Thank you very much in advance,

Best regards,

Pedro Ferreira

-- 

José Pedro Ferreira
(Software Developer, Indico Project)

IT-UDS-AVC
CERN
Geneva, Switzerland

Office 513-R-042
Tel. +41 22 76 77159