[Zope] Corrupt Bobo Database?

[Jim Fulton]

Tim Hawes wrote:
> 
> Zope was running fine until a couple days ago. It seemed to have locked
> up. I tried restarting pcgi,
> but got a SERIOUS APPLICATION ERROR when I tried to re-connect. This is
> what I found in the log file:
> 
> Tue May  4 13:21:03 1999  unable to publish module
>   Traceback (innermost last):
>   File "/usr/local/https/Zope/pcgi/pcgi_publisher.py", line 301, in
> handler
> 
> self.publish_module(self.moduleName,stdin=stdin,stdout=stdout,stderr=stderr,environ=env)
> 
>   File "/usr/local/https/Zope/lib/python/ZPublisher/Publish.py", line
> 878, in publish_module
>     debug=debug)
>   File "/usr/local/https/Zope/lib/python/ZPublisher/Publish.py", line
> 325, in publish
>     (bobo_before, bobo_after, request_params,
>   File "/usr/local/https/Zope/lib/python/ZPublisher/Publish.py", line
> 616, in get_module_info
>     module=__import__(module_name, globals(), globals(), ('__doc__',))
>   File "/usr/local/https/Zope/lib/python/Main.py", line 103, in ?
>     Bobobase=OFS.Application.open_bobobase()
>   File "/usr/local/https/Zope/lib/python/OFS/Application.py", line 319,
> in open_bobobase
>     Bobobase=Globals.Bobobase=Globals.PickleDictionary(
>   File "/usr/local/https/Zope/lib/python/BoboPOS/PickleDictionary.py",
> line 159, in __init__
>     db=apply(DBType,(),storage_options)
>   File "/usr/local/https/Zope/lib/python/BoboPOS/SimpleDB.py", line 221,
> in __init__
>     self.open(file_name,create,revision_time,meta_index,read_only)
>   File "/usr/local/https/Zope/lib/python/BoboPOS/SimpleDB.py", line 287,
> in open
>     try: self.index,self.pos=read_index(
>   File "/usr/local/https/Zope/lib/python/BoboPOS/SimpleDB.py", line
> 1365, in read_index
>     raise DatabaseError, ('Corrupted data record at %s' % pos)
> ImportError: ('DatabaseError', 'Corrupted data record at 224044315')
> 
> I did not have much in the database, and it did fine after I moved all
> the database files into a new directory (except, of course, I had to
> start with a clean slate).

You had at least 22 megs.  Do you have any idea why it is that big?
Did you have a bunch of big file objects? What was the file size?

Note that you could, at least, truncate the file at 224044315
and most likely recover to whatever point you were at when you reached
that size.

What version of Zope were you running?  What OS/rev?
What products did you have installed?  Did you have MySQL?

Were the database files on a network (eg NFS) mounted 
partition?

> 
> Questions:
> 
> 1) What might have caused this corruption? 

I don't know.  Would you mind sending me (a gzipped tar ball or
win zip file of) the database files (in a separate email)?  I'd
like to take a look at them.

Depending on what version of Zope you are running, you might
get this sort of message if the time-stamps in your file are bad, 
possibly due to a system clock problem.  If this is the case, then
the data is not really corrupted and can be recovered.  Similarly, if
the database could not completely write a record, due to a disk being full
or due to a system crash, then this could happen with earlier versions
of Zope.

Using a network-mounted file system can definately cause problems.

I suspect that memory errors in the application or products
can cause the database to fail.  For example, a non-standard
extension module that has memory errors might cause this sort of
problem.

> Two people logged in as admin (superuser) at the same time?

No.

> Running a seperate instance of Zope (the
> latest CVS version) in another directory?

No.
 
> 2) Can a corrupted bobobase file be fixed relatively easily (i.e.
> without an hex editor)?

If the error occurs near the end of the file, you can simply truncate the
file at the location where the error occurs.  It should also be pretty
easy to provide a repair tool that cuts out damaged sections of the file.
We should do this.
 
> I am not fond of the idea of building an enterprise-wide web site with
> tens to hundreds developers working on the same site, and then see the
> whole thing get flushed down the toilet by the corruption of one file.

I don't blame you.  Of course, in such a situation, I assume that 
you would at least back up the database regularly.
 
> Possible remedies might include creating a seperate database file for
> each division. But then how is acl security maintained throughout the
> site? Can one database use objects from another?
> 
> Ideas, or answers?

I'd like to try to figure out why your file got damaged.

I am hard at work on the next generation of our database.  I home
to make a beta release of the new database very soon.  Some features
related to this problem include:

  - A more robust data format,

  - Support for multiple databases in one object system,

  - Ability alternate storage systems, such as RDBMS,

  - An optional client-server facility that separates database
    storage management from the application address space, so that
    errors in C modules don't lead to database problems.

Jim