[Zope-DB] Re: Database Corruption (1 possible solution)

[Nick Bower]

Here is some more info just in case it's of use.

An end-user had reported problems logging in.  Pages (no doubt cached by 
ZEO clients) were being displayed, but interactivity generated tracebacks.

Upon checking, I found the ZODB was no longer running.  After failing to 
then start, I observed that the logs were owned by root (we of course 
start zodb as an unprivleged user so this was incorrect).  What had 
happened was that a sys-admin had started log rotation via SIGUSR 
(apprently he said this could be done - haven't found it myself), but 
issued the command as root, and subsequently the ZODB stopped because it 
couldn't write to it's logs anymore.  I wasn't actually involved in 
this, so can't be more specific.

So when finally having the zodb startup, event.log and "zopectl debug" 
only showed generic read conflict errors.  I *think* I had to use 
"runzope" instead to temporarily get more specific errors, and these 
highlighted the object reassignment problem which tipped me off to the 
fact that the index file may need regenerating.  I'm not sure this will 
be of use, but knowing that separate levels of error reporting occurs 
depending on how you start the ZEO clients may be handy.  I wish I could 
be more specific - sorry.

Nick


Eugene wrote:
> Hello Nick,
> 
> Wednesday, June 30, 2004, 9:47:59 PM, you wrote:
> 
> NB> Nope - the genericness of the reported error in event.log and many other
> NB> frustrated posts of users in lists/web pages experiencing similar things
> NB> (during frantic google searching) led me to believe ZODB problems like
> NB> this are commonplace.
> I tried to find info about a problem in logs. Unsuccessful :(
> I've even turn on trace.log.
> I see log only when I get Read Error - but I dont see moment when DB
> makes errorneous record.
> 
> NB> I'll see if I can dig up a log then.  The log was not particularly
> NB> descriptive however as the problem happened on a production ZEO platform
> NB> both times with logging turned down.
> 
> NB> Jim Fulton wrote:
> 
>>>Nick Bower wrote:
>>>
>>>
>>>>I was going to post this tip on it's own, but seeing how it's relevant
>>>>to the current discussion, I'll post it here.
>>>>
>>>>Recently we've been having "read conflict errors" preventing Zope from
>>>>connecting to a ZODB on startup in a ZEO deployment.  (Moving the 
>>>>Data.fs* to a standalone non-ZEO configuration made no difference).
>>>>
>>>>Upon a little digging, I found an error relating to an illegal 
>>>>attempted reassignment of an oid.
>>>>
>>>>I tried the recovery tools, but these did nothing.
>>>>
>>>>The solution was to remove the Data.fs.index and Data.fs.tmp files and
>>>>let ZODB rebuild them on startup.  Why this worked I have no idea (it
>>>>was a stab in the dark at the time) as I don't know much about ZODB
>>>>internals.  However, this happened to us twice in a single week (which
>>>>on it's own is quite sad).
>>>
>>>
>>>Did you report this to either the zodb-dev list or to the
>>>zodb bug tracker?
>>>
>>>Jim
>>>
> 
> 
> NB> _______________________________________________
> NB> Zope-DB mailing list
> NB> Zope-DB at zope.org
> NB> http://mail.zope.org/mailman/listinfo/zope-db
> 
> 
>