[ZODB-Dev] ConflictError: database conflict error

[Bob Horvath]

Tim Peters wrote:

>[Bob Horvath]
>  
>
>>
>>Are database errors in the middle of a Data.fs hopeless?
>>    
>>
>
>Uusally not, but by all accounts they're rarely easy to fix.  While I
>participate in these threads so that people don't feel too lonely <wink>, I
>should note that I've never repaired an error pointed out by fsrefs.py.
>That's simply because, in my time here, Zope Corp has never had an .fs file
>that fsrefs.py didn't like (or, if we did, nobody told me about it -- and
>that's not because they're shy about dumping .fs problems on me <wink>).
>  
>

It is a lonely feeling, but having finally found this mailing list, at 
least I feel there is hope, and it isn't quite the black magic it used 
to be. 


>Ethan Fremen finished repaired a damaged .fs file last week, and I suspect
>it started in worse shape than yours.  See the list archives for last week.
>Maybe you could interest him in taking on yours.  A problem is that once
>damage has persisted beyond the point of a backup, there's no purely
>mechanical way to repair anything -- someone has to spend (and possibly a
>lot of) time figuring out what role(s) the damaged objects play in your
>application, and craft ways to worm around the damage that minimize
>collateral damage to the rest of the app's data.  That usually requires
>knowing a lot about your app and how you use it, which pieces of your data
>are and aren't critical, how they all fit together, etc.
>  
>

Does the offset nature of it make it that much tougher?  Once you cut 
off something from the middle, are the other offsets are all thrown 
off?  Is there some way to have null objects or "noop" parts of the file 
to skip the bad stuff without throwing the rest of the offsets out of whack?

>It's somewhat akin to, say, pieces of your Unix mbox file getting
>overwritten with nonsense.  How do you repair that?  Well, if you don't have
>a backup, you stare at all the messages and try to guess the missing
>content, based on your extensive unique knowledge of your unique email
>usage.
>
>  
>

That makes a lot of sense, except either my Zope knowledge is still 
infantile, or ZODB and other stuctures aren't as simple as mbox, or more 
likely both.

One thought I had was would it be possible to write a tool that 
iteratively adds transactions one by one and does a fsrefs on it until 
an error is seen to see what transaction it is, and then skip it?  Again 
there is probably the offset problem.

What would be my next step in understanding enough to debug it?  Reading 
the FileStorage code? ZODB?