[ZODB-Dev] Recovering from BTree corruption

[Jim Fulton]

On Sep 10, 2007, at 10:34 AM, Alan Runyan wrote:

> Hi guys.
>
> It seems that one of our customers has a corrupted BTree. I would love
> for someone to provide some insight on how we can recover the data.
>
> we have two databases: 1 for resources and 1 for 'content'.  resources
> contain lots of very big files.
>
> The system is configured to have a mount point at /plone/resources is
> a subclass of BTreeFolder, using internal data struct of OOBTree.

And, as you said in another node, the BTree folder actually loves in  
the resources database.

> anytime I iterate over the keys I get POSKeyError.  anytime I iterate
> over the values the same.  if I run BTree.check() on the data
> structure's tree attribute (the OOBTree itself) I get a POSKeyError.
>
> Running the utils.checkbtrees doesnt say this btree has a problem.
>
> While debugging this I had a conversation with sidnei about mounted
> databases.  He recalled that if your using a mounted database you
> should not pack.  If for some reason your mounted database had a cross
> reference to another database and somehow you had a dangling reference
> to the other database it would cause POSKeyError.

Cross database references are inherently weak.  A reference from a  
foreign database doesn't prevent an object from being treated as  
garbage.  So, if the only reference to an object is from a foreign  
database, then the object is considered garbage.  It doesn't sound  
like this is what's affecting you.  The cross-database reference is  
to the BTree.  It sounds like the internal references are within  
database.



> Is there any other ways of "testing consistency" of FileStorage  
> other than:
>   - checkbtrees.py
>   - fstest.py

There's an fsrefs script that checks internal references I believe.

> And any ideas how I can salvage the data? This BTree, of course, had
> the most valuable data.

Possibly, there's a backup that has data records for the missing OIDs.

Jim

--
Jim Fulton			mailto:jim at zope.com		Python Powered!
CTO 				(540) 361-1714			http://www.python.org
Zope Corporation	http://www.zope.com		http://www.zope.org