[Zope-dev] corrupted data.fs

[Chris Kratz]

Hello all,

My apologies if this is not appropriate on this list and my apologies for 
this "book".  But, we have been having more and more problems with Zope and 
it is getting quite frustrating.  I thought since I had posted on the zope 
list and no one had any suggestions that I would try this list next.

The behavior we are seeing is as follows.  We are using versions heavily.  
There are two of us primary developers, soon to be three working heavily 
within the same data.fs.  We use versions, often running several at the same 
time.  When the code is ready, we save the version incorporating the changes 
to the live site.

Ever since we upgraded to Zope 2.4.1, saving the version rarely actually does 
that.  It will save some objects, but usually only a few.  We have to then go 
to each object and save it again (sometimes making changes) and continue 
saving the version until we get all the locked objects unlocked and 
committed.  This is very annoying, but not a show stopper.  What is starting 
to scare me is that I wonder if this problem with the version commits is 
somehow causing the corruption of the database.

Now, I say all this because it may or may not be related to the other 
problems we continue to have.  We periodically, get folders that are no 
longer visible within the ZMI and produce a keyerror in filestorage.py 
(traceback at end of msg).  But, the same folder is still happily being 
served externally (ie it generates the web pages based within that folder 
without error to our customers).  This tells me that it is still in the ZODB, 
it is just unreachable via the ZODB.  Since we cannot access it from 
within zope any longer, we have to load the data.fs from backup and restore 
the offending folder losing any work done since the previous night.  It never 
seems to effect any other parts of the website and it has happened in several 
different areas.  This is becoming more and more of a showstopper especially 
when we lose code.

1. I have packed the database.
2. I have run fsrecover.py over the data.fs. - Nothing to recover.  So, if we 
aren't corrupted, why can we no longer work with that folder?
3. I have exported and imported the offending folder without effect (XML and 
regular).
4. We were packing the zope database nightly via a cron job to control ZODB 
bloat.  I have turned that off for now on the theory that somehow mixing 
overnight packs with versions is causing problems.

Help!!! This continues to happen and it is really scary when your database 
continues to get corrupted underneath you on a live site when you are in 
heavy development.  Can anyone give any ideas on how to track down this 
problem?  Could it be with the versioning code?  Is there a better way to 
work with a production site and still get the benefits of the versioning?  Is 
versioning just dangerous and shouldn't be used?  Pardon my tone, but I am a 
little frustrated.

I will say that the last time this happened, thankfully, that particular 
folder was being worked on in a version and even though the folder was 
inaccessible outside of the version, when working within the version, it was 
still there.  So, when that version was cut in, we were again able to work 
with that folder.  This time however, the folder in question wasn't begin 
worked on in a version.

Thank you for your time,

Sincerely

-Chris

-------------

**Traceback**


Error Type: KeyError
Error Value: ?f

Traceback
Traceback (innermost last): File 
/usr/local/zope2.4/Zope-2.4.0-linux2-x86/lib/python/ZPublisher/Publish.py, 
line 171, in publish File 
/usr/local/zope2.4/Zope-2.4.0-linux2-x86/lib/python/ZPublisher/mapply.py, 
line 160, in mapply (Object: manage_main) File 
/usr/local/zope2.4/Zope-2.4.0-linux2-x86/lib/python/ZPublisher/Publish.py, 
line 112, in call_object (Object: manage_main) File 
/usr/local/zope2.4/Zope-2.4.0-linux2-x86/lib/python/Shared/DC/Scripts/Bindings.py, 
line 324, in __call__ (Object: manage_main) File 
/usr/local/zope2.4/Zope-2.4.0-linux2-x86/lib/python/Shared/DC/Scripts/Bindings.py, 
line 354, in _bindAndExec (Object: manage_main) File 
/usr/local/zope/lib/python/App/special_dtml.py, line 244, in _exec (Object: 
manage_main) File /usr/local/zope/lib/python/DocumentTemplate/DT_In.py, line 
711, in renderwob (Object: objectItems) File 
/usr/local/zope/lib/python/DocumentTemplate/DT_In.py, line 839, in 
sort_sequence (Object: objectItems) File 
/usr/local/zope2.4/Zope-2.4.0-linux2-x86/lib/python/ZODB/Connection.py, line 
519, in setstate File lib/python/ZODB/FileStorage.py, line 588, in load 
(Object: /usr/local/zope/var/Data.fs) File lib/python/ZODB/FileStorage.py, 
line 564, in _load (Object: /usr/local/zope/var/Data.fs) KeyError: ?f


----------------
**Instalation Info**

Platform: RH 7.2
Zope Version: (Zope 2.4.1 (binary release, python 2.1, linux2-x86), python 
2.1.1, linux2)
Python Version: 2.1.1 (#1, Aug 8 2001, 11:57:42) [GCC 2.96 20000731 (Red Hat 
Linux 7.1 2.96-81)]
System Platform: linux2

Products installed:

eCrumbler (Installed product CookieCrumbler (CookieCrumbler-0.3))
ExternalMethod (Installed product ExternalMethod (External Method-1-0-0))
MIMETools (Installed product MIMETools)
MailHost (Installed product MailHost (MailHost-1-2-0))
OFSP (Installed product OFSP (OFSP-1-0-0))
PlugIns (Installed product PlugIns (PlugIns-0-4-3p1))
PluginIndexes (Installed product PluginIndexes)
PythonScripts (Installed product PythonScripts (PythonScripts-2-0-0))
Renderable (Installed product Renderable)
SiteAccess (Installed product SiteAccess (SiteAccess-2-0-0))
Squishdot (Installed product Squishdot (Squishdot-1-2-1))
StandardCacheManagers (Installed product StandardCacheManagers 
(StandardCacheManagers-1-1-0))
ThreadSafeCounter (Installed product ThreadSafeCounter (0.0.3))
TinyTablePlus (Installed product TinyTablePlus (TinyTablePlus-0-9))
UpdateSupport (Installed product UpdateSupport (UpdateSupport-1_0_0))
WarpFramework (Installed product WarpFramework (2001-05-02-1222))
ZCatalog (Installed product ZCatalog (ZCatalog-2-2-0))
ZGDChart (Installed product ZGDChart (ZGDChart 0.5.1))
ZGadflyDA (Installed product ZGadflyDA)
ZPatterns (Installed product ZPatterns (ZPatterns-0-3-0))
ZPoPyDA (Installed product ZPoPyDA)
ZPsycopgDA (Installed product ZPsycopgDA)
ZSQLMethods (Installed product ZSQLMethods)
ZSyncer (Installed product ZSyncer)
ZWiki (Installed product ZWiki (ZWiki-0-9-6))
ZopeTutorial (Installed product ZopeTutorial (Zope Tutorial 1.0))
iTrack (Installed product iTrack (0.2.0))

-- 
Systems Analyst/Programmer
VistaShare LLC
chris.kratz-no-spam-please-vistashare.com
(replace -no-spam-please- by @)
www.vistashare.com