[Zope] server instability: ZODB corruption?

[Guido A.J. Stevens]

I'm hunting down an arcane problem in one of our Zope sites and find
myself hitting new and wildly diverse problems every day. I hope
somebody can point me in the direction of some solution strategies.

Main suspect is a corrupted Data.fs.in file. How can I audit and
repair it?

problem A: access denied vs. KeyError
============================

For starters, we built a website with a lot of custom programming. I
created the NFGnav product
http://www.zope.org/Members/gyst/NFGnav
and my partner built a collection of ZClasses. 

All this was Tested And Working Fine (TM).

Since some weeks, part of the site gives an 'access denied' error. We
had to shut down that subtree. Strange thing is, it also gives access
denied for the manager/owner logins of the site - that's me and I
really have access to everything. What is even stranger, is that when
I logon as superuser, I get a KeyError instead of an access denied
error.

=> Question: which parts of zope would be prone to denying access to a 
privileged user, while generating KeyErrors for the superuser?


problem B: hanging zope thread zombies all zope sites
========================================

While debugging this problem, I found a strange correlation with
another problem we're having: zope threads consuming 99.9% CPU. Now
and again, a zope thread hangs. All zope sites are dead. I have to not
only restart zope; I have to stop zope, restart apache, and start zope
again to get rid of the hanging thread and get the zope sites alive
again.

As soon as I start digging around in the buggy site, the number of
thread crashes jumps up. In combination with the problems we're having 
with that site itself, this suggests two possible causes:

1) ZODB curruption. We've had a complete server crash 3 weeks ago and
it is quite possible the ZODB isn't 100% healthy. In my zope
installation I found a utility 'bbb.py' to audit data storage
files. Alas, it is of no use:
 | zeno:/usr/lib/zope/utilities# python bbb.py /var/lib/zope/import/www.intratuin.nl.zexp
 | 
 | The input file is not a ZODB 2 database file.
 | This script only works with ZODB 2 (BoboPOS) data or export files.

=> Question: is there another utility for checking and repairing
object stores?

2) Another possible cause would be an infinite loop in one of our
custom scripts. To try and debug this I wanted to import a copy of the 
offending site into one of our testlab machines. However, on importing 
the zexp file I got:

  <STRONG>Error Type: TypeError</STRONG><BR>
  <STRONG>Error Value: ('len() of unsized object', <class Shared.DC.ZRDB.DA.SQL at 845bd28>, None)</STRONG><BR> 
 
<!--
Traceback (innermost last):
  File /usr/lib/zope/lib/python/ZPublisher/Publish.py, line 214, in publish_module
  File /usr/lib/zope/lib/python/ZPublisher/Publish.py, line 179, in publish
  File /usr/lib/zope/lib/python/Zope/__init__.py, line 202, in zpublisher_exception_hook
    (Object: ApplicationDefaultPermissions)
  File /usr/lib/zope/lib/python/ZPublisher/Publish.py, line 165, in publish
  File /usr/lib/zope/lib/python/ZPublisher/mapply.py, line 160, in mapply
    (Object: manage_importObject)
  File /usr/lib/zope/lib/python/ZPublisher/Publish.py, line 102, in call_object
    (Object: manage_importObject)
  File /usr/lib/zope/lib/python/OFS/ObjectManager.py, line 478, in manage_importObject
    (Object: ApplicationDefaultPermissions)
  File /usr/lib/zope/lib/python/ZODB/ExportImport.py, line 284, in importFile
TypeError: (see above)

I get the same error if I export another part of our production zope
installation and try to import it to another test machine.

Which points back into the direction of a file corruption again,
doesn't it? 

=> So, what can I do about it?

:*CU#
-- 
***    Guido A.J. Stevens      ***    mailto:gyst@nfg.nl    ***
***    Net Facilities Group    ***    tel:+31.43.3618933    ***
***    Postbus 1143            ***    fax:+31.43.3560502    ***
***    6201 BC  Maastricht     ***    http://www.nfg.nl     ***

Liefde en emoties, haat en afkeer zitten erin verstopt. Zoals
mensen cyborgs zijn, is het technische ding altijd een hybride.
Dat maakt zowel mensen als dingen zo fascinerend.
[Achterhuis, ISBN 90-263-1524-4, p.391]