I'm hunting down an arcane problem in one of our Zope sites and find myself hitting new and wildly diverse problems every day. I hope somebody can point me in the direction of some solution strategies. Main suspect is a corrupted Data.fs.in file. How can I audit and repair it? problem A: access denied vs. KeyError ============================ For starters, we built a website with a lot of custom programming. I created the NFGnav product http://www.zope.org/Members/gyst/NFGnav and my partner built a collection of ZClasses. All this was Tested And Working Fine (TM). Since some weeks, part of the site gives an 'access denied' error. We had to shut down that subtree. Strange thing is, it also gives access denied for the manager/owner logins of the site - that's me and I really have access to everything. What is even stranger, is that when I logon as superuser, I get a KeyError instead of an access denied error. => Question: which parts of zope would be prone to denying access to a privileged user, while generating KeyErrors for the superuser? problem B: hanging zope thread zombies all zope sites ======================================== While debugging this problem, I found a strange correlation with another problem we're having: zope threads consuming 99.9% CPU. Now and again, a zope thread hangs. All zope sites are dead. I have to not only restart zope; I have to stop zope, restart apache, and start zope again to get rid of the hanging thread and get the zope sites alive again. As soon as I start digging around in the buggy site, the number of thread crashes jumps up. In combination with the problems we're having with that site itself, this suggests two possible causes: 1) ZODB curruption. We've had a complete server crash 3 weeks ago and it is quite possible the ZODB isn't 100% healthy. In my zope installation I found a utility 'bbb.py' to audit data storage files. Alas, it is of no use: | zeno:/usr/lib/zope/utilities# python bbb.py /var/lib/zope/import/www.intratuin.nl.zexp | | The input file is not a ZODB 2 database file. | This script only works with ZODB 2 (BoboPOS) data or export files. => Question: is there another utility for checking and repairing object stores? 2) Another possible cause would be an infinite loop in one of our custom scripts. To try and debug this I wanted to import a copy of the offending site into one of our testlab machines. However, on importing the zexp file I got: <STRONG>Error Type: TypeError</STRONG><BR> <STRONG>Error Value: ('len() of unsized object', <class Shared.DC.ZRDB.DA.SQL at 845bd28>, None)</STRONG><BR> <!-- Traceback (innermost last): File /usr/lib/zope/lib/python/ZPublisher/Publish.py, line 214, in publish_module File /usr/lib/zope/lib/python/ZPublisher/Publish.py, line 179, in publish File /usr/lib/zope/lib/python/Zope/__init__.py, line 202, in zpublisher_exception_hook (Object: ApplicationDefaultPermissions) File /usr/lib/zope/lib/python/ZPublisher/Publish.py, line 165, in publish File /usr/lib/zope/lib/python/ZPublisher/mapply.py, line 160, in mapply (Object: manage_importObject) File /usr/lib/zope/lib/python/ZPublisher/Publish.py, line 102, in call_object (Object: manage_importObject) File /usr/lib/zope/lib/python/OFS/ObjectManager.py, line 478, in manage_importObject (Object: ApplicationDefaultPermissions) File /usr/lib/zope/lib/python/ZODB/ExportImport.py, line 284, in importFile TypeError: (see above) I get the same error if I export another part of our production zope installation and try to import it to another test machine. Which points back into the direction of a file corruption again, doesn't it? => So, what can I do about it? :*CU# -- *** Guido A.J. Stevens *** mailto:gyst@nfg.nl *** *** Net Facilities Group *** tel:+31.43.3618933 *** *** Postbus 1143 *** fax:+31.43.3560502 *** *** 6201 BC Maastricht *** http://www.nfg.nl *** Liefde en emoties, haat en afkeer zitten erin verstopt. Zoals mensen cyborgs zijn, is het technische ding altijd een hybride. Dat maakt zowel mensen als dingen zo fascinerend. [Achterhuis, ISBN 90-263-1524-4, p.391]