[Zope] FYI: Quick review of the downtime

[Paul Everitt]

Just as an attempt to share a lesson learned in the downtime, here's a
quick report.  Since this is from me, instead of somebody smart, don't
look for too much wisdom. :^)

Earlier this week Ken and Michel worked hard on getting the site
migrated to a new machine.  Though there were a lot of painful things
associated with getting the mail archives moved over, mail delivery,
etc., everything went OK.

Once that settled down, we wanted to get the site updated with some
fixes from CVS and to add in some stuff like ZTopics that had been
tested.  We did a cvs update and recompiled some of the Python C
extensions that had changed.

Here's where we hit (but didn't notice) a snag.  The Python development
environment hadn't been installed on the new machine and we didn't
notice that our extensions didn't get updated.

When the site came back up and it came time to unpickle some things, the
loading failed because the dynamically loaded extensions failed.  Of
course it *looked* like the database had bad state, as the exception got
handled without passing through the import error.

We opened up the site in the Python interpreter and tried to grab some
objects.  That's when we noticed, for instance, the XMLDocuments were
failing to load because the C extension couldn't be imported, because
there was a Py_something_or_other symbol error.

So Ken installed the include files, did the compile, and things came
back.

There was one other gotcha but I don't have the scoop on it.  Still,
hope you find this helpful!

--Paul

Paul Everitt       Digital Creations
paul@digicool.com  540.371.6909
-----------------------------------------
The Open Source Zope application server
http://www.zope.org/