[Zope] Random Crashes/Freezes on FreeBSD 5.4-RELEASE

[alex at schnarff.com]

Hello All,

I've recently inherited the job of co-administering an existing Zope 
2.9 system that is performing very flakily. While I admit to knowing 
nothing about Zope administration before the last couple of weeks, I've 
been busy reading the mailing list archives, etc., and I'm convinced 
that something obscure is wrong in this particular installation. I'm 
hoping some kind/brave soul here can help me figure it out.

Here's the setup, described in as much detail as I can (I'll be happy 
to provide additional tidbits as requested):

* FreeBSD 5.4-RELEASE system w/generic kernel. Yes, I know this is old, 
I'm working on getting it upgraded.
* Loads of good hardware: 3.2GHz P-4 processor w/hyperthreading 
enabled; 2GB memory; 240GB Seagate SCSI-3 disk.
* Python 2.4.3
* Zope 2.9
* Apache 2.0.58, compiled with:
  -D APACHE_MPM_DIR="server/mpm/prefork"
  -D APR_HAS_SENDFILE
  -D APR_HAS_MMAP
  -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
  -D APR_USE_FLOCK_SERIALIZE
  -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
  -D APR_HAS_OTHER_CHILD
  -D AP_HAVE_RELIABLE_PIPED_LOGS
  -D HTTPD_ROOT="/usr/local"
  -D SUEXEC_BIN="/usr/local/bin/suexec"
  -D DEFAULT_PIDLOG="/var/run/httpd.pid"
  -D DEFAULT_SCOREBOARD="/var/run/apache_runtime_status"
  -D DEFAULT_LOCKFILE="/var/run/accept.lock"
  -D DEFAULT_ERRORLOG="/var/log/httpd-error.log"
  -D AP_TYPES_CONFIG_FILE="etc/apache2/mime.types"
  -D SERVER_CONFIG_FILE="etc/apache2/httpd.conf"
  Compiled-In Modules:
   core.c
   prefork.c
   http_core.c
   mod_so.c
  Dynamically Loaded Modules (don't even get me started on the size of 
this list):
LoadModule access_module libexec/apache2/mod_access.so
LoadModule auth_module libexec/apache2/mod_auth.so
LoadModule auth_anon_module libexec/apache2/mod_auth_anon.so
LoadModule auth_dbm_module libexec/apache2/mod_auth_dbm.so
LoadModule charset_lite_module libexec/apache2/mod_charset_lite.so
LoadModule include_module libexec/apache2/mod_include.so
LoadModule deflate_module libexec/apache2/mod_deflate.so
LoadModule log_config_module libexec/apache2/mod_log_config.so
LoadModule logio_module libexec/apache2/mod_logio.so
LoadModule env_module libexec/apache2/mod_env.so
LoadModule mime_magic_module libexec/apache2/mod_mime_magic.so
LoadModule cern_meta_module libexec/apache2/mod_cern_meta.so
LoadModule expires_module libexec/apache2/mod_expires.so
LoadModule headers_module libexec/apache2/mod_headers.so
LoadModule usertrack_module libexec/apache2/mod_usertrack.so
LoadModule unique_id_module libexec/apache2/mod_unique_id.so
LoadModule setenvif_module libexec/apache2/mod_setenvif.so
LoadModule proxy_module libexec/apache2/mod_proxy.so
LoadModule proxy_http_module libexec/apache2/mod_proxy_http.so
<IfDefine SSL>
LoadModule ssl_module libexec/apache2/mod_ssl.so
</IfDefine>
LoadModule mime_module libexec/apache2/mod_mime.so
LoadModule dav_module libexec/apache2/mod_dav.so
LoadModule status_module libexec/apache2/mod_status.so
LoadModule autoindex_module libexec/apache2/mod_autoindex.so
LoadModule asis_module libexec/apache2/mod_asis.so
LoadModule info_module libexec/apache2/mod_info.so
LoadModule cgi_module libexec/apache2/mod_cgi.so
LoadModule dav_fs_module libexec/apache2/mod_dav_fs.so
LoadModule vhost_alias_module libexec/apache2/mod_vhost_alias.so
LoadModule negotiation_module libexec/apache2/mod_negotiation.so
LoadModule dir_module libexec/apache2/mod_dir.so
LoadModule imap_module libexec/apache2/mod_imap.so
LoadModule actions_module libexec/apache2/mod_actions.so
LoadModule speling_module libexec/apache2/mod_speling.so
LoadModule userdir_module libexec/apache2/mod_userdir.so
LoadModule alias_module libexec/apache2/mod_alias.so
LoadModule rewrite_module libexec/apache2/mod_rewrite.so

I will happily send along the full httpd.conf or zope.conf if requested.

That all in mind, the behavior is difficult to accurately describe. On 
what appears to be a semi-random basis, things being served up by Zope 
are inaccessible from the web; clients get timeout messages from their 
web browser. While this is going on, things being served up directly by 
Apache are fine. Just to make things more confusing, when Zope 
flatlined this morning, I got the following message:

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /portal.
Reason: Error reading from remote server

...which is the first I've seen of any actual error messages when this 
behavior is occuring.

For some reason that I'm unaware of, Zope will typically be restarted 
when this flatlining occurs; I think it's a process monitor tool one of 
my co-admins have installed, but I'm not 100% certain of that. Once 
it's restarted, things work fine.

Does anyone have a clue what could be going on? Again, I'll be more 
than happy to supply additional info, run tests, whatever may be 
necessary to help figure out what the deal may be.

Thanks,
Alex Kirk