Random Crashes/Freezes on FreeBSD 5.4-RELEASE
Hello All, I've recently inherited the job of co-administering an existing Zope 2.9 system that is performing very flakily. While I admit to knowing nothing about Zope administration before the last couple of weeks, I've been busy reading the mailing list archives, etc., and I'm convinced that something obscure is wrong in this particular installation. I'm hoping some kind/brave soul here can help me figure it out. Here's the setup, described in as much detail as I can (I'll be happy to provide additional tidbits as requested): * FreeBSD 5.4-RELEASE system w/generic kernel. Yes, I know this is old, I'm working on getting it upgraded. * Loads of good hardware: 3.2GHz P-4 processor w/hyperthreading enabled; 2GB memory; 240GB Seagate SCSI-3 disk. * Python 2.4.3 * Zope 2.9 * Apache 2.0.58, compiled with: -D APACHE_MPM_DIR="server/mpm/prefork" -D APR_HAS_SENDFILE -D APR_HAS_MMAP -D APR_HAVE_IPV6 (IPv4-mapped addresses enabled) -D APR_USE_FLOCK_SERIALIZE -D SINGLE_LISTEN_UNSERIALIZED_ACCEPT -D APR_HAS_OTHER_CHILD -D AP_HAVE_RELIABLE_PIPED_LOGS -D HTTPD_ROOT="/usr/local" -D SUEXEC_BIN="/usr/local/bin/suexec" -D DEFAULT_PIDLOG="/var/run/httpd.pid" -D DEFAULT_SCOREBOARD="/var/run/apache_runtime_status" -D DEFAULT_LOCKFILE="/var/run/accept.lock" -D DEFAULT_ERRORLOG="/var/log/httpd-error.log" -D AP_TYPES_CONFIG_FILE="etc/apache2/mime.types" -D SERVER_CONFIG_FILE="etc/apache2/httpd.conf" Compiled-In Modules: core.c prefork.c http_core.c mod_so.c Dynamically Loaded Modules (don't even get me started on the size of this list): LoadModule access_module libexec/apache2/mod_access.so LoadModule auth_module libexec/apache2/mod_auth.so LoadModule auth_anon_module libexec/apache2/mod_auth_anon.so LoadModule auth_dbm_module libexec/apache2/mod_auth_dbm.so LoadModule charset_lite_module libexec/apache2/mod_charset_lite.so LoadModule include_module libexec/apache2/mod_include.so LoadModule deflate_module libexec/apache2/mod_deflate.so LoadModule log_config_module libexec/apache2/mod_log_config.so LoadModule logio_module libexec/apache2/mod_logio.so LoadModule env_module libexec/apache2/mod_env.so LoadModule mime_magic_module libexec/apache2/mod_mime_magic.so LoadModule cern_meta_module libexec/apache2/mod_cern_meta.so LoadModule expires_module libexec/apache2/mod_expires.so LoadModule headers_module libexec/apache2/mod_headers.so LoadModule usertrack_module libexec/apache2/mod_usertrack.so LoadModule unique_id_module libexec/apache2/mod_unique_id.so LoadModule setenvif_module libexec/apache2/mod_setenvif.so LoadModule proxy_module libexec/apache2/mod_proxy.so LoadModule proxy_http_module libexec/apache2/mod_proxy_http.so <IfDefine SSL> LoadModule ssl_module libexec/apache2/mod_ssl.so </IfDefine> LoadModule mime_module libexec/apache2/mod_mime.so LoadModule dav_module libexec/apache2/mod_dav.so LoadModule status_module libexec/apache2/mod_status.so LoadModule autoindex_module libexec/apache2/mod_autoindex.so LoadModule asis_module libexec/apache2/mod_asis.so LoadModule info_module libexec/apache2/mod_info.so LoadModule cgi_module libexec/apache2/mod_cgi.so LoadModule dav_fs_module libexec/apache2/mod_dav_fs.so LoadModule vhost_alias_module libexec/apache2/mod_vhost_alias.so LoadModule negotiation_module libexec/apache2/mod_negotiation.so LoadModule dir_module libexec/apache2/mod_dir.so LoadModule imap_module libexec/apache2/mod_imap.so LoadModule actions_module libexec/apache2/mod_actions.so LoadModule speling_module libexec/apache2/mod_speling.so LoadModule userdir_module libexec/apache2/mod_userdir.so LoadModule alias_module libexec/apache2/mod_alias.so LoadModule rewrite_module libexec/apache2/mod_rewrite.so I will happily send along the full httpd.conf or zope.conf if requested. That all in mind, the behavior is difficult to accurately describe. On what appears to be a semi-random basis, things being served up by Zope are inaccessible from the web; clients get timeout messages from their web browser. While this is going on, things being served up directly by Apache are fine. Just to make things more confusing, when Zope flatlined this morning, I got the following message: Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /portal. Reason: Error reading from remote server ...which is the first I've seen of any actual error messages when this behavior is occuring. For some reason that I'm unaware of, Zope will typically be restarted when this flatlining occurs; I think it's a process monitor tool one of my co-admins have installed, but I'm not 100% certain of that. Once it's restarted, things work fine. Does anyone have a clue what could be going on? Again, I'll be more than happy to supply additional info, run tests, whatever may be necessary to help figure out what the deal may be. Thanks, Alex Kirk
--On 20. Februar 2007 11:10:06 -0500 alex@schnarff.com wrote:
* Zope 2.9
Which Zope 2.9 version?
* Apache 2.0.58, compiled with:
The apache configuration isn't really of interest. You might check your Zope log files for additional informations. Enable coredumps (ulimit) and check if there are any core files floating around after a crash. There were also some reports with Python crashes on Freebsd because a too small thread stack size (I think). You might find something when you google for "freebsd python zope stack size". -aj -- ZOPYX Ltd. & Co. KG - Charlottenstr. 37/1 - 72070 Tübingen - Germany Web: www.zopyx.com - Email: info@zopyx.com - Phone +49 - 7071 - 793376 Registergericht: Amtsgericht Stuttgart, Handelsregister A 381535 Geschäftsführer/Gesellschafter: ZOPYX Limited, Birmingham, UK ------------------------------------------------------------------------ E-Publishing, Python, Zope & Plone development, Consulting
* Zope 2.9
Which Zope 2.9 version?
From the Control Panel: Plone version overview Plone 2.5.1, CMF-1.6.2, Zope (Zope 2.9.5-final, python 2.4.3, freebsd5), Five 1.3.7, Python 2.4.3 (#2, Jul 2 2006, 16:14:54) [GCC 3.4.2 [FreeBSD] 20040728], PIL 1.1.5
* Apache 2.0.58, compiled with:
The apache configuration isn't really of interest.
Sorry for the crud, then, just trying to find anything that might cause problems.
You might check your Zope log files for additional informations. Enable coredumps (ulimit) and check if there are any core files floating around after a crash.
Already had: alex@tms: /usr/local/www/$ limits Resource limits (current): cputime infinity secs filesize infinity kb datasize 524288 kb stacksize 65536 kb coredumpsize infinity kb memoryuse infinity kb memorylocked infinity kb maxprocesses 5547 openfiles 11095 sbsize infinity bytes vmemoryuse infinity kb ...and unless they don't contain ".core" in their name, there are no core dumps from Zope on the system.
There were also some reports with Python crashes on Freebsd because a too small thread stack size (I think). You might find something when you google for "freebsd python zope stack size".
I've already seen this issue, and it's apparently not the cause here. According to the guy who installed it, Python was compiled with the "HUGE_STACK_SIZE" option, which was the root of those problems. Good thought, though. :-) Alex Kirk
Hi Alex, alex@schnarff.com schrieb:
Hello All,
I've recently inherited the job of co-administering an existing Zope 2.9 system that is performing very flakily. While I admit to knowing nothing about Zope administration before the last couple of weeks, I've been busy reading the mailing list archives, etc., and I'm convinced that something obscure is wrong in this particular installation. I'm hoping some kind/brave soul here can help me figure it out.
Here's the setup, described in as much detail as I can (I'll be happy to provide additional tidbits as requested):
* FreeBSD 5.4-RELEASE system w/generic kernel. Yes, I know this is old, * Python 2.4.3 * Zope 2.9
the main error cause of Zope on FreeBSD is a wrong configuration of the python 2.4 port. You need to enable the "HUGE_STACK_SIZE" option, e.g. with "make config" in /usr/ports/lang/python24 and reinstall python. Ciao, Jochen -- ---------------------------- Jochen Knuth WebMaster http://www.ipro.de IPRO GmbH Steinbeisstr. 6 71229 Leonberg Tel. +49-7152-93330 Fax +49-7152-933330 E-Mail: ipro@ipro.de Geschäftsführer: Martin Himmelsbach, Thomas Barth Amtsgericht Leonberg HRB 251557 UST-ID-Nummer: DE 146022368
* FreeBSD 5.4-RELEASE system w/generic kernel. Yes, I know this is old, * Python 2.4.3 * Zope 2.9
the main error cause of Zope on FreeBSD is a wrong configuration of the python 2.4 port. You need to enable the "HUGE_STACK_SIZE" option, e.g. with "make config" in /usr/ports/lang/python24 and reinstall python.
In case you didn't see this in my e-mail to Andreas, this was enabled at compile-time by the guy who installed Python on this system...or so he says. Is there any way to check that it's been properly compiled in? Meanwhile, I just got an e-mail from my co-admin on this system, and he says that this morning's crash was accompanied by the following "in the event log" (not sure which one he's referring to): MemoryError: Can't allocate memory for compression object Does that help make sense of things? Thanks, Alex
participants (3)
-
alex@schnarff.com -
Andreas Jung -
Jochen Knuth