Running out of sockets
Coming out of lurk mode here with a really tricky one. We had just moved our server from: Zope Version (unreleased version, python 2.3.3, linux2) Python Version 2.3.3 (#2, Feb 21 2004, 21:17:53) [GCC 3.2.2 20030222 (Red Hat Linux 3.2.2-5)] To: Zope Version (Zope 2.7.5-1.fc3, python 2.3.4, linux2) Python Version 2.3.4 (#1, Feb 2 2005, 11:44:49) [GCC 3.4.3 20041212 (Red Hat 3.4.3-9.EL4)] The first thing we noticed was: Proxy Error The proxy server received an invalid response from an upstream server. The proxy server could not handle the request GET /xp/pub/topp/invest/. Reason: Error reading from remote server Apache/2.0.52 (Red Hat) Server at www.foo.bar Port 80 ( We are running a homegrown virtual host product very similar to the "virtual host monster" behind apache via rewrites. No ZEO.) This seems to corespond to a lot of: PROBLEM(100) ZServer warning: server accept() threw an exception in the log. Restarting zope has fixed it for now, but "lsof -a -U -uzope" and "lsof -uzope | wc -l" Shows the number of open sockets steadily growing, and apparently none closing.. We have not seen the Proxy Error again yet, but it seems pretty certain sockets will run out at some point. What to do? Where to dig? I have found quite a number of references to similar problems by googling parts of this, but nothing leading me closer to figuring it out. "ulimit -a" shows the limit is 1024 ( at 1004 now everything seems fine except the occasional PROBLEM(100) XPTopic seemingly unrelated, not accept() ) "netstat -axp" gives me loads of lines like this: unix 2 [ ] STREAM CONNECTED 2792969 572/python We have tried various combinations for config options, but without any luck: "force-connection-close on" did not seem to make any difference "max-listen-sockets 500" did not keep "lsof -a -U -uzope | wc -l" from exceeding 500. maximum-number-of-session-objects 10000 session-timeout-minutes 90 This should not be related no? I realize this is becoming a very rambling question, I just thought I would include as much info as possible right away :) Regards Gaute Amundsen
Gaute Amundsen wrote at 2005-5-12 00:17 +0200:
... Restarting zope has fixed it for now, but ... "lsof -a -U -uzope" and "lsof -uzope | wc -l" Shows the number of open sockets steadily growing, and apparently none closing..
You might be able to determine the second partner for the socket (with "netstat", I think). This may give you a hint for what purpose the sockets are used. -- Dieter
On Thursday 12 May 2005 20:45, Dieter Maurer wrote:
Gaute Amundsen wrote at 2005-5-12 00:17 +0200:
... Restarting zope has fixed it for now, but ... "lsof -a -U -uzope" and "lsof -uzope | wc -l" Shows the number of open sockets steadily growing, and apparently none closing..
You might be able to determine the second partner for the socket (with "netstat", I think). This may give you a hint for what purpose the sockets are used.
It turned out the problem was MySQL-python-1.0.0-1 vs. Mysql 4.1 Upgrading MySQL-python to 1.2 solved the problem. Thanks Gaute Amundsen
participants (2)
-
Dieter Maurer -
Gaute Amundsen