I've recently had an issue where my site becomes intermittently unavailable
after years of working fine and I'm uncertain on how to troubleshoot where
it's coming from.
My Zope site is accessed through Apache via proxy. I randomly started
receiving Apaches Unavailable page as it is unable to pass off the request.
After looking through both Apaches logs and Zopes logs I see that Zope is
throwing a ton of ZODB warnings and criticals relating to pool_size and
open connections.
2018-08-09T06:17:46 WARNING ZODB.DB DB.open() has 9 open connections with a
pool_size of 7
------
2018-08-09T06:22:01 CRITICAL ZODB.DB DB.open() has 293 open connections
with a pool_size of 7
------
2018-08-09T06:23:00 CRITICAL ZODB.DB DB.open() has 294 open connections
with a pool_size of 7
------
2018-08-09T06:23:32 WARNING ZODB.DB DB.open() has 8 open connections with a
pool_size of 7
------
2018-08-09T06:23:57 CRITICAL ZODB.DB DB.open() has 295 open connections
with a pool_size of 7
------
2018-08-09T06:29:41 WARNING ZODB.DB DB.open() has 8 open connections with a
pool_size of 7
------
2018-08-09T06:29:41 WARNING ZODB.DB DB.open() has 9 open connections with a
pool_size of 7
------
2018-08-09T06:34:10 WARNING ZODB.DB DB.open() has 8 open connections with a
pool_size of 7
------
2018-08-09T06:37:16 WARNING ZODB.DB DB.open() has 8 open connections with a
pool_size of 7
------
2018-08-09T06:37:16 WARNING ZODB.DB DB.open() has 9 open connections with a
pool_size of 7
------
2018-08-09T06:41:04 CRITICAL ZODB.DB DB.open() has 296 open connections
with a pool_size of 7
------
2018-08-09T06:44:27 WARNING ZODB.DB DB.open() has 8 open connections with a
pool_size of 7
------
2018-08-09T06:44:41 CRITICAL ZODB.DB DB.open() has 297 open connections
with a pool_size of 7
------
2018-08-09T06:47:13 WARNING ZODB.DB DB.open() has 8 open connections with a
pool_size of 7
------
2018-08-09T06:47:13 WARNING ZODB.DB DB.open() has 9 open connections with a
pool_size of 7
------
2018-08-09T06:47:14 WARNING ZODB.DB DB.open() has 10 open connections with
a pool_size of 7
During this, Apache is throwing the following errors:
[Thu Aug 09 08:39:55 2018] [error] (OS 10048)Only one usage of each socket
address (protocol/network address/port) is normally permitted. : proxy:
HTTP: attempt to connect to 127.0.0.1:1337 (*) failed
[Thu Aug 09 08:41:00 2018] [error] (OS 10048)Only one usage of each socket
address (protocol/network address/port) is normally permitted. : proxy:
HTTP: attempt to connect to 127.0.0.1:1337 (*) failed
[Thu Aug 09 08:41:38 2018] [error] (OS 10061)No connection could be made
because the target machine actively refused it. : proxy: HTTP: attempt to
connect to 127.0.0.1:1337 (*) failed
[Thu Aug 09 08:41:39 2018] [error] (OS 10061)No connection could be made
because the target machine actively refused it. : proxy: HTTP: attempt to
connect to 127.0.0.1:1337 (*) failed
I'm thinking that because of the large amount of open connections Zope is
no longer serving requests and Apache is throwing the unavailable upon
proxy.
My problem is that I'm not quite sure how to figure out what's using the
amount of connections and how to get them to release. Any help would be
appreciated.
Thanks!
