Hello, We have been having problem with our production servers. We currently have: zope 2.8.5 + zeo python 2.3.4 (red hat distribution) Red Hat RHEL 4 Plone 2.1.2 Our zeo clients hang intermittently. We have no way of reproducing the problem, but it occurs daily. The client hangs and a restart seems to fix the problem. In the event log with tracing on we get Trace zeo.zrpc.Connection(C) wait(16697) {server:8100} pending, async=0 There are hundreds to thousands of these until the server is restarted. In the zeo log we get Error caught in asyncore asyncore.py error:(110,'Connection timed out') We have been trying to track this down and have had no luck. Does anyone have any suggestions? Thanks, Paul
zope 2.8.5 + zeo python 2.3.4 (red hat distribution) Red Hat RHEL 4 Plone 2.1.2
Our zeo clients hang intermittently. We have no way of reproducing the problem, but it occurs daily. The client hangs and a restart seems to fix the problem.
Install DeadlockDebugger and you'll be able to see what is the state of your threads when Zope becomes unresponsive.
In the event log with tracing on we get
Trace zeo.zrpc.Connection(C) wait(16697) {server:8100} pending, async=0
There are hundreds to thousands of these until the server is restarted.
In the zeo log we get
Error caught in asyncore asyncore.py
error:(110,'Connection timed out') I'm not sure what these errors are. Maybe firewall between zeo server and zeo client closes the connections or something like that?
-- Maciej Wisniowski
On Wed, Oct 18, 2006 at 07:32:38AM +0200, Maciej Wisniowski wrote:
zope 2.8.5 + zeo python 2.3.4 (red hat distribution) Red Hat RHEL 4 Plone 2.1.2
Our zeo clients hang intermittently. We have no way of reproducing the problem, but it occurs daily. The client hangs and a restart seems to fix the problem.
Install DeadlockDebugger and you'll be able to see what is the state of your threads when Zope becomes unresponsive.
In the event log with tracing on we get
Trace zeo.zrpc.Connection(C) wait(16697) {server:8100} pending, async=0
There are hundreds to thousands of these until the server is restarted.
In the zeo log we get
Error caught in asyncore asyncore.py
error:(110,'Connection timed out') I'm not sure what these errors are. Maybe firewall between zeo server and zeo client closes the connections or something like that?
That was my guess too. See this thread: http://aspn.activestate.com/ASPN/Mail/Message/zope-list/2916870 -- Paul Winkler http://www.slinkp.com
I see references to asyncore.py near the end of the thread listing when using DeadLockDebugger and seeing a stuck thread process. The tread eventually clears out but sometimes an impatient user reloads, reloads, reloads...and then all 4 threads are stuck on exact same thing. Then you can see the same URL jump to the other Zope/ZEO client and same thing happens there. Another few reloads and one user has effectively crashed the site....until the threads clear themselves out. Usually is a catalog search for a news article or something spanning our 6 years of online news articles searching on the word 'bob' or something like that. +1 for DeadlockDebugger... love it. Allen Paul Winkler wrote:
On Wed, Oct 18, 2006 at 07:32:38AM +0200, Maciej Wisniowski wrote:
zope 2.8.5 + zeo python 2.3.4 (red hat distribution) Red Hat RHEL 4 Plone 2.1.2
Our zeo clients hang intermittently. We have no way of reproducing the problem, but it occurs daily. The client hangs and a restart seems to fix the problem.
Install DeadlockDebugger and you'll be able to see what is the state of your threads when Zope becomes unresponsive.
In the event log with tracing on we get
Trace zeo.zrpc.Connection(C) wait(16697) {server:8100} pending, async=0
There are hundreds to thousands of these until the server is restarted.
In the zeo log we get
Error caught in asyncore asyncore.py
error:(110,'Connection timed out')
I'm not sure what these errors are. Maybe firewall between zeo server and zeo client closes the connections or something like that?
That was my guess too. See this thread: http://aspn.activestate.com/ASPN/Mail/Message/zope-list/2916870
participants (4)
-
Allen Schmidt Sr. -
Maciej Wisniowski -
Paul Williams -
Paul Winkler