Perhaps I'm wrong but I suspect that a Zope I'm running is dying because of running out of threads. They probably die when being designated to open an external relational database connection or something dodgy. What are my options for monitoring and debugging what zope's threads are doing? Is there a simple tool to just get a simple readout of how many threads are busy? -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
On Jan 22, 2008, at 16:23 , Peter Bengtsson wrote:
Perhaps I'm wrong but I suspect that a Zope I'm running is dying because of running out of threads. They probably die when being designated to open an external relational database connection or something dodgy.
What are my options for monitoring and debugging what zope's threads are doing? Is there a simple tool to just get a simple readout of how many threads are busy?
One simple option is the ZMI Control_Panel Debug screen. Go to / Control_Panel/DebugInfo/manage_main on your instance and scroll to the bottom. It shows current connections to the Zope server from the outside, and how long the connection has been around. You will always at least see one entry, which is the request to fetch that debug screen. If you have connections beyond a few seconds of age they may be stuck. jens
Peter Bengtsson wrote:
Perhaps I'm wrong but I suspect that a Zope I'm running is dying because of running out of threads. They probably die when being designated to open an external relational database connection or something dodgy.
What are my options for monitoring and debugging what zope's threads are doing? Is there a simple tool to just get a simple readout of how many threads are busy?
May be you can try by using just 1 thread and then debug it to see how it dies. Suresh
Peter Bengtsson wrote at 2008-1-22 15:23 +0000:
Perhaps I'm wrong but I suspect that a Zope I'm running is dying because of running out of threads.
This is very unlikely. It may no longer respond when all its threads are working -- but it will not die for this reason. Zope uses a worker thread pool (of size 4, by default), allocated quite early. It does not create further threads. It is therefore very unlikely that it hits a thread limit. -- Dieter
On 22/01/2008, Dieter Maurer <dieter@handshake.de> wrote:
Peter Bengtsson wrote at 2008-1-22 15:23 +0000:
Perhaps I'm wrong but I suspect that a Zope I'm running is dying because of running out of threads.
This is very unlikely. It may no longer respond when all its threads are working -- but it will not die for this reason.
By dying I don't mean crashing and stopping but instead staying up and running but not responding to anything. Like a coma.
Zope uses a worker thread pool (of size 4, by default), allocated quite early. It does not create further threads. It is therefore very unlikely that it hits a thread limit.
I'm just not certain if it's this. It seems highly related to MS SQL connections but I could be wrong. The strange thing is, restarting the MS SQL server might be (we're not certain) a solution to get Zope responding again. My theory was that the threads get too attached to a network service that doesn't time out or raise an error. Sadly I can't reproduce it but the next time it happens I'll look at both DeadlockDebugger and the Control_Panel Debug info if there are any clues there. -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
On Jan 22, 2008, at 23:33 , Peter Bengtsson wrote:
On 22/01/2008, Dieter Maurer <dieter@handshake.de> wrote:
Peter Bengtsson wrote at 2008-1-22 15:23 +0000:
Perhaps I'm wrong but I suspect that a Zope I'm running is dying because of running out of threads.
This is very unlikely. It may no longer respond when all its threads are working -- but it will not die for this reason.
By dying I don't mean crashing and stopping but instead staying up and running but not responding to anything. Like a coma.
This symptom, especially when these "hanging" threads cause no load at all on the system, certainly point to connections to outside systems that do not respond for some reason. I have seen this issue with Zope instances that make HTTP calls to other systems, or even with a hanging LDAP server which accepted connections from Zope but never returned. Any software that makes outgoing network connections from Zope is a potential suspect. GDB can help you determine where the hanging condition originates. Here's some links: http://plone.org/documentation/how-to/use-gdb-to-get-backtraces-of-hung-spin... http://zopelabs.com/cookbook/1073504990 jens
GDB can help you determine where the hanging condition originates. Here's some links:
http://plone.org/documentation/how-to/use-gdb-to-get-backtraces-of-hung-spin...
This is great! I haven't used it yet but it definitely appears to me the right tool for me. -- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
----- Original Message ----- From: "Peter Bengtsson" <peter@fry-it.com> To: "Dieter Maurer" <dieter@handshake.de> Cc: "zope user list" <zope@zope.org> Sent: Tuesday, January 22, 2008 5:33 PM Subject: Re: [Zope] Counting threads
On 22/01/2008, Dieter Maurer <dieter@handshake.de> wrote:
Peter Bengtsson wrote at 2008-1-22 15:23 +0000:
Perhaps I'm wrong but I suspect that a Zope I'm running is dying because of running out of threads.
This is very unlikely. It may no longer respond when all its threads are working -- but it will not die for this reason.
By dying I don't mean crashing and stopping but instead staying up and running but not responding to anything. Like a coma.
Sadly I can't reproduce it but the next time it happens I'll look at both DeadlockDebugger and the Control_Panel Debug info if there are any clues there.
You could also try this: http://www.zopelabs.com/cookbook/1073504990 Jonathan
----- Original Message ----- From: "Peter Bengtsson" <peter@fry-it.com>
On 22/01/2008, Dieter Maurer <dieter@handshake.de> wrote:
Peter Bengtsson wrote at 2008-1-22 15:23 +0000:
Perhaps I'm wrong but I suspect that a Zope I'm running is dying because of running out of threads.
I'm just not certain if it's this. It seems highly related to MS SQL connections but I could be wrong. The strange thing is, restarting the MS SQL server might be (we're not certain) a solution to get Zope responding again.
My theory was that the threads get too attached to a network service that doesn't time out or raise an error.
You have absolutelly true! I was the same problems in past. I was make up to 150 threads in some rare cases, but 50..100 in general if I use MSSQL. It was with Zope 2.5.1, with modern zope, see attachment about pool_size. Now I use 80 threads and 60 pool_size and my Zope+MSSQL is fine. MSSQL hits in day peaks 100% CPU load (about hour, two) , but it is not my business. But my business is functional zope webs which eats a lot of RAM due caching policy - about 5..10 times more than before... So you have been pointed and warned ;-) Best Regards, JL.
Peter Bengtsson wrote at 2008-1-22 22:33 +0000:
... By dying I don't mean crashing and stopping but instead staying up and running but not responding to anything. Like a coma.
Use the "DeadlockDebugger" to find out what your Zope is doing. -- Dieter
participants (6)
-
Dieter Maurer -
Jaroslav Lukesh -
Jens Vagelpohl -
Jonathan -
Peter Bengtsson -
Suresh V.