Following a hard disk crash 2 weeks ago we have installed Zope onto a new server and all was fine until yesterday morning when Zope stopped responding and required a restart to get it working. It did the same at 3-30pm today. We are using Zope 2.9.0 on Centos 5.1 on a quad Zeon server with 4gb. Ram. At the moment the server is in quite heavy use with teachers trying to write reports for a deadline tomorrow. These are sent to a Postgres DB via psycopg. There is nothing in Z2.log or event.log to point me to the problem. I am therefore asking for advice on the sorts of things that can cause Zope to stop responding and whether there is anything we can do to mitigate against such an event. Thanks and regards Garry
Have you tried looking at it with the deadlock debugger? That should give you an idea of what its actually running (and if the deadlock debugger is usually responsive even when zope isn't and if it isn't that's also information). Kevin -----Original Message----- From: zope-bounces@zope.org [mailto:zope-bounces@zope.org] On Behalf Of Garry Saddington Sent: Thursday, July 03, 2008 9:00 AM To: zope list user Subject: [Zope] zope stops responding Following a hard disk crash 2 weeks ago we have installed Zope onto a new server and all was fine until yesterday morning when Zope stopped responding and required a restart to get it working. It did the same at 3-30pm today. We are using Zope 2.9.0 on Centos 5.1 on a quad Zeon server with 4gb. Ram. At the moment the server is in quite heavy use with teachers trying to write reports for a deadline tomorrow. These are sent to a Postgres DB via psycopg. There is nothing in Z2.log or event.log to point me to the problem. I am therefore asking for advice on the sorts of things that can cause Zope to stop responding and whether there is anything we can do to mitigate against such an event. Thanks and regards Garry _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Yes, couldn't agree more with kevin. DeadlockDebugger is crucial. It won't change anything other than that you'll have information about what really needs to change. As an intermediate solution you can start to increase the number of Zope threads from 4 to 6 or 8 and see if that kills the CPU of your beefy server. Too many threads can hurt performance especially when working with badly coded external connectors that selfishly hook itself to the python GIL too much. A lightweight alternative to DeadlockDebugger is the table at the bottom of the Debug Information page in Control_Panel. If any of the dates in the first column in that table shows times longer than a second you have a problem with those requests. Peter 2008/7/3 Garry Saddington <garry@schoolteachers.co.uk>:
Following a hard disk crash 2 weeks ago we have installed Zope onto a new server and all was fine until yesterday morning when Zope stopped responding and required a restart to get it working. It did the same at 3-30pm today. We are using Zope 2.9.0 on Centos 5.1 on a quad Zeon server with 4gb. Ram. At the moment the server is in quite heavy use with teachers trying to write reports for a deadline tomorrow. These are sent to a Postgres DB via psycopg. There is nothing in Z2.log or event.log to point me to the problem. I am therefore asking for advice on the sorts of things that can cause Zope to stop responding and whether there is anything we can do to mitigate against such an event. Thanks and regards Garry _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
My experience is that increasing the number of threads past 4 does absolutely nothing to assist performance. Multi threads will help with the occasional long running request, but if its more than occasional then the thread list will eventually clog up no matter how many threads you have (and the more threads you have, the more they fight over the processor). The server sounds beefy, but since Zope is only going to be using one of those four cores, that might be deceptive. Kevin -----Original Message----- From: zope-bounces+ksours=carsdirect.com@zope.org [mailto:zope-bounces+ksours=carsdirect.com@zope.org] On Behalf Of Peter Bengtsson Sent: Thursday, July 03, 2008 9:19 AM To: Garry Saddington Cc: zope list user Subject: Re: [Zope] zope stops responding Yes, couldn't agree more with kevin. DeadlockDebugger is crucial. It won't change anything other than that you'll have information about what really needs to change. As an intermediate solution you can start to increase the number of Zope threads from 4 to 6 or 8 and see if that kills the CPU of your beefy server. Too many threads can hurt performance especially when working with badly coded external connectors that selfishly hook itself to the python GIL too much. A lightweight alternative to DeadlockDebugger is the table at the bottom of the Debug Information page in Control_Panel. If any of the dates in the first column in that table shows times longer than a second you have a problem with those requests. Peter 2008/7/3 Garry Saddington <garry@schoolteachers.co.uk>:
Following a hard disk crash 2 weeks ago we have installed Zope onto a new server and all was fine until yesterday morning when Zope stopped responding and required a restart to get it working. It did the same at 3-30pm today. We are using Zope 2.9.0 on Centos 5.1 on a quad Zeon server with 4gb. Ram. At the moment the server is in quite heavy use with teachers trying to write reports for a deadline tomorrow. These are sent to a Postgres DB via psycopg. There is nothing in Z2.log or event.log to point me to the problem. I am therefore asking for advice on the sorts of things that can cause Zope to stop responding and whether there is anything we can do to mitigate against such an event. Thanks and regards Garry _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Hi, I does not agree with You, because my servers runs with 20..150 threads, because lower number of threads makes for somebody infinity waiting and Zope under load responding very slow (with external database access). After increasing threads, response under load is more quickly and more paralel users should access to the web. ----- Original Message ----- From: "Sours, Kevin" <KSours@carsdirect.com> My experience is that increasing the number of threads past 4 does absolutely nothing to assist performance. Multi threads will help with the occasional long running request, but if its more than occasional then the thread list will eventually clog up no matter how many threads you have (and the more threads you have, the more they fight over the processor). The server sounds beefy, but since Zope is only going to be using one of those four cores, that might be deceptive. Kevin
Thanks for all your help. I have solved the problem with the Deadlock Debugger which showed up threads locked by a call to Aspell. Re-installing Aspell was the solution. I was very impressed with this product and all your assistance. Thanks again Garry
I was same problems with new IBM hardware 2 yrs ago and etc at past. Wait, please, your HW will be good after 2..3 months - it needs "burning process". It is known problem with very fresh silicon chips about diffusion of non-homogenities - look at mfg date at "asphalt squares". But my english is not as good to detailed explain whats happen. ----- Original Message ----- From: "Garry Saddington" <garry@schoolteachers.co.uk>
Following a hard disk crash 2 weeks ago we have installed Zope onto a new server and all was fine until yesterday morning when Zope stopped responding and required a restart to get it working. It did the same at 3-30pm today. We are using Zope 2.9.0 on Centos 5.1 on a quad Zeon server with 4gb. Ram. At the moment the server is in quite heavy use with teachers trying to write reports for a deadline tomorrow. These are sent to a Postgres DB via psycopg. There is nothing in Z2.log or event.log to point me to the problem. I am therefore asking for advice on the sorts of things that can cause Zope to stop responding and whether there is anything we can do to mitigate against such an event.
participants (4)
-
Garry Saddington -
Jaroslav Lukesh -
Peter Bengtsson -
Sours, Kevin