I am having a problem with zope 2.9.4 occasionally pegging the CPU and thereafter being unusable -- there are some indications in the event log that this is related to "pool_size" being too small. Lacking any other strategy to determine the cause of the problem, I am considering experimenting increasing "pool_size" to see what effect it has. A search returns some inprecise information on how to do this. My understanding is that this can be modified in the zope.conf for the instance. Can anybody give me precise information on modifying this parameter or point me to a concise resource on configuring the zodb in general,
On 1/15/07, Brian Sullivan <briansullivan@gmail.com> wrote:
I am having a problem with zope 2.9.4 occasionally pegging the CPU and thereafter being unusable -- there are some indications in the event log that this is related to "pool_size" being too small.
Lacking any other strategy to determine the cause of the problem, I am considering experimenting increasing "pool_size" to see what effect it has. A search returns some inprecise information on how to do this. My understanding is that this can be modified in the zope.conf for the instance. Can anybody give me precise information on modifying this parameter or point me to a concise resource on configuring the zodb in general,
Anybody?
Here's what I think I know (though I could be wrong): - The pool size is the number of ZODB connections maintained. - It should be at least as high as the number of zserver threads + 1 I think? - I use zserver threads 6 and pool_size 10, works fine. - zserver threads is basically a measure of how many concurrent requests you want your server to be able to handle, by default, it's 4 (and pool size 7 IIRC?). - The downside is you have to balance this with the cache size, since the cache size is per thread ... So your RAM usage could go up significantly. J.F. -----Original Message----- From: zope-bounces@zope.org [mailto:zope-bounces@zope.org] On Behalf Of Brian Sullivan Sent: January 16, 2007 10:31 AM To: Zope Users Subject: [Zope] Re: pool_size On 1/15/07, Brian Sullivan <briansullivan@gmail.com> wrote:
I am having a problem with zope 2.9.4 occasionally pegging the CPU and
thereafter being unusable -- there are some indications in the event log that this is related to "pool_size" being too small.
Lacking any other strategy to determine the cause of the problem, I am
considering experimenting increasing "pool_size" to see what effect it
has. A search returns some inprecise information on how to do this. My
understanding is that this can be modified in the zope.conf for the instance. Can anybody give me precise information on modifying this parameter or point me to a concise resource on configuring the zodb in
general,
Anybody? _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
On 1/16/07, Doyon, Jean-Francois <jdoyon@nrcan.gc.ca> wrote:
Here's what I think I know (though I could be wrong):
- The pool size is the number of ZODB connections maintained. - It should be at least as high as the number of zserver threads + 1 I think? - I use zserver threads 6 and pool_size 10, works fine. - zserver threads is basically a measure of how many concurrent requests you want your server to be able to handle, by default, it's 4 (and pool size 7 IIRC?). - The downside is you have to balance this with the cache size, since the cache size is per thread ... So your RAM usage could go up significantly.
Useful info -- I see a config setting for zserver-threads but how do you set the pool_size ? -- in the zope.conf? How?
Here's my setup using ZEO/ZRS: zserver-threads 6 <zodb_db main> mount-point / # ZODB cache, in number of objects cache-size 20000 pool-size 10 <zeoclient> server hostname1:port1 server hostname2:port2 read-only-fallback true storage 1 name zeostorage var $INSTANCE/var # ZEO client cache, in bytes cache-size 100MB # Uncomment to have a persistent disk cache #client zeo1 </zeoclient> </zodb_db> J.F. -----Original Message----- From: zope-bounces@zope.org [mailto:zope-bounces@zope.org] On Behalf Of Brian Sullivan Sent: January 16, 2007 11:02 AM To: Zope Users Subject: Re: [Zope] Re: pool_size On 1/16/07, Doyon, Jean-Francois <jdoyon@nrcan.gc.ca> wrote:
Here's what I think I know (though I could be wrong):
- The pool size is the number of ZODB connections maintained. - It should be at least as high as the number of zserver threads + 1 I
think? - I use zserver threads 6 and pool_size 10, works fine. - zserver threads is basically a measure of how many concurrent requests you want your server to be able to handle, by default, it's 4
(and pool size 7 IIRC?). - The downside is you have to balance this with the cache size, since the cache size is per thread ... So your RAM usage could go up significantly.
Useful info -- I see a config setting for zserver-threads but how do you set the pool_size ? -- in the zope.conf? How? _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Brian Sullivan wrote at 2007-1-15 11:27 -0500:
I am having a problem with zope 2.9.4 occasionally pegging the CPU and thereafter being unusable -- there are some indications in the event log that this is related to "pool_size" being too small.
Under under very special cases should a too small "pool_size" be responsible for unusablility of Zope. Such cases are e.g. when your requests call back to Zope (e.g. you have recursive request processing). In modern Zope versions, this is even less likely, as "pool_size" is used only as a hint and the ZODB is ready to create connections beyond the "pool_size" limit (issuing warnings however). Try Florent's "DeadlockDebugger" to determine what you "unuable" Zope is doing. It may tell you where Zope hangs. -- Dieter
On 1/16/07, Brian Sullivan <briansullivan@gmail.com> wrote:
On 1/16/07, Dieter Maurer <dieter@handshake.de> wrote:
Try Florent's "DeadlockDebugger" to determine what you "unuable" Zope is doing. It may tell you where Zope hangs.
Ok thanks -- I will have a look.
Ok so I have DeadlockDebugger installed and running ( I think) -- it produces an on screen "dump" that looks the following (this is under "normal" circustances -- without the failure that I was trying to debug happening). I am just trying to get a feel for what I am seeing. I see this same information every time -- so I am assuming that this is the norm? I am presuming there will be different information if I am in the throes of the cpu pegging event I am trying to debug? Threads traceback dump at 2007-01-17 13:24:11 Thread 3396: File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZServerPublisher.py", line 19, in __init__ name, request, response=accept() File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZRendezvous.py", line 73, in accept l.acquire() Thread 5644: File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZServerPublisher.py", line 19, in __init__ name, request, response=accept() File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZRendezvous.py", line 73, in accept l.acquire() Thread 1848: File "C:\Program Files\Zope-2.9.1\bin\lib\threading.py", line 442, in __bootstrap self.run() File "C:\Program Files\Zope-2.9.1\bin\lib\threading.py", line 422, in run self.__target(*self.__args, **self.__kwargs) File "C:\Program Files\Zope-2.9.1\lib\python\Signals\WinSignalHandler.py", line 185, in signalCheckerThread win32event.INFINITE) Thread 4240: File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZServerPublisher.py", line 19, in __init__ name, request, response=accept() File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZRendezvous.py", line 73, in accept l.acquire() Thread 2808: File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZServerPublisher.py", line 19, in __init__ name, request, response=accept() File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZRendezvous.py", line 73, in accept l.acquire() Thread 3124: File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZServerPublisher.py", line 19, in __init__ name, request, response=accept() File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZRendezvous.py", line 73, in accept l.acquire() Thread 4280: File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZServerPublisher.py", line 19, in __init__ name, request, response=accept() File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZRendezvous.py", line 73, in accept l.acquire() End of dump
Ok so I have DeadlockDebugger installed and running ( I think) -- it produces an on screen "dump" that looks the following (this is under "normal" circustances -- without the failure that I was trying to debug happening). I am just trying to get a feel for what I am seeing. I see this same information every time -- so I am assuming that this is the norm?
What you see is traceback from all your zope threads. This is traceback from thread 3396:
Thread 3396: File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZServerPublisher.py", line 19, in __init__ name, request, response=accept() File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZRendezvous.py", line 73, in accept l.acquire()
This shows that thread 3396 is not busy now (waiting in acquire function). Thread 1848 does something different. I'm not sure but maybe this is a main thread under windows or something like that.
I am presuming there will be different information if I am in the throes of the cpu pegging event I am trying to debug?
As soon as your Zope will be doing something (rendering page etc.) you'll see different results. Try to write a long running script, execute this, and then check deadlock debugger. You should see that one thread is busy. You just need to have your zope under some load. -- Maciej Wisniowski
On 1/17/07, Maciej Wisniowski <maciej.wisniowski@coig.katowice.pl> wrote:
Ok so I have DeadlockDebugger installed and running ( I think) -- it produces an on screen "dump" that looks the following (this is under "normal" circustances -- without the failure that I was trying to debug happening). I am just trying to get a feel for what I am seeing. I see this same information every time -- so I am assuming that this is the norm?
What you see is traceback from all your zope threads. This is traceback from thread 3396:
Thread 3396: File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZServerPublisher.py", line 19, in __init__ name, request, response=accept() File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZRendezvous.py", line 73, in accept l.acquire()
This shows that thread 3396 is not busy now (waiting in acquire function).
Thread 1848 does something different. I'm not sure but maybe this is a main thread under windows or something like that.
I am presuming there will be different information if I am in the throes of the cpu pegging event I am trying to debug?
As soon as your Zope will be doing something (rendering page etc.) you'll see different results.
Try to write a long running script, execute this, and then check deadlock debugger. You should see that one thread is busy. You just need to have your zope under some load.
OK - thanks - I will keep poking.
This information is also available in the Control Panel's debug section. You can see which thread is busy doing what ... Depending on exactly how "locked up" it is, you can try going there, see what the current requests are pending, and in which thread. That panel also shows you cache information, how many objects are loaded and so on. This could help you determine if you're leaking references to certain objects for instance? J.F. -----Original Message----- From: zope-bounces@zope.org [mailto:zope-bounces@zope.org] On Behalf Of Brian Sullivan Sent: January 17, 2007 2:06 PM To: Zope Users Subject: Re: [Zope] pool_size On 1/17/07, Maciej Wisniowski <maciej.wisniowski@coig.katowice.pl> wrote:
Ok so I have DeadlockDebugger installed and running ( I think) -- it produces an on screen "dump" that looks the following (this is under "normal" circustances -- without the failure that I was trying
to debug happening). I am just trying to get a feel for what I am seeing. I see this same information every time -- so I am assuming that this
is the norm? What you see is traceback from all your zope threads. This is traceback from thread 3396:
Thread 3396: File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZServerPublisher.py", line 19, in __init__ name, request, response=accept() File "C:\Program Files\Zope-2.9.1\lib\python\ZServer\PubCore\ZRendezvous.py", line 73, in accept l.acquire()
This shows that thread 3396 is not busy now (waiting in acquire function).
Thread 1848 does something different. I'm not sure but maybe this is a
main thread under windows or something like that.
I am presuming there will be different information if I am in the throes of the cpu pegging event I am trying to debug?
As soon as your Zope will be doing something (rendering page etc.) you'll see different results.
Try to write a long running script, execute this, and then check deadlock debugger. You should see that one thread is busy. You just need to have your zope under some load.
OK - thanks - I will keep poking. _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
On 1/17/07, Doyon, Jean-Francois <jdoyon@nrcan.gc.ca> wrote:
This information is also available in the Control Panel's debug section. You can see which thread is busy doing what ...
Depending on exactly how "locked up" it is, you can try going there, see what the current requests are pending, and in which thread.
That panel also shows you cache information, how many objects are loaded and so on. This could help you determine if you're leaking references to certain objects for instance?
When it is locked up I can't access the control panel.
Used to use the debug a lot but then when we found DeadLockDebugger (DLD), I was hooked...mainly because there is no 'baggage' to loading the page. Debug actually uses one of the threads to show you that page. The content of the thread sorta looks like the whole REQUEST object, which is helpful sometimes too. DLD is fast (normally) and sometimes rather addictive to watch...to see what your servers are doing. We use ZEO and 2 public clients fronted by squid and a third we use as an internal admin client that is not served by squid. I just use a set of bookmarked tabs in FireFox and open a tab for each server. Then I just hit 'reload all' and see what they are all doing. Just my 2 cents. Allen Brian Sullivan wrote:
On 1/17/07, Doyon, Jean-Francois <jdoyon@nrcan.gc.ca> wrote:
This information is also available in the Control Panel's debug section. You can see which thread is busy doing what ...
Depending on exactly how "locked up" it is, you can try going there, see what the current requests are pending, and in which thread.
That panel also shows you cache information, how many objects are loaded and so on. This could help you determine if you're leaking references to certain objects for instance?
When it is locked up I can't access the control panel. _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
participants (5)
-
Allen Schmidt Sr. -
Brian Sullivan -
Dieter Maurer -
Doyon, Jean-Francois -
Maciej Wisniowski