Threads dying on zope 2.7.0 on one Zeo setup, ZServer exception
Hi ya, I noticed some weird bahivour that was bringing a setup down on our servers on testing installation. In our setup we create development sandboxes with makefile and in similar fashion also testing/production servers. Our testing/production setup has a little bit different config files where from standard config all the commented out lines have been removed. Also compared to development instances the instance home is in other place. Other than that setups are same. At the moment with testing setup under load test ZServer dies - leaving all but one thread at almost 0% cpu and that one thread working in high loads of 90%. Testing similar zeo-setup with development instances do not produce this. We are a bit scared on this, since we do not have clue on what causes following behaviour - and eventhough we see that we can get it to work, we would like to have an idea on what might be behind this - so that it does not bite our butt again. Below is log snip and traceback of the exceptions when other threads go down. In log-file we get: 2004-05-17T16:47:17 ERROR(200) ZServer uncaptured python exception, closing channel <ZServer.HTTPServer.zhttp_channel connected 127.0.0.1:47855 at 0x42fdb36c channel#: 589 requests:> (socket.error:(104, 'Connection reset by peer') [/opt/portaali/ContentManagement-1.0/python-2.3.3/lib/python2.3/asynchat.py|initiate_send|218] [/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZServer/medusa/http_server.py|send|417] [/opt/portaali/ContentManagement-1.0/python-2.3.3/lib/python2.3/asyncore.py|send|337]) 2004-05-17T16:48:49 ERROR(200) ZServer uncaptured python exception, closing channel <ZServer.HTTPServer.zhttp_channel connected 127.0.0.1:47924 at 0x430538ac channel#: 651 requests:> (socket.error:(104, 'Connection reset by peer') [/opt/portaali/ContentManagement-1.0/python-2.3.3/lib/python2.3/asynchat.py|initiate_send|218] [/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZServer/medusa/http_server.py|send|417] [/opt/portaali/ContentManagement-1.0/python-2.3.3/lib/python2.3/asyncore.py|send|337]) 2004-05-17T16:50:05 ERROR(200) ZServer uncaptured python exception, closing channel <ZServer.HTTPServer.zhttp_channel connected 127.0.0.1:47961 at 0x42f3ba4c channel#: 683 requests:> (socket.error:(104, 'Connection reset by peer') [/opt/portaali/ContentManagement-1.0/python-2.3.3/lib/python2.3/asynchat.py|initiate_send|218] [/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZServer/medusa/http_server.py|send|417] [/opt/portaali/ContentManagement-1.0/python-2.3.3/lib/python2.3/asyncore.py|send|337]) In tracback we will get: 2004-05-18T11:44:14 INFO(0) Zope Ready to handle requests Unhandled exception in thread started by <class ZServer.PubCore.ZServerPublisher.ZServerPublisher at 0x405435cc> Traceback (most recent call last): File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZServer/PubCore/ZServerPublisher.py", line 23, in __init__ response=response) File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", line 372, in publish_module environ, debug, request, response) File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", line 173, in publish_module_standard request.response.exception() AttributeError: 'NoneType' object has no attribute 'exception' Unhandled exception in thread started by <class ZServer.PubCore.ZServerPublisher.ZServerPublisher at 0x405435cc> Traceback (most recent call last): File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZServer/PubCore/ZServerPublisher.py", line 23, in __init__ response=response) File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", line 372, in publish_module environ, debug, request, response) File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", line 173, in publish_module_standard request.response.exception() AttributeError: 'NoneType' object has no attribute 'exception' Unhandled exception in thread started by <class ZServer.PubCore.ZServerPublisher.ZServerPublisher at 0x405435cc> Traceback (most recent call last): File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZServer/PubCore/ZServerPublisher.py", line 23, in __init__ response=response) File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", line 372, in publish_module environ, debug, request, response) File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", line 173, in publish_module_standard request.response.exception() AttributeError: 'NoneType' object has no attribute 'exception' Unhandled exception in thread started by <class ZServer.PubCore.ZServerPublisher.ZServerPublisher at 0x405435cc> Traceback (most recent call last): File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZServer/PubCore/ZServerPublisher.py", line 23, in __init__ response=response) File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", line 372, in publish_module environ, debug, request, response) File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", line 173, in publish_module_standard request.response.exception() AttributeError: 'NoneType' object has no attribute 'exception' Unhandled exception in thread started by <class ZServer.PubCore.ZServerPublisher.ZServerPublisher at 0x405435cc> Traceback (most recent call last): File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZServer/PubCore/ZServerPublisher.py", line 23, in __init__ response=response) File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", line 372, in publish_module environ, debug, request, response) File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", line 173, in publish_module_standard request.response.exception() AttributeError: 'NoneType' object has no attribute 'exception' Unhandled exception in thread started by <class ZServer.PubCore.ZServerPublisher.ZServerPublisher at 0x405435cc> Traceback (most recent call last): File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZServer/PubCore/ZServerPublisher.py", line 23, in __init__ response=response) File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", line 372, in publish_module environ, debug, request, response) File "/opt/portaali/ContentManagement-1.0/zope-2.7.0/lib/python/ZPublisher/Publish.py", line 173, in publish_module_standard request.response.exception() AttributeError: 'NoneType' object has no attribute 'exception' -huima
Heimo Laukkanen wrote at 2004-5-18 14:10 +0300:
... At the moment with testing setup under load test ZServer dies - leaving all but one thread at almost 0% cpu and that one thread working in high loads of 90%. Testing similar zeo-setup with development instances do not produce this.
Provided, you work under Linux, you can try to attach to the process with GDB and analyse the thread behaviour. I use the functions "ps" (Print String) and "pfr" (Print FRame) during such analysis. They are defined by: def ps x/s ({PyStringObject}$arg0)->ob_sval end def pfr ps f->f_code->co_filename ps f->f_code->co_name p f->f_lineno end You can apply "ps" to an string object. You can call "pfr" in an "eval_frame" frame context. -- Dieter
participants (2)
-
Dieter Maurer -
Heimo Laukkanen