Ok, here is what we have. I did a netstat on both machines, client and server. The client sees and established connection and the server does not. In the server log there is a disconnect. As far as hardware between them, there is a switch (dell powerconnect 6024). Web Server Directors might get hold of it but there are no hops on traceroute. Traceroute only shows the client machine and the server machine. So the client is just continuously polling the connection but getting nothing back. What we are thinking about doing is changing the code in zrpc/connection.py to close the connection in wait (line 638 zope version 2.9.5) if the wait time gets too large or the poll has happened too many times. We are great at plone development, but have very little backend zope development. Would someone please advise me as to whether this is going to cause more problems? Thanks, Paul Williams Paul Williams wrote:
I have posted this several times, but have not until now been able to get DeadlockDebugger installed. I see several people have had this problem, but no-one has posted a solution.
zope 2.9.5 + zeo
pythonm2.4.3
Red Hat RHEL 4
Plone 2.5.1
Our zeo clients hang intermittently. We have no way of reproducing the
problem, but it occurs daily. The client hangs and a restart seems to fix the
problem.
In the event log with tracing on we get
Trace zeo.zrpc.Connection(C) wait(16697) {server:8100} pending, async=0
There are hundreds to thousands of these until the server is restarted.
In the zeo log we get
Error caught in asyncor asyncore.py
error:(110,'Connection timed out')
We have been trying to track this down and have had no luck. Does anyone have
any suggestions? Below is our deadlock debugger output
Threads traceback dump at 2007-02-23 15:26:50
Thread -1269564496 (GET /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content///training):
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubCore/ZServerPublisher.py", line 23, in __init__
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 395, in publish_module
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 196, in publish_module_standard
File "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/PatchStringIO.py", line 34, in new_publish
x = Publish.old_publish(request, module_name, after_list, debug)
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 115, in publish
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/mapply.py", line 88, in mapply
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 41, in call_object
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Shared/DC/Scripts/Bindings.py", line 311, in __call__
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Shared/DC/Scripts/Bindings.py", line 348, in _bindAndExec
File "/apps1/zope2.9.5/navo_instance/Products/CMFCore/FSPageTemplate.py", line 195, in _exec
result = self.pt_render(extra_context=bound_names)
File "/apps1/zope2.9.5/navo_instance/Products/CacheSetup/patch_cmf.py", line 38, in FSPT_pt_render
result = FSPageTemplate.inheritedAttribute('pt_render')(
File "/apps1/zope2.9.5/navo_instance/Products/CacheSetup/patch_cmf.py", line 92, in PT_pt_render
tal=not source, strictinsert=0)()
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 238, in __call__
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 281, in interpret
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 749, in do_useMacro
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 281, in interpret
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 457, in do_optTag_tal
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 442, in do_optTag
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 437, in no_tag
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 281, in interpret
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 749, in do_useMacro
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 281, in interpret
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py", line 507, in do_setLocal_tal
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/PageTemplates/TALES.py", line 221, in evaluate
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/PageTemplates/Expressions.py", line 185, in __call__
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/PageTemplates/Expressions.py", line 180, in _eval
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/PageTemplates/Expressions.py", line 85, in render
File "/apps1/zope2.9.5/navo_instance/Products/CMFPlone/browser/plone.py", line 66, in globalize
self._initializeData(options=options)
File "/apps1/zope2.9.5/navo_instance/Products/CMFPlone/browser/plone.py", line 147, in _initializeData
self._data['language'] = self.request.get('language', None) or \
File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/ClassGen.py", line 58, in generatedAccessor
return schema[name].get(self, **kw)
File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/Field.py", line 802, in get
value = ObjectField.get(self, instance, **kwargs)
File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/Field.py", line 671, in get
return self.getStorage(instance).get(self.getName(), instance, **kwargs)
File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/Storage/__init__.py", line 175, in get
value = base._md[name]
File "/var/tmp/python2.4-2.4.3-root/apps1/python/lib/python2.4/UserDict.py", line 17, in __getitem__
def __getitem__(self, key): return self.data[key]
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 732, in setstate
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 786, in _setstate
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 604, in setGhostState
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 597, in getState
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 471, in _persistent_load
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 537, in load_oid
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 201, in get
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py", line 746, in load
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py", line 760, in loadEx
Thread -1290544208 (GET /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content//nav):
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubCore/ZServerPublisher.py", line 23, in __init__
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 395, in publish_module
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 196, in publish_module_standard
File "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/PatchStringIO.py", line 34, in new_publish
x = Publish.old_publish(request, module_name, after_list, debug)
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 106, in publish
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/BaseRequest.py", line 366, in traverse
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 732, in setstate
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 786, in _setstate
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 604, in setGhostState
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 597, in getState
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 471, in _persistent_load
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 537, in load_oid
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 201, in get
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py", line 746, in load
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py", line 760, in loadEx
Thread -1246884944 (GET /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content/carrier.jpg):
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubCore/ZServerPublisher.py", line 23, in __init__
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 395, in publish_module
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 196, in publish_module_standard
File "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/PatchStringIO.py", line 34, in new_publish
x = Publish.old_publish(request, module_name, after_list, debug)
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 115, in publish
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/mapply.py", line 88, in mapply
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 41, in call_object
File "/apps1/zope2.9.5/navo_instance/Products/ATContentTypes/content/base.py", line 414, in index_html
if data:
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 732, in setstate
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 786, in _setstate
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 604, in setGhostState
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 597, in getState
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 471, in _persistent_load
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 537, in load_oid
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 201, in get
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py", line 746, in load
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py", line 769, in loadEx
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ServerStub.py", line 192, in loadEx
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/zrpc/connection.py", line 531, in call
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/zrpc/connection.py", line 638, in wait
File "/var/tmp/python2.4-2.4.3-root/apps1/python/lib/python2.4/asyncore.py", line 122, in poll
r, w, e = select.select(r, w, e, timeout)
Thread -1280054352 (GET /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content/):
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubCore/ZServerPublisher.py", line 23, in __init__
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 395, in publish_module
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 196, in publish_module_standard
File "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/PatchStringIO.py", line 34, in new_publish
x = Publish.old_publish(request, module_name, after_list, debug)
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py", line 106, in publish
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/BaseRequest.py", line 366, in traverse
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 732, in setstate
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 786, in _setstate
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 604, in setGhostState
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 597, in getState
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 471, in _persistent_load
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py", line 537, in load_oid
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py", line 201, in get
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py", line 746, in load
File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py", line 760, in loadEx
End of dump
Thank you,
Paul Williams
------------------------------------------------------------------------
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )