I have posted this several times, but have not until now been able to get DeadlockDebugger installed. zope 2.9.5 + zeo pythonm2.4.3 Red Hat RHEL 4 Plone 2.5.1 Our zeo clients hang intermittently. We have no way of reproducing the problem, but it occurs daily. The client hangs and a restart seems to fix the problem. In the event log with tracing on we get Trace zeo.zrpc.Connection(C) wait(16697) {server:8100} pending, async=0 There are hundreds to thousands of these until the server is restarted. In the zeo log we get Error caught in asyncor asyncore.py error:(110,'Connection timed out') We have been trying to track this down and have had no luck. Does anyone have any suggestions? Below is our deadlock debugger output Threads traceback dump at 2007-02-23 15:26:50 Thread -1269564496 (GET /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content///training): File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubC ore/ZServerPublisher.py", line 23, in __init__ File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 395, in publish_module File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 196, in publish_module_standard File "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/Pat chStringIO.py", line 34, in new_publish x = Publish.old_publish(request, module_name, after_list, debug) File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 115, in publish File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/m apply.py", line 88, in mapply File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 41, in call_object File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Shared/DC/Sc ripts/Bindings.py", line 311, in __call__ File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Shared/DC/Sc ripts/Bindings.py", line 348, in _bindAndExec File "/apps1/zope2.9.5/navo_instance/Products/CMFCore/FSPageTemplate.py", line 195, in _exec result = self.pt_render(extra_context=bound_names) File "/apps1/zope2.9.5/navo_instance/Products/CacheSetup/patch_cmf.py", line 38, in FSPT_pt_render result = FSPageTemplate.inheritedAttribute('pt_render')( File "/apps1/zope2.9.5/navo_instance/Products/CacheSetup/patch_cmf.py", line 92, in PT_pt_render tal=not source, strictinsert=0)() File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 238, in __call__ File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 281, in interpret File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 749, in do_useMacro File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 281, in interpret File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 457, in do_optTag_tal File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 442, in do_optTag File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 437, in no_tag File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 281, in interpret File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 749, in do_useMacro File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 281, in interpret File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter preter.py", line 507, in do_setLocal_tal File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/Pag eTemplates/TALES.py", line 221, in evaluate File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/Pag eTemplates/Expressions.py", line 185, in __call__ File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/Pag eTemplates/Expressions.py", line 180, in _eval File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/Pag eTemplates/Expressions.py", line 85, in render File "/apps1/zope2.9.5/navo_instance/Products/CMFPlone/browser/plone.py", line 66, in globalize self._initializeData(options=options) File "/apps1/zope2.9.5/navo_instance/Products/CMFPlone/browser/plone.py", line 147, in _initializeData self._data['language'] = self.request.get('language', None) or \ File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/ClassGen.py", line 58, in generatedAccessor return schema[name].get(self, **kw) File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/Field.py", line 802, in get value = ObjectField.get(self, instance, **kwargs) File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/Field.py", line 671, in get return self.getStorage(instance).get(self.getName(), instance, **kwargs) File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/Storage/__init__.py" , line 175, in get value = base._md[name] File "/var/tmp/python2.4-2.4.3-root/apps1/python/lib/python2.4/UserDict.py", line 17, in __getitem__ def __getitem__(self, key): return self.data[key] File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect ion.py", line 732, in setstate File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect ion.py", line 786, in _setstate File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali ze.py", line 604, in setGhostState File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali ze.py", line 597, in getState File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali ze.py", line 471, in _persistent_load File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali ze.py", line 537, in load_oid File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect ion.py", line 201, in get File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientSt orage.py", line 746, in load File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientSt orage.py", line 760, in loadEx Thread -1290544208 (GET /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content//nav): File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubC ore/ZServerPublisher.py", line 23, in __init__ File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 395, in publish_module File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 196, in publish_module_standard File "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/Pat chStringIO.py", line 34, in new_publish x = Publish.old_publish(request, module_name, after_list, debug) File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 106, in publish File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/B aseRequest.py", line 366, in traverse File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect ion.py", line 732, in setstate File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect ion.py", line 786, in _setstate File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali ze.py", line 604, in setGhostState File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali ze.py", line 597, in getState File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali ze.py", line 471, in _persistent_load File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali ze.py", line 537, in load_oid File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect ion.py", line 201, in get File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientSt orage.py", line 746, in load File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientSt orage.py", line 760, in loadEx Thread -1246884944 (GET /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content/carrier.jpg): File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubC ore/ZServerPublisher.py", line 23, in __init__ File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 395, in publish_module File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 196, in publish_module_standard File "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/Pat chStringIO.py", line 34, in new_publish x = Publish.old_publish(request, module_name, after_list, debug) File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 115, in publish File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/m apply.py", line 88, in mapply File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 41, in call_object File "/apps1/zope2.9.5/navo_instance/Products/ATContentTypes/content/base.py" , line 414, in index_html if data: File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect ion.py", line 732, in setstate File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect ion.py", line 786, in _setstate File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali ze.py", line 604, in setGhostState File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali ze.py", line 597, in getState File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali ze.py", line 471, in _persistent_load File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali ze.py", line 537, in load_oid File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect ion.py", line 201, in get File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientSt orage.py", line 746, in load File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientSt orage.py", line 769, in loadEx File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ServerSt ub.py", line 192, in loadEx File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/zrpc/con nection.py", line 531, in call File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/zrpc/con nection.py", line 638, in wait File "/var/tmp/python2.4-2.4.3-root/apps1/python/lib/python2.4/asyncore.py", line 122, in poll r, w, e = select.select(r, w, e, timeout) Thread -1280054352 (GET /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content/): File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubC ore/ZServerPublisher.py", line 23, in __init__ File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 395, in publish_module File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 196, in publish_module_standard File "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/Pat chStringIO.py", line 34, in new_publish x = Publish.old_publish(request, module_name, after_list, debug) File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P ublish.py", line 106, in publish File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/B aseRequest.py", line 366, in traverse File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect ion.py", line 732, in setstate File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect ion.py", line 786, in _setstate File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali ze.py", line 604, in setGhostState File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali ze.py", line 597, in getState File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali ze.py", line 471, in _persistent_load File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali ze.py", line 537, in load_oid File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect ion.py", line 201, in get File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientSt orage.py", line 746, in load File "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientSt orage.py", line 760, in loadEx End of dump Thank you, Paul Williams