[Zope] Zeo Client Hanging Unresponsive

Paul Williams PWilliams at diamonddata.com
Fri Feb 23 18:21:09 EST 2007 

I have posted this several times, but have not until now been able to
get DeadlockDebugger installed.  

 

zope 2.9.5 + zeo
pythonm2.4.3
Red Hat RHEL 4
Plone 2.5.1
 
Our zeo clients hang intermittently.  We have no way of reproducing the 
problem, but it occurs daily.  The client hangs and a restart seems to
fix the 
problem.
 
In the event log with tracing on we get
 
Trace zeo.zrpc.Connection(C) wait(16697) {server:8100} pending, async=0
 
There are hundreds to thousands of these until the server is restarted.
 
In the zeo log we get
 
Error caught in asyncor asyncore.py
 
error:(110,'Connection timed out')
 
We have been trying to track this down and have had no luck.  Does
anyone have 
any suggestions?  Below is our deadlock debugger output

 

 

Threads traceback dump at 2007-02-23 15:26:50

 

Thread -1269564496 (GET
/VirtualHostBase/https/soawds:443/VirtualHostRoot/Content///training):

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubC
ore/ZServerPublisher.py", line 23, in __init__

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py", line 395, in publish_module

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py", line 196, in publish_module_standard

  File
"/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/Pat
chStringIO.py", line 34, in new_publish

    x = Publish.old_publish(request, module_name, after_list, debug)

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py", line 115, in publish

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/m
apply.py", line 88, in mapply

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py", line 41, in call_object

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Shared/DC/Sc
ripts/Bindings.py", line 311, in __call__

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Shared/DC/Sc
ripts/Bindings.py", line 348, in _bindAndExec

  File
"/apps1/zope2.9.5/navo_instance/Products/CMFCore/FSPageTemplate.py",
line 195, in _exec

    result = self.pt_render(extra_context=bound_names)

  File
"/apps1/zope2.9.5/navo_instance/Products/CacheSetup/patch_cmf.py", line
38, in FSPT_pt_render

    result = FSPageTemplate.inheritedAttribute('pt_render')(

  File
"/apps1/zope2.9.5/navo_instance/Products/CacheSetup/patch_cmf.py", line
92, in PT_pt_render

    tal=not source, strictinsert=0)()

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py", line 238, in __call__

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py", line 281, in interpret

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py", line 749, in do_useMacro

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py", line 281, in interpret

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py", line 457, in do_optTag_tal

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py", line 442, in do_optTag

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py", line 437, in no_tag

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py", line 281, in interpret

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py", line 749, in do_useMacro

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py", line 281, in interpret

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInter
preter.py", line 507, in do_setLocal_tal

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/Pag
eTemplates/TALES.py", line 221, in evaluate

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/Pag
eTemplates/Expressions.py", line 185, in __call__

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/Pag
eTemplates/Expressions.py", line 180, in _eval

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/Pag
eTemplates/Expressions.py", line 85, in render

  File
"/apps1/zope2.9.5/navo_instance/Products/CMFPlone/browser/plone.py",
line 66, in globalize

    self._initializeData(options=options)

  File
"/apps1/zope2.9.5/navo_instance/Products/CMFPlone/browser/plone.py",
line 147, in _initializeData

    self._data['language'] =  self.request.get('language', None) or \

  File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/ClassGen.py",
line 58, in generatedAccessor

    return schema[name].get(self, **kw)

  File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/Field.py",
line 802, in get

    value = ObjectField.get(self, instance, **kwargs)

  File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/Field.py",
line 671, in get

    return self.getStorage(instance).get(self.getName(), instance,
**kwargs)

  File
"/apps1/zope2.9.5/navo_instance/Products/Archetypes/Storage/__init__.py"
, line 175, in get

    value = base._md[name]

  File
"/var/tmp/python2.4-2.4.3-root/apps1/python/lib/python2.4/UserDict.py",
line 17, in __getitem__

    def __getitem__(self, key): return self.data[key]

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect
ion.py", line 732, in setstate

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect
ion.py", line 786, in _setstate

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali
ze.py", line 604, in setGhostState

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali
ze.py", line 597, in getState

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali
ze.py", line 471, in _persistent_load

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali
ze.py", line 537, in load_oid

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect
ion.py", line 201, in get

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientSt
orage.py", line 746, in load

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientSt
orage.py", line 760, in loadEx

 

Thread -1290544208 (GET
/VirtualHostBase/https/soawds:443/VirtualHostRoot/Content//nav):

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubC
ore/ZServerPublisher.py", line 23, in __init__

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py", line 395, in publish_module

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py", line 196, in publish_module_standard

  File
"/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/Pat
chStringIO.py", line 34, in new_publish

    x = Publish.old_publish(request, module_name, after_list, debug)

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py", line 106, in publish

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/B
aseRequest.py", line 366, in traverse

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect
ion.py", line 732, in setstate

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect
ion.py", line 786, in _setstate

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali
ze.py", line 604, in setGhostState

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali
ze.py", line 597, in getState

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali
ze.py", line 471, in _persistent_load

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali
ze.py", line 537, in load_oid

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect
ion.py", line 201, in get

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientSt
orage.py", line 746, in load

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientSt
orage.py", line 760, in loadEx

 

Thread -1246884944 (GET
/VirtualHostBase/https/soawds:443/VirtualHostRoot/Content/carrier.jpg):

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubC
ore/ZServerPublisher.py", line 23, in __init__

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py", line 395, in publish_module

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py", line 196, in publish_module_standard

  File
"/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/Pat
chStringIO.py", line 34, in new_publish

    x = Publish.old_publish(request, module_name, after_list, debug)

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py", line 115, in publish

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/m
apply.py", line 88, in mapply

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py", line 41, in call_object

  File
"/apps1/zope2.9.5/navo_instance/Products/ATContentTypes/content/base.py"
, line 414, in index_html

    if data:

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect
ion.py", line 732, in setstate

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect
ion.py", line 786, in _setstate

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali
ze.py", line 604, in setGhostState

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali
ze.py", line 597, in getState

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali
ze.py", line 471, in _persistent_load

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali
ze.py", line 537, in load_oid

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect
ion.py", line 201, in get

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientSt
orage.py", line 746, in load

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientSt
orage.py", line 769, in loadEx

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ServerSt
ub.py", line 192, in loadEx

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/zrpc/con
nection.py", line 531, in call

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/zrpc/con
nection.py", line 638, in wait

  File
"/var/tmp/python2.4-2.4.3-root/apps1/python/lib/python2.4/asyncore.py",
line 122, in poll

    r, w, e = select.select(r, w, e, timeout)

 

Thread -1280054352 (GET
/VirtualHostBase/https/soawds:443/VirtualHostRoot/Content/):

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubC
ore/ZServerPublisher.py", line 23, in __init__

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py", line 395, in publish_module

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py", line 196, in publish_module_standard

  File
"/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/Pat
chStringIO.py", line 34, in new_publish

    x = Publish.old_publish(request, module_name, after_list, debug)

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/P
ublish.py", line 106, in publish

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/B
aseRequest.py", line 366, in traverse

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect
ion.py", line 732, in setstate

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect
ion.py", line 786, in _setstate

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali
ze.py", line 604, in setGhostState

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali
ze.py", line 597, in getState

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali
ze.py", line 471, in _persistent_load

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/seriali
ze.py", line 537, in load_oid

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connect
ion.py", line 201, in get

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientSt
orage.py", line 746, in load

  File
"/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientSt
orage.py", line 760, in loadEx

 

End of dump

 

Thank you,

Paul Williams

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mail.zope.org/pipermail/zope/attachments/20070223/64e999d4/attachment.htm

More information about the Zope mailing list