[Zope] Re: zope unresponsive
Paul Williams
pwilliams at diamonddata.com
Tue Feb 27 08:20:01 EST 2007
Ok, here is what we have. I did a netstat on both machines, client and
server. The client sees and established connection and the server does
not. In the server log there is a disconnect. As far as hardware
between them, there is a switch (dell powerconnect 6024). Web Server
Directors might get hold of it but there are no hops on traceroute.
Traceroute only shows the client machine and the server machine.
So the client is just continuously polling the connection but getting
nothing back.
What we are thinking about doing is changing the code in
zrpc/connection.py to close the connection in wait (line 638 zope
version 2.9.5) if the wait time gets too large or the poll has happened
too many times.
We are great at plone development, but have very little backend zope
development. Would someone please advise me as to whether this is going
to cause more problems?
Thanks,
Paul Williams
Paul Williams wrote:
> I have posted this several times, but have not until now been able to
> get DeadlockDebugger installed. I see several people have had this
> problem, but no-one has posted a solution.
>
>
>
> zope 2.9.5 + zeo
>
> pythonm2.4.3
>
> Red Hat RHEL 4
>
> Plone 2.5.1
>
>
>
> Our zeo clients hang intermittently. We have no way of reproducing the
>
> problem, but it occurs daily. The client hangs and a restart seems to fix the
>
> problem.
>
>
>
> In the event log with tracing on we get
>
>
>
> Trace zeo.zrpc.Connection(C) wait(16697) {server:8100} pending, async=0
>
>
>
> There are hundreds to thousands of these until the server is restarted.
>
>
>
> In the zeo log we get
>
>
>
> Error caught in asyncor asyncore.py
>
>
>
> error:(110,'Connection timed out')
>
>
>
> We have been trying to track this down and have had no luck. Does anyone have
>
> any suggestions? Below is our deadlock debugger output
>
>
>
>
>
> Threads traceback dump at 2007-02-23 15:26:50
>
>
>
> Thread -1269564496 (GET
> /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content///training):
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubCore/ZServerPublisher.py",
> line 23, in __init__
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py",
> line 395, in publish_module
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py",
> line 196, in publish_module_standard
>
> File
> "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/PatchStringIO.py",
> line 34, in new_publish
>
> x = Publish.old_publish(request, module_name, after_list, debug)
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py",
> line 115, in publish
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/mapply.py",
> line 88, in mapply
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py",
> line 41, in call_object
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Shared/DC/Scripts/Bindings.py",
> line 311, in __call__
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Shared/DC/Scripts/Bindings.py",
> line 348, in _bindAndExec
>
> File
> "/apps1/zope2.9.5/navo_instance/Products/CMFCore/FSPageTemplate.py",
> line 195, in _exec
>
> result = self.pt_render(extra_context=bound_names)
>
> File
> "/apps1/zope2.9.5/navo_instance/Products/CacheSetup/patch_cmf.py", line
> 38, in FSPT_pt_render
>
> result = FSPageTemplate.inheritedAttribute('pt_render')(
>
> File
> "/apps1/zope2.9.5/navo_instance/Products/CacheSetup/patch_cmf.py", line
> 92, in PT_pt_render
>
> tal=not source, strictinsert=0)()
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py",
> line 238, in __call__
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py",
> line 281, in interpret
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py",
> line 749, in do_useMacro
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py",
> line 281, in interpret
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py",
> line 457, in do_optTag_tal
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py",
> line 442, in do_optTag
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py",
> line 437, in no_tag
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py",
> line 281, in interpret
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py",
> line 749, in do_useMacro
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py",
> line 281, in interpret
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/TAL/TALInterpreter.py",
> line 507, in do_setLocal_tal
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/PageTemplates/TALES.py",
> line 221, in evaluate
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/PageTemplates/Expressions.py",
> line 185, in __call__
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/PageTemplates/Expressions.py",
> line 180, in _eval
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/Products/PageTemplates/Expressions.py",
> line 85, in render
>
> File
> "/apps1/zope2.9.5/navo_instance/Products/CMFPlone/browser/plone.py",
> line 66, in globalize
>
> self._initializeData(options=options)
>
> File
> "/apps1/zope2.9.5/navo_instance/Products/CMFPlone/browser/plone.py",
> line 147, in _initializeData
>
> self._data['language'] = self.request.get('language', None) or \
>
> File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/ClassGen.py",
> line 58, in generatedAccessor
>
> return schema[name].get(self, **kw)
>
> File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/Field.py",
> line 802, in get
>
> value = ObjectField.get(self, instance, **kwargs)
>
> File "/apps1/zope2.9.5/navo_instance/Products/Archetypes/Field.py",
> line 671, in get
>
> return self.getStorage(instance).get(self.getName(), instance, **kwargs)
>
> File
> "/apps1/zope2.9.5/navo_instance/Products/Archetypes/Storage/__init__.py",
> line 175, in get
>
> value = base._md[name]
>
> File
> "/var/tmp/python2.4-2.4.3-root/apps1/python/lib/python2.4/UserDict.py",
> line 17, in __getitem__
>
> def __getitem__(self, key): return self.data[key]
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py",
> line 732, in setstate
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py",
> line 786, in _setstate
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py",
> line 604, in setGhostState
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py",
> line 597, in getState
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py",
> line 471, in _persistent_load
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py",
> line 537, in load_oid
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py",
> line 201, in get
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py",
> line 746, in load
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py",
> line 760, in loadEx
>
>
>
> Thread -1290544208 (GET
> /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content//nav):
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubCore/ZServerPublisher.py",
> line 23, in __init__
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py",
> line 395, in publish_module
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py",
> line 196, in publish_module_standard
>
> File
> "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/PatchStringIO.py",
> line 34, in new_publish
>
> x = Publish.old_publish(request, module_name, after_list, debug)
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py",
> line 106, in publish
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/BaseRequest.py",
> line 366, in traverse
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py",
> line 732, in setstate
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py",
> line 786, in _setstate
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py",
> line 604, in setGhostState
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py",
> line 597, in getState
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py",
> line 471, in _persistent_load
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py",
> line 537, in load_oid
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py",
> line 201, in get
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py",
> line 746, in load
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py",
> line 760, in loadEx
>
>
>
> Thread -1246884944 (GET
> /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content/carrier.jpg):
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubCore/ZServerPublisher.py",
> line 23, in __init__
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py",
> line 395, in publish_module
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py",
> line 196, in publish_module_standard
>
> File
> "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/PatchStringIO.py",
> line 34, in new_publish
>
> x = Publish.old_publish(request, module_name, after_list, debug)
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py",
> line 115, in publish
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/mapply.py",
> line 88, in mapply
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py",
> line 41, in call_object
>
> File
> "/apps1/zope2.9.5/navo_instance/Products/ATContentTypes/content/base.py",
> line 414, in index_html
>
> if data:
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py",
> line 732, in setstate
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py",
> line 786, in _setstate
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py",
> line 604, in setGhostState
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py",
> line 597, in getState
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py",
> line 471, in _persistent_load
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py",
> line 537, in load_oid
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py",
> line 201, in get
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py",
> line 746, in load
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py",
> line 769, in loadEx
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ServerStub.py",
> line 192, in loadEx
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/zrpc/connection.py",
> line 531, in call
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/zrpc/connection.py",
> line 638, in wait
>
> File
> "/var/tmp/python2.4-2.4.3-root/apps1/python/lib/python2.4/asyncore.py",
> line 122, in poll
>
> r, w, e = select.select(r, w, e, timeout)
>
>
>
> Thread -1280054352 (GET
> /VirtualHostBase/https/soawds:443/VirtualHostRoot/Content/):
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZServer/PubCore/ZServerPublisher.py",
> line 23, in __init__
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py",
> line 395, in publish_module
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py",
> line 196, in publish_module_standard
>
> File
> "/apps1/zope2.9.5/navo_instance/Products/PlacelessTranslationService/PatchStringIO.py",
> line 34, in new_publish
>
> x = Publish.old_publish(request, module_name, after_list, debug)
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/Publish.py",
> line 106, in publish
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZPublisher/BaseRequest.py",
> line 366, in traverse
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py",
> line 732, in setstate
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py",
> line 786, in _setstate
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py",
> line 604, in setGhostState
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py",
> line 597, in getState
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py",
> line 471, in _persistent_load
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/serialize.py",
> line 537, in load_oid
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZODB/Connection.py",
> line 201, in get
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py",
> line 746, in load
>
> File
> "/var/tmp/Zope-2.9.5-1-buildroot/apps1/zope2.9.5/lib/python/ZEO/ClientStorage.py",
> line 760, in loadEx
>
>
>
> End of dump
>
>
>
> Thank you,
>
> Paul Williams
>
>
>
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Zope maillist - Zope at zope.org
> http://mail.zope.org/mailman/listinfo/zope
> ** No cross posts or HTML encoding! **
> (Related lists -
> http://mail.zope.org/mailman/listinfo/zope-announce
> http://mail.zope.org/mailman/listinfo/zope-dev )
More information about the Zope
mailing list