[Zope-dev] Re: [Zope] Re: Zope hanging (poss. threads-related)

[Tres Seaver]

Marcus Collins wrote:
> 
> I'm convinced there's some deep, dark, timing problem, and that it's
> thread-related...
> 
> This morning, I had a thread hang, again on Zope 2.1.3 running with four
> threads. I was watching for unfinished requests at the time, and caught this
> twenty minutes after it occurred.
> 
> I tried viewing a frameset page on the site. No problem. Then I tried
> launching /manage, and suddenly *all* the threads hung. This is the first
> time all the threads have hung. It's also the first time I've managed to
> catch an unfinished request and try viewing framesets within half an hour,
> which leads me to believe that, previously, ZServer was doing a good job of
> cleaning up the hung zombie threads (recall the zombie_timeout of 30
> minutes).
> 
> I'm actually wondering if reducing that zombie_timeout (and
> maintenance_interval in medusa/http_server.py) would go anywhere towards
> alleviating this problem as a temporary measure. Would there be any reasons
> not to try this?
> 
> I've added quite a lot of DebugLogger stuff to ZServer/PCGIServer.py, and
> modified the log analyser accordingly, in the hopes of nailing this sucker:
> 
> def send_response(self):
>     # create an output pipe by passing request to ZPublisher,
>     # and requesting a callback of self.log with the module
>     # name and PATH_INFO as an argument.
>     self.done=1
>     # MC 2000-04-13 additional logging
>     DebugLogger.log('X', id(self), 'send_response: before PCGIResponse')
>     response=PCGIResponse(stdout=PCGIPipe(self), stderr=StringIO())
>     # MC 2000-04-13 additional logging
>     DebugLogger.log('X', id(self), 'send_response: before HTTPRequest')
>     request=HTTPRequest(self.data, self.env, response)
>     # MC 2000-04-13 additional logging
>     DebugLogger.log('X', id(self), 'send_response: before handle')
>     handle(self.server.module, request, response)
> 
> Everything before the call to handle() is 100%. Sometimes, however, we don't
> get from handle() to the next stage. This is on Zope 2.1.6, which I've been
> running with up to 100 threads, although I unfortunately can't excercise
> that many! In fact, it has proven quite a mission to get Zope to hang, maybe
> because of the increased latency in serving requests due to the additional
> logging.
> 
> I've added this logging to the Zope 2.1.3 serving the live site, and will
> report my findings as soon as something untoward occurs. Maybe others who
> are experiencing hanging would also be able to do some extra logging and
> report the results [now, there, I see Wiki would be really useful!].
> 
> In the meantime, any suggestions as to where to go next will be keenly acted
> on!
> 
> Thanks all!
> 
> -- Marcus

There is an outstanding Collector issue related to zombies:

   http://classic.zope.org:8080/Collector/871/view

Amos, does this suggest anything?
-- 
=========================================================
Tres Seaver  tseaver@digicool.com   tseaver@palladion.com