[Zope] Re: Zope 2.8.4 strange behavior

[Dennis Allison]

Just went throught that exercise with Andy and installed a patch to 
MySQLDA that effectively ignores the 'release unlocked lock' problem that 
has been plaguing us.   I shoulda guessed that is the first place to look.

I'll get and install the DeadlockDebugger forthwith.

Thanks.


On Mon, 28 Nov 2005, Florent Guillaume wrote:

> Dennis Allison wrote:
> > We have two recent instances in our production sites where Zope suddenly
> > stops responding.  It is not a new problem, but we've now been confronted
> > with two clean examples and nothing to blame them on.  The problem appears
> > to be independent of load as both incidents were on lightly loaded
> > machines.
> > 
> > A check of the logs (Linux and Zope) shows nothing obviously amiss except
> > that the trace log (the old -M log) shows a sudden increase in active
> > requests from the typical 0 or 1 to 1300 or more.  In this context an
> > "active request' is total number of requests pending at the end of this
> > request and is computed by post-processing.  We front-end Zope with pound 
> > and make heavy use of MySQL.  Both show a plethora of incomplete 
> > transactions.  
> > 
> > Examination of the raw trace log shows that Zope is continuing to accept
> > requests, but nothing getting done.  The raw log date-stamps four internal
> > states for each transaction.  The states are Begin (B), Input (I),
> > action (A), and End (E).  Inputs are gathered between B and I, outputs is
> > made between A and E.  The raw log shows B and I transactions, but
> > apparently no processing is completing.  I suspect that nothing is getting
> > scheduled.
> > 
> > I am at a loss as to where to begin to track this one down.  The failure
> > is spontaneous and apparently not triggered by any readily distinguishable
> > inputs or pattern of inputs.  The behavior smells a bit of resource limits
> > or process synchronization problems, but there is not real evidence for
> > either being the root cause.   I am not sure what monitoring I should be 
> > doing to help locate the source of the problem.
> > 
> > Has anyone seen seen a similar problem?  Any advice as to how to proceed?
> 
> Threads are hanging. You should install my DeadlockDebugger and track 
> down where the hung threads are blocked at.
> 
>  From the description I'd wager that you'll find your threads stuck in a 
> corner of the MySQL DA. In which case you'd have to find why it 
> deadlocks and find a fix.
> 
> Florent
> 
> 

--