[Zope] Zope 2.8.4 strange behavior

[Paul Winkler]

On Sun, Nov 27, 2005 at 05:13:21PM -0800, Dennis Allison wrote:
> Examination of the raw trace log shows that Zope is continuing to accept
> requests, but nothing getting done.  The raw log date-stamps four internal
> states for each transaction.  The states are Begin (B), Input (I),
> action (A), and End (E).  Inputs are gathered between B and I, outputs is
> made between A and E.  The raw log shows B and I transactions, but
> apparently no processing is completing.  I suspect that nothing is getting
> scheduled.

I've seen the same symptoms a number of times recently with zope 2.7.x.
In our case, it seems to be related to ZEO. Zope seems to have lost
its connection to ZEO but doesn't realize it somehow.  My theory is
that the symptom starts when all worker threads are waiting for objects
that aren't in the ZEO client cache, so they're all waiting on ZEO
requests.  Meanwhile, requests keep piling up in the queue.
These are all the B and I lines you noticed.

Eventually (10 minutes or so) one of these ZEO requests times out, and 
Zope then realizes it's lost its ZEO connection and successfully reconnects.
This shows up in the Zope event log as a series of ZEO "Client
disconnected" errors, and immediately thereafter we see a successful
reconnect in the ZEO server's log. 

I have no idea what is causing the loss of connection in the first
place, or why it takes Zope so long to realize there's a problem.
Nothing in the logs so far gives me any clue.  I've bumped up our log
levels to BLATHER and hopefully next time this happens I'll get some
more to work with and see if there's anythign I can do about it.

It may be network issues between our Zope and Zeo boxes, which
are on different subnets with a firewall between them, over which
I have no control.
 
-- 

Paul Winkler
http://www.slinkp.com