[Zope] Zope 2.8.4 strange behavior

[Dennis Allison]

Thanks Paul.

I checked and did not find any "Client disconnected" entries in the 
event log so this may not be the cause of what I am seeing.  I will 
bump the log level to ALL (which I think includes BLATHER) when I restart 
things tonight.

We use a remote ZEO but at the moment Zope and ZEO share the same machine.

On Sun, 27 Nov 2005, Paul Winkler wrote:

> On Sun, Nov 27, 2005 at 05:13:21PM -0800, Dennis Allison wrote:
> > Examination of the raw trace log shows that Zope is continuing to accept
> > requests, but nothing getting done.  The raw log date-stamps four internal
> > states for each transaction.  The states are Begin (B), Input (I),
> > action (A), and End (E).  Inputs are gathered between B and I, outputs is
> > made between A and E.  The raw log shows B and I transactions, but
> > apparently no processing is completing.  I suspect that nothing is getting
> > scheduled.
> 
> I've seen the same symptoms a number of times recently with zope 2.7.x.
> In our case, it seems to be related to ZEO. Zope seems to have lost
> its connection to ZEO but doesn't realize it somehow.  My theory is
> that the symptom starts when all worker threads are waiting for objects
> that aren't in the ZEO client cache, so they're all waiting on ZEO
> requests.  Meanwhile, requests keep piling up in the queue.
> These are all the B and I lines you noticed.
> 
> Eventually (10 minutes or so) one of these ZEO requests times out, and 
> Zope then realizes it's lost its ZEO connection and successfully reconnects.
> This shows up in the Zope event log as a series of ZEO "Client
> disconnected" errors, and immediately thereafter we see a successful
> reconnect in the ZEO server's log. 
> 
> I have no idea what is causing the loss of connection in the first
> place, or why it takes Zope so long to realize there's a problem.
> Nothing in the logs so far gives me any clue.  I've bumped up our log
> levels to BLATHER and hopefully next time this happens I'll get some
> more to work with and see if there's anythign I can do about it.
> 
> It may be network issues between our Zope and Zeo boxes, which
> are on different subnets with a firewall between them, over which
> I have no control.
>  
> 

--