[Zope] ZEO disconnects, Zope auto restarts (via zopectl)

[Dennis Allison]

Paul, 

Thanks for the assist.  Comments on your comments interlinearly below.

I have increased cache and other resources to see what the impact will be. 



On Fri, 3 Feb 2006, Paul Winkler wrote:

> On Fri, Feb 03, 2006 at 01:00:45AM -0800, Dennis Allison wrote:
> > 
> > Zope 2.9.0
> > 
> > We are seeing spontaneous restarts of Zope with no indication in any of 
> > the standard Zope logs.  Looking at the ZEO log indicates that the 
> > restarts of Zope are due to a lost connection between Zope & ZEO but 
> > with no other information.  The logging level is set at the distribution 
> > default (INFO).
> 
> Are you *sure* that is the cause, rather than the effect?  

	No, I am not and there's nothing in the logs which hints at why
	it restarted.  We are running under load.  The failures are 
	silent.  We do have a fairly high rate of conflict errors (which 
	all get resolved finally!).

> 
> If zope restarts for any reason, I'd expect the zeo log to 
> show a disconnect and reconnect as a result.
> 
> Check the clocks on your zope and zeo boxes and make sure the timing of
> events in your logs is really what you think it is.  (Systems that
> aren't running ntpd are the bane of my existence...)

	Timing correlates to the second.  Zope and ZEO live on the 
	same physical box.

> 
> Wild guess: Any chance your Zope process is running out of
> memory?  I've had that on several occasions, when some
> naively-written software attempts to do something huge in
> memory that should really use a temp file on disk.
> (Zope itself used to have some code like that in the FTP
> server, don't know if it still does.)
> 
	I doubt if I am hitting a limit.  The box has nearly 8GB of 
	memory most of which (6GB) is used by linux as a cache.
	No messages in the logs.

> I discovered this by looking  in /var/log/messages.
> At least on linux, the kernel will log something there
> when it kills a process that consumes all available memory.
> 
> > We are running a fairly vanilla setup, excerpted below:
> > [snip...] 
> 
> Unrelated to your problem, and maybe you know this, but depending on the
> size of your storage, I'd consider increasing the zeo client cache size.
> It's a disk cache and you can safely make it huge.  But if you don't see
> "cache flipping" messages in your event log, it may not matter.
> 
	Done, but I cannot report on the effect.