Paul, Thanks for the assist. Comments on your comments interlinearly below. I have increased cache and other resources to see what the impact will be. On Fri, 3 Feb 2006, Paul Winkler wrote:
On Fri, Feb 03, 2006 at 01:00:45AM -0800, Dennis Allison wrote:
Zope 2.9.0
We are seeing spontaneous restarts of Zope with no indication in any of the standard Zope logs. Looking at the ZEO log indicates that the restarts of Zope are due to a lost connection between Zope & ZEO but with no other information. The logging level is set at the distribution default (INFO).
Are you *sure* that is the cause, rather than the effect?
No, I am not and there's nothing in the logs which hints at why it restarted. We are running under load. The failures are silent. We do have a fairly high rate of conflict errors (which all get resolved finally!).
If zope restarts for any reason, I'd expect the zeo log to show a disconnect and reconnect as a result.
Check the clocks on your zope and zeo boxes and make sure the timing of events in your logs is really what you think it is. (Systems that aren't running ntpd are the bane of my existence...)
Timing correlates to the second. Zope and ZEO live on the same physical box.
Wild guess: Any chance your Zope process is running out of memory? I've had that on several occasions, when some naively-written software attempts to do something huge in memory that should really use a temp file on disk. (Zope itself used to have some code like that in the FTP server, don't know if it still does.)
I doubt if I am hitting a limit. The box has nearly 8GB of memory most of which (6GB) is used by linux as a cache. No messages in the logs.
I discovered this by looking in /var/log/messages. At least on linux, the kernel will log something there when it kills a process that consumes all available memory.
We are running a fairly vanilla setup, excerpted below: [snip...]
Unrelated to your problem, and maybe you know this, but depending on the size of your storage, I'd consider increasing the zeo client cache size. It's a disk cache and you can safely make it huge. But if you don't see "cache flipping" messages in your event log, it may not matter.
Done, but I cannot report on the effect.