[ZODB-Dev] Very Weird Behaviour :-(

[Chris Withers]

Hi,

This is all with Zope 2.6.1, Python 2.1.3 and ZEO 2.0.1b1 and the minor patch in 
the ps. There's also DBTab in play and 'Content' might well be a mounted DB, 
although I would expected it to be called 'content' so I'm a bit sceptical.

I awoke to a customer complaint that their site wasn't responding. On going 
there, it was indeed not responding. ssh'ed to the box to find one process using 
about 1GB of memory. Turned out to be one of the threads on the web-serving ZEO 
client.

Zope's stop script didn't work, so I had to kill -9 one of the worker threads 
for the web client to die. (This on it's own seems to be quite a common pattern, 
why is that?)

Now for the different bit, in the logs of the web client, I found lots of these:

2003-06-09T08:13:56 ERROR(200) ZODB Couldn't load state for 
'\x00\x00\x00\x00\x00\x12\x1d\x92'
Traceback (innermost last):
   File /usr/local/zope/2.6.1_rshl/lib/python/ZODB/Connection.py, line 509, in 
setstate
   File /usr/local/zope/2.6.1_rshl/lib/python/ZEO/ClientStorage.py, line 524, in 
load
     (Object: Content)
   File /usr/local/zope/2.6.1_rshl/lib/python/ZEO/ServerStub.py, line 73, in zeoLoad
   File /usr/local/zope/2.6.1_rshl/lib/python/ZEO/zrpc/connection.py, line 322, 
in call
POSKeyError: 0000000000121d92

Identical OIDs in all the errors...

Now, I don't know if this was a symptom of the memory bloat or the cause, but 
these errors started at pretty much the same time as the server was first 
reported as being unresponsive...

Interesting to note that the storage server appears to have survived unscathed 
throughout this.

So, I'm left rather nervous and wondering:

1. What caused the memory bloat, which apparently arrived out of the blue?

2. What those POSKeyErrors meanand should I be worried about them?

Any ideas? zodb-dev seemed the right place as I suspect this is a ZODB problem 
at its base...

cheers,

Chris

ps:

diff -r 2.6.1/lib/python/ZPublisher/HTTPResponse.py 
2.6.1_rshl/lib/python/ZPublisher/HTTPResponse.py
147a148
 >     _locked_body = 0
200c201
<     def setStatus(self, status, reason=None):
---
 >     def setStatus(self, status, reason=None, lock=None):
227a229,231
 >         # lock the status if we're told to
 >         if lock:
 >              self._locked_status = 1
262c266,267
<                 r'(iso[-_]8859[-_]1(:1987)?)))?$',re.I).match
---
 >                 r'(iso[-_]8859[-_]1(:1987)?)))?$',re.I).match,
 >                 lock=None
278a284,289
 >         # allow locking of the body in the same way as the status
 >         if self._locked_body:
 >             return
 >         elif lock:
 >             self._locked_body = 1
 >