[Zope] KeyErrors in Zope using ZEO

[Mike Patterson]

Hi,
    We've been experiencing two KeyError problems with Zope since moving 
to a ZEO environment at the beginning of the year.  On the server that 
the ZEO server resides we have 3 Zope instances running, and each of 
them gets a KeyError about once or twice a week individually.  
Restarting the offending instance takes care of the issue, but I was 
wondering if anyone has an idea of why we would be seeing these.

Thanks,
    Mike

The error that we get is:

2007-05-02T19:18:45 ERROR ZODB.Connection Couldn't load state for 0x1c2d6e
Traceback (most recent call last):
  File "/opt/Plone-2.5/lib/python/ZODB/Connection.py", line 732, in setstate
    self._setstate(obj)
  File "/opt/Plone-2.5/lib/python/ZODB/Connection.py", line 768, in 
_setstate
    p, serial = self._storage.load(obj._p_oid, self._version)
  File "/opt/Plone-2.5/lib/python/ZEO/ClientStorage.py", line 746, in load
    return self.loadEx(oid, version)[:2]
  File "/opt/Plone-2.5/lib/python/ZEO/ClientStorage.py", line 774, in loadEx
    self._cache.store(oid, ver, tid, None, data)
  File "/opt/Plone-2.5/lib/python/ZEO/cache.py", line 293, in store
    self.fc.add(o)
  File "/opt/Plone-2.5/lib/python/ZEO/cache.py", line 980, in add
    available = self._makeroom(size)
  File "/opt/Plone-2.5/lib/python/ZEO/cache.py", line 915, in _makeroom
    size, e = self.filemap.pop(ofs)
KeyError: 19826745

--------------------------------------------------------------
Each Zope instance has a configuration file containing this:

<zodb_db main>
    # Main FileStorage database
  <zeoclient>
    server localhost:8100
    storage 1
    name zeostorage
    var $INSTANCE/var
  </zeoclient>
        pool-size 10
        cache-size 10000
    mount-point /
</zodb_db>

<zodb_db temporary>
    # Temporary storage database (for sessions)
    mount-point /temp_folder
    container-class Products.TemporaryFolder.TemporaryContainer
        pool-size 10
    <zeoclient>
                server localhost:8100
                storage temp
                name temp
                var $INSTANCE/var
        </zeoclient>
</zodb_db>

--------------------------------------------------------------
And the zeo.conf file looks like:

%define INSTANCE /opt/Plone-2.5/zeocluster/server

<zeo>
  address 8100
  read-only false
  invalidation-queue-size 1000
  # pid-filename $INSTANCE/var/ZEO.pid
  # monitor-address PORT
  # transaction-timeout SECONDS
</zeo>

<filestorage 1>
  path $INSTANCE/var/Data.fs
</filestorage>

%import tempstorage
<temporarystorage temp>
        name temporary storage
</temporarystorage>

<eventlog>
  level info
  <logfile>
    path $INSTANCE/log/zeo.log
  </logfile>
</eventlog>

<runner>
  program $INSTANCE/bin/runzeo
  socket-name $INSTANCE/var/zeo.zdsock
  daemon true
  forever false
  backoff-limit 10
  exit-codes 0, 2
  directory $INSTANCE
  default-to-interactive true
  user plone
  python /opt/Plone-2.5/Python-2.4.3/bin/python
  zdrun /opt/Plone-2.5/lib/python/zdaemon/zdrun.py

  # This logfile should match the one in the zeo.conf file.
  # It is used by zdctl's logtail command, zdrun/zdctl doesn't write it.
  logfile $INSTANCE/log/zeo.log
</runner>
--------------------------------------------------------------

Also, on a different server that connects to the same ZEO server, 
occasionally upon startup we'll be greeted by the following error 
error.  The line of code that causes the error is REQUEST.SESSION.get

2007-05-11T09:34:20 ERROR Zope.SiteErrorLog 
https://www.ltgc.com/Expert/ltgc.com/ltgc.com/page
Traceback (innermost last):
  Module ZPublisher.Publish, line 115, in publish
  Module ZPublisher.mapply, line 88, in mapply
  Module ZPublisher.Publish, line 41, in call_object
  Module Shared.DC.Scripts.Bindings, line 311, in __call__
  Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec
  Module Products.PythonScripts.PythonScript, line 325, in _exec
  Module None, line 3, in page
   - <PythonScript at /page used for /Expert/ltgc.com>
   - Line 3
  Module Products.ExternalMethod.ExternalMethod, line 225, in __call__
   - __traceback_info__: ((<ATFolder at /Expert/ltgc.com used for 
/Expert/ltgc.com>, '0Mdys-DaqxT
SIoTxC9_IuCETrPXoHGO-XtKcH9JMpwzkYNjKdUb7jLl_574agJt5RoVrBopnWX-meqaV7CnjNX4rxVFGQjMVj2knwS05zf_Z
Dv3BrFnagaO1s9QgY3OB9xmgF_otwtEP8wTwNNg9pE3qPmDjlq0FERlEPcj1q9xB8ZYd-3OWMaRkq_XrrSGn0BP0cXV3GexAl
Km_74g0OA'), {}, None)
  Module /opt/Plone-2.5/zeocluster/client1/Extensions/wwizparser.py, 
line 15, in redirect
  Module Products.CMFCore.PortalContent, line 108, in __call__
  Module Products.ExpertMenu.ExpertMenu, line 252, in view
  Module ZPublisher.HTTPRequest, line 1217, in __getattr__
  Module ZPublisher.HTTPRequest, line 1177, in get
  Module Products.Sessions.SessionDataManager, line 96, in getSessionData
  Module Products.Sessions.SessionDataManager, line 183, in 
_getSessionDataObject
  Module Products.Transience.Transience, line 843, in new_or_existing
  Module Products.Transience.Transience, line 493, in get
  Module Products.Transience.Transience, line 330, in _move_item
KeyError: 1178897660