KeyErrors in Zope using ZEO
Hi, We've been experiencing two KeyError problems with Zope since moving to a ZEO environment at the beginning of the year. On the server that the ZEO server resides we have 3 Zope instances running, and each of them gets a KeyError about once or twice a week individually. Restarting the offending instance takes care of the issue, but I was wondering if anyone has an idea of why we would be seeing these. Thanks, Mike The error that we get is: 2007-05-02T19:18:45 ERROR ZODB.Connection Couldn't load state for 0x1c2d6e Traceback (most recent call last): File "/opt/Plone-2.5/lib/python/ZODB/Connection.py", line 732, in setstate self._setstate(obj) File "/opt/Plone-2.5/lib/python/ZODB/Connection.py", line 768, in _setstate p, serial = self._storage.load(obj._p_oid, self._version) File "/opt/Plone-2.5/lib/python/ZEO/ClientStorage.py", line 746, in load return self.loadEx(oid, version)[:2] File "/opt/Plone-2.5/lib/python/ZEO/ClientStorage.py", line 774, in loadEx self._cache.store(oid, ver, tid, None, data) File "/opt/Plone-2.5/lib/python/ZEO/cache.py", line 293, in store self.fc.add(o) File "/opt/Plone-2.5/lib/python/ZEO/cache.py", line 980, in add available = self._makeroom(size) File "/opt/Plone-2.5/lib/python/ZEO/cache.py", line 915, in _makeroom size, e = self.filemap.pop(ofs) KeyError: 19826745 -------------------------------------------------------------- Each Zope instance has a configuration file containing this: <zodb_db main> # Main FileStorage database <zeoclient> server localhost:8100 storage 1 name zeostorage var $INSTANCE/var </zeoclient> pool-size 10 cache-size 10000 mount-point / </zodb_db> <zodb_db temporary> # Temporary storage database (for sessions) mount-point /temp_folder container-class Products.TemporaryFolder.TemporaryContainer pool-size 10 <zeoclient> server localhost:8100 storage temp name temp var $INSTANCE/var </zeoclient> </zodb_db> -------------------------------------------------------------- And the zeo.conf file looks like: %define INSTANCE /opt/Plone-2.5/zeocluster/server <zeo> address 8100 read-only false invalidation-queue-size 1000 # pid-filename $INSTANCE/var/ZEO.pid # monitor-address PORT # transaction-timeout SECONDS </zeo> <filestorage 1> path $INSTANCE/var/Data.fs </filestorage> %import tempstorage <temporarystorage temp> name temporary storage </temporarystorage> <eventlog> level info <logfile> path $INSTANCE/log/zeo.log </logfile> </eventlog> <runner> program $INSTANCE/bin/runzeo socket-name $INSTANCE/var/zeo.zdsock daemon true forever false backoff-limit 10 exit-codes 0, 2 directory $INSTANCE default-to-interactive true user plone python /opt/Plone-2.5/Python-2.4.3/bin/python zdrun /opt/Plone-2.5/lib/python/zdaemon/zdrun.py # This logfile should match the one in the zeo.conf file. # It is used by zdctl's logtail command, zdrun/zdctl doesn't write it. logfile $INSTANCE/log/zeo.log </runner> -------------------------------------------------------------- Also, on a different server that connects to the same ZEO server, occasionally upon startup we'll be greeted by the following error error. The line of code that causes the error is REQUEST.SESSION.get 2007-05-11T09:34:20 ERROR Zope.SiteErrorLog https://www.ltgc.com/Expert/ltgc.com/ltgc.com/page Traceback (innermost last): Module ZPublisher.Publish, line 115, in publish Module ZPublisher.mapply, line 88, in mapply Module ZPublisher.Publish, line 41, in call_object Module Shared.DC.Scripts.Bindings, line 311, in __call__ Module Shared.DC.Scripts.Bindings, line 348, in _bindAndExec Module Products.PythonScripts.PythonScript, line 325, in _exec Module None, line 3, in page - <PythonScript at /page used for /Expert/ltgc.com> - Line 3 Module Products.ExternalMethod.ExternalMethod, line 225, in __call__ - __traceback_info__: ((<ATFolder at /Expert/ltgc.com used for /Expert/ltgc.com>, '0Mdys-DaqxT SIoTxC9_IuCETrPXoHGO-XtKcH9JMpwzkYNjKdUb7jLl_574agJt5RoVrBopnWX-meqaV7CnjNX4rxVFGQjMVj2knwS05zf_Z Dv3BrFnagaO1s9QgY3OB9xmgF_otwtEP8wTwNNg9pE3qPmDjlq0FERlEPcj1q9xB8ZYd-3OWMaRkq_XrrSGn0BP0cXV3GexAl Km_74g0OA'), {}, None) Module /opt/Plone-2.5/zeocluster/client1/Extensions/wwizparser.py, line 15, in redirect Module Products.CMFCore.PortalContent, line 108, in __call__ Module Products.ExpertMenu.ExpertMenu, line 252, in view Module ZPublisher.HTTPRequest, line 1217, in __getattr__ Module ZPublisher.HTTPRequest, line 1177, in get Module Products.Sessions.SessionDataManager, line 96, in getSessionData Module Products.Sessions.SessionDataManager, line 183, in _getSessionDataObject Module Products.Transience.Transience, line 843, in new_or_existing Module Products.Transience.Transience, line 493, in get Module Products.Transience.Transience, line 330, in _move_item KeyError: 1178897660
On 5/12/07, Mike Patterson <mpatterson@ltgc.com> wrote:
Hi, We've been experiencing two KeyError problems with Zope since moving to a ZEO environment at the beginning of the year. On the server that the ZEO server resides we have 3 Zope instances running, and each of them gets a KeyError about once or twice a week individually. Restarting the offending instance takes care of the issue, but I was wondering if anyone has an idea of why we would be seeing these.
Thanks, Mike
The error that we get is:
2007-05-02T19:18:45 ERROR ZODB.Connection Couldn't load state for 0x1c2d6e Traceback (most recent call last): File "/opt/Plone-2.5/lib/python/ZODB/Connection.py", line 732, in setstate self._setstate(obj) File "/opt/Plone-2.5/lib/python/ZODB/Connection.py", line 768, in _setstate p, serial = self._storage.load(obj._p_oid, self._version) File "/opt/Plone-2.5/lib/python/ZEO/ClientStorage.py", line 746, in load return self.loadEx(oid, version)[:2] File "/opt/Plone-2.5/lib/python/ZEO/ClientStorage.py", line 774, in loadEx self._cache.store(oid, ver, tid, None, data) File "/opt/Plone-2.5/lib/python/ZEO/cache.py", line 293, in store self.fc.add(o) File "/opt/Plone-2.5/lib/python/ZEO/cache.py", line 980, in add available = self._makeroom(size) File "/opt/Plone-2.5/lib/python/ZEO/cache.py", line 915, in _makeroom size, e = self.filemap.pop(ofs) KeyError: 19826745
looks like there may be problems with the cache file. try not using the cache file and see whether that helps by removing name zeostrorage on each clients. hth -- http://myzope.kedai.com.my - my-zope org
Removing the 'name zeostorage' didn't have any affect on the issue. I was able to fix the KeyErrors by increasing the cache-size in the zeo client configuration part of the zope.conf though. However, I'm still experiencing intermittent KeyErrors with the session stuff. The line of code that always breaks is "session = REQUEST.SESSION" and it has happened everywhere in my code that I use that. Is there a better way to access the session that I should be using? The traceback ends like this: * Module ZPublisher.HTTPRequest, line 1217, in __getattr__ * Module ZPublisher.HTTPRequest, line 1177, in get * Module Products.Sessions.SessionDataManager, line 96, in getSessionData * Module Products.Sessions.SessionDataManager, line 183, in _getSessionDataObject * Module Products.Transience.Transience, line 846, in new_or_existing * Module Products.Transience.Transience, line 452, in __setitem__ I'm now using the exact same configuration for sessions as detailed in http://www.zopelabs.com/cookbook/1061234337 except the zeo server is on a different box than the clients. Any ideas? Thanks, Mike Bakhtiar A Hamid wrote:
On 5/12/07, Mike Patterson <mpatterson@ltgc.com> wrote:
Hi, We've been experiencing two KeyError problems with Zope since moving to a ZEO environment at the beginning of the year. On the server that the ZEO server resides we have 3 Zope instances running, and each of them gets a KeyError about once or twice a week individually. Restarting the offending instance takes care of the issue, but I was wondering if anyone has an idea of why we would be seeing these.
Thanks, Mike
looks like there may be problems with the cache file. try not using the cache file and see whether that helps by removing name zeostrorage on each clients.
hth
However, I'm still experiencing intermittent KeyErrors with the session stuff. The line of code that always breaks is "session = REQUEST.SESSION" and it has happened everywhere in my code that I use that. Is there a better way to access the session that I should be using? I think there is no better way although there are different ways. Maybe you should think again if you really have to use sessions shared between zeo clients. If so then you should possibly try Faster (another session implementation) that is said to be better especially in such setup with ZEO.
-- Maciej Wisniowski
participants (3)
-
Bakhtiar A Hamid -
Maciej Wisniowski -
Mike Patterson