CoreSession, ZEO and BerkeleyStorage = CorruptedDataError
I'm pooped. I'll apologise in advance that there doesn't seem enough info. In this email, but the setup is so complex that to put it all in would exceed the mailing list message size :) (and to tell the truth, I'm hoping that someone can point a finger at a likely spot). I really like CST, but if this doesn't work, I'm thinking about using SQLSession as I'm really getting out of my depth here. Any thoughts would be really appreciated. I've got a ZEO system set up using this HowTo (which is very good BTW) 1) http://dev.zope.org/Members/dshaw/AdvancedSiteSetup Then I installed BerkeleyStorage from a combination of 2) http://www.zope.org/Members/Duncan/BerkeleyStorage and 3) http://www.zope.org/Members/andyd/zopeberkeleydb Then following the instructions at 4) http://www.zope.org/Members/randy/ZEO-Sessions I set up a ZEO based CoreSessionTracking system and set up several client zope systems (from (1)) and had them accessed 'randomly' using Apache rewriterules. I managed to get the system up and running. But once I 'invalidate' a CoreSession, I get this error message; Error Type: CorruptedDataError Error Value: data_container/manage_container Traceback: <!-- Traceback (innermost last): File /home/nnle/Zope-2.4.0-src/lib/python/ZPublisher/Publish.py, line 223, in publish_module File /home/nnle/Zope-2.4.0-src/lib/python/ZPublisher/Publish.py, line 187, in publish File /home/nnle/Zope-2.4.0-src/lib/python/Zope/__init__.py, line 226, in zpublisher_exception_hook (Object: RoleManager) File /home/nnle/Zope-2.4.0-src/lib/python/ZPublisher/Publish.py, line 171, in publish File /home/nnle/Zope-2.4.0-src/lib/python/ZPublisher/mapply.py, line 160, in mapply (Object: logout) File /home/nnle/Zope-2.4.0-src/lib/python/ZPublisher/Publish.py, line 112, in call_object (Object: logout) File /home/nnle/Zope-2.4.0-src/lib/python/Products/UserDb/UserDb.py, line 334, in logout (Object: RoleManager) File /home/nnle/Zope-2.4.0-src/lib/python/OFS/DTMLMethod.py, line 194, in __call__ (Object: docLogout) File /home/nnle/Zope-2.4.0-src/lib/python/DocumentTemplate/DT_String.py, line 544, in __call__ (Object: docLogout) File /home/nnle/Zope-2.4.0-src/lib/python/DocumentTemplate/DT_Util.py, line 230, in eval (Object: session_info.getSessionData().invalidate()) (Info: session_info) File <string>, line 2, in f (Object: guarded_getattr) File /home/nnle/Zope-2.4.0-src/lib/python/Products/CoreSessionTracking/SessionDat aManager.py, line 228, in getSessionData (Object: session_info) File /home/nnle/Zope-2.4.0-src/lib/python/Products/CoreSessionTracking/SessionDat aManager.py, line 323, in _getSessionDataObject (Object: session_info) File /home/nnle/Zope-2.4.0-src/lib/python/Products/CoreSessionTracking/AutoExpire Mapping.py, line 235, in get (Object: session_data_container) File /home/nnle/Zope-2.4.0-src/lib/python/Products/CoreSessionTracking/AutoExpire Mapping.py, line 221, in __getitem__ (Object: session_data_container) File /home/nnle/Zope-2.4.0-src/lib/python/Products/CoreSessionTracking/AutoExpire Mapping.py, line 154, in _getCurrentBucket (Object: session_data_container) File /home/nnle/Zope-2.4.0-src/lib/python/ZODB/Connection.py, line 519, in setstate File /home/nnle/Zope-2.4.0-src/lib/python/ZODB/FileStorage.py, line 587, in load (Object: /home/nnle/ZEO_FARM/SessionData.fs) File /home/nnle/Zope-2.4.0-src/lib/python/ZODB/FileStorage.py, line 568, in _load (Object: /home/nnle/ZEO_FARM/SessionData.fs) CorruptedDataError: (see above) --> Versions Solaris 8 Python 2.1 Zope 2.4.0 src Berkeley3.2 CoreSessionTracking 0.8 -- Dr Tony McDonald, Assistant Director, FMCC, http://www.fmcc.org.uk/ The Medical School, Newcastle University Tel: +44 191 243 6140 A Zope list for UK HE/FE http://www.fmcc.org.uk/mailman/listinfo/zope
I'm confused... where does BerkeleyStorage come in here? It appears that the data is being saved to a FileStorage named "SessionStorage". In any case, this is sort of besides the point... I unfortunately have no idea what the problem could be. CST does not use the ZODB in any particularly taxing way. About the best I can do is recommend that you try to put the ZEO server that serves the session data on another system and see if the errors continue. If they do, and the problem is reproducible, we can fix it. - C Tony McDonald wrote:
I'm pooped. I'll apologise in advance that there doesn't seem enough info. In this email, but the setup is so complex that to put it all in would exceed the mailing list message size :) (and to tell the truth, I'm hoping that someone can point a finger at a likely spot).
I really like CST, but if this doesn't work, I'm thinking about using SQLSession as I'm really getting out of my depth here.
Any thoughts would be really appreciated.
I've got a ZEO system set up using this HowTo (which is very good BTW) 1) http://dev.zope.org/Members/dshaw/AdvancedSiteSetup
Then I installed BerkeleyStorage from a combination of 2) http://www.zope.org/Members/Duncan/BerkeleyStorage and 3) http://www.zope.org/Members/andyd/zopeberkeleydb
Then following the instructions at 4) http://www.zope.org/Members/randy/ZEO-Sessions
I set up a ZEO based CoreSessionTracking system and set up several client zope systems (from (1)) and had them accessed 'randomly' using Apache rewriterules.
I managed to get the system up and running. But once I 'invalidate' a CoreSession, I get this error message;
Error Type: CorruptedDataError Error Value: data_container/manage_container
Traceback: <!-- Traceback (innermost last): File /home/nnle/Zope-2.4.0-src/lib/python/ZPublisher/Publish.py, line 223, in publish_module File /home/nnle/Zope-2.4.0-src/lib/python/ZPublisher/Publish.py, line 187, in publish File /home/nnle/Zope-2.4.0-src/lib/python/Zope/__init__.py, line 226, in zpublisher_exception_hook (Object: RoleManager) File /home/nnle/Zope-2.4.0-src/lib/python/ZPublisher/Publish.py, line 171, in publish File /home/nnle/Zope-2.4.0-src/lib/python/ZPublisher/mapply.py, line 160, in mapply (Object: logout) File /home/nnle/Zope-2.4.0-src/lib/python/ZPublisher/Publish.py, line 112, in call_object (Object: logout) File /home/nnle/Zope-2.4.0-src/lib/python/Products/UserDb/UserDb.py, line 334, in logout (Object: RoleManager) File /home/nnle/Zope-2.4.0-src/lib/python/OFS/DTMLMethod.py, line 194, in __call__ (Object: docLogout) File /home/nnle/Zope-2.4.0-src/lib/python/DocumentTemplate/DT_String.py, line 544, in __call__ (Object: docLogout) File /home/nnle/Zope-2.4.0-src/lib/python/DocumentTemplate/DT_Util.py, line 230, in eval (Object: session_info.getSessionData().invalidate()) (Info: session_info) File <string>, line 2, in f (Object: guarded_getattr) File /home/nnle/Zope-2.4.0-src/lib/python/Products/CoreSessionTracking/SessionDat aManager.py, line 228, in getSessionData (Object: session_info) File /home/nnle/Zope-2.4.0-src/lib/python/Products/CoreSessionTracking/SessionDat aManager.py, line 323, in _getSessionDataObject (Object: session_info) File /home/nnle/Zope-2.4.0-src/lib/python/Products/CoreSessionTracking/AutoExpire Mapping.py, line 235, in get (Object: session_data_container) File /home/nnle/Zope-2.4.0-src/lib/python/Products/CoreSessionTracking/AutoExpire Mapping.py, line 221, in __getitem__ (Object: session_data_container) File /home/nnle/Zope-2.4.0-src/lib/python/Products/CoreSessionTracking/AutoExpire Mapping.py, line 154, in _getCurrentBucket (Object: session_data_container) File /home/nnle/Zope-2.4.0-src/lib/python/ZODB/Connection.py, line 519, in setstate File /home/nnle/Zope-2.4.0-src/lib/python/ZODB/FileStorage.py, line 587, in load (Object: /home/nnle/ZEO_FARM/SessionData.fs) File /home/nnle/Zope-2.4.0-src/lib/python/ZODB/FileStorage.py, line 568, in _load (Object: /home/nnle/ZEO_FARM/SessionData.fs) CorruptedDataError: (see above)
-->
Versions Solaris 8 Python 2.1 Zope 2.4.0 src Berkeley3.2 CoreSessionTracking 0.8
-- Chris McDonough Zope Corporation http://www.zope.org http://www.zope.com "Killing hundreds of birds with thousands of stones"
On 3/9/01 10:00 pm, "Chris McDonough" <chrism@digicool.com> wrote:
I'm confused... where does BerkeleyStorage come in here? It appears that the data is being saved to a FileStorage named "SessionStorage".
In any case, this is sort of besides the point... I unfortunately have no idea what the problem could be. CST does not use the ZODB in any particularly taxing way. About the best I can do is recommend that you try to put the ZEO server that serves the session data on another system and see if the errors continue. If they do, and the problem is reproducible, we can fix it.
- C
I knew having a 'big brain' on it would help. Your comments made me look again at the script (in Extensions) that creates the session storage. This is used by ExternalMount and looks like this from ZODB.BerkeleyStorage import SleepyStorage def createSessionDB(): return ZODB.DB(SleepyStorage('/home/nnle/ZEO_FARM/SessionData.fs')) Whereas what I had was from ZODB.FileStorage import FileStorage def createDB(): return ZODB.DB(FileStorage('/home/nnle/ZEO_FARM/SessionData.fs')) Ie not good Now I can start my ZEO cluster with % zctl.py start_zeo And then cd to the other instances of my Zclients and do % zctl.py start A netstat -a | grep LISTEN shows the ZEO server and the clients running. One thing: I find that if SessionData.fs is *not there* when I start up my ZEO server with % zctl.py start_zeo Then it is created anyhow. I find that when I start ZEO server several __db.* files are created (__db.001 .. __db.005) in the var directory of that Instance. However, when I start my ZClients these files are created in the var directory of *that* Instance as well. *They* are the files that are changed with CST activity. The file /home/nnle/ZEO_FARM/SessionData.fs shows no activity at all. A clue methinks? Part of my zope.conf is ZEO['ZEO_STORAGES'] = {} ZEO['ZEO_STORAGES']['main'] = 'StorageConfig:main_storage' ZEO['ZEO_STORAGES']['session'] = 'StorageConfig:session_storage' And this is repeated for each Instance (InstanceHome .. InstanceHome5) My custom_zodb.py is import ZEO.ClientStorage, os, string host=os.environ.get('ZEO_SERVER_NAME', '') port=string.atoi(os.environ['ZEO_SERVER_PORT']) Storage=ZEO.ClientStorage.ClientStorage( (host, port), name='ZEO eGuides', storage='main') (there's no mention of the SessionData.fs here, should there be?) And StorageConfig in lib/python/ZEO is: import ZODB.FileStorage, ZODB.BerkeleyStorage # Change the filenames to suit your system main_storage = ZODB.FileStorage.FileStorage('/home/nnle/InstanceHome/var/Data.fs') session_storage = ZODB.BerkeleyStorage.SleepyStorage('/home/nnle/ZEO_FARM/SessionData.fs') The ZEO stuff works really fine without CST, but I have grandiose plans for that, so I'm *really* loathe to lose it. Tell me if I'm being a pain and I'll go lurk again... tone -- Dr Tony McDonald, Assistant Director, FMCC, http://www.fmcc.org.uk/ The Medical School, Newcastle University Tel: +44 191 243 6140 A Zope list for UK HE/FE http://www.fmcc.org.uk/mailman/listinfo/zope
I knew having a 'big brain' on it would help.
That's gotta be yours, because it's not mine. ;-)
Your comments made me look again at the script (in Extensions) that creates the session storage. This is used by ExternalMount and looks like this
from ZODB.BerkeleyStorage import SleepyStorage def createSessionDB(): return ZODB.DB(SleepyStorage('/home/nnle/ZEO_FARM/SessionData.fs'))
Whereas what I had was from ZODB.FileStorage import FileStorage def createDB(): return ZODB.DB(FileStorage('/home/nnle/ZEO_FARM/SessionData.fs'))
Ie not good
Now I can start my ZEO cluster with % zctl.py start_zeo And then cd to the other instances of my Zclients and do % zctl.py start
A netstat -a | grep LISTEN shows the ZEO server and the clients running.
One thing: I find that if SessionData.fs is *not there* when I start up my ZEO server with % zctl.py start_zeo Then it is created anyhow.
Uh huh. This is the default behavior of FileStorage.
I find that when I start ZEO server several __db.* files are created (__db.001 .. __db.005) in the var directory of that Instance.
Hmm. I dunno why... these are Berkeley files (but you knew that).
However, when I start my ZClients these files are created in the var directory of *that* Instance as well. *They* are the files that are changed with CST activity. The file /home/nnle/ZEO_FARM/SessionData.fs shows no activity at all. A clue methinks?
Maybe. But the corrupted data error points at the filestorage. So something was trying to read or write to it at some point? FWIW, bsddb3Storage is available. If I were doing this, I'd probably use it instead of Ty Sarna's Berkeley Storage... particularly the Packless variant. See http://www.zope.org/Products/bsddb3Storage . It's based on BerkeleyDB 3.X instead of 1.X, so you'll need a new bsddb3 and PyBSDDB3. Not really an imperative to use one over the other, but the "packless" properties of the Packless bsddb3Storage variant make it ideal for a session storage.
Part of my zope.conf is ZEO['ZEO_STORAGES'] = {} ZEO['ZEO_STORAGES']['main'] = 'StorageConfig:main_storage' ZEO['ZEO_STORAGES']['session'] = 'StorageConfig:session_storage'
And this is repeated for each Instance (InstanceHome .. InstanceHome5)
OK, I'm not familiar with this idiom.
My custom_zodb.py is import ZEO.ClientStorage, os, string
host=os.environ.get('ZEO_SERVER_NAME', '') port=string.atoi(os.environ['ZEO_SERVER_PORT'])
Storage=ZEO.ClientStorage.ClientStorage( (host, port), name='ZEO eGuides', storage='main')
(there's no mention of the SessionData.fs here, should there be?)
No.. custom_zodb.py need only define the "main" storage.
And StorageConfig in lib/python/ZEO is: import ZODB.FileStorage, ZODB.BerkeleyStorage # Change the filenames to suit your system main_storage = ZODB.FileStorage.FileStorage('/home/nnle/InstanceHome/var/Data.fs') session_storage = ZODB.BerkeleyStorage.SleepyStorage('/home/nnle/ZEO_FARM/SessionData.fs')
Geez... I don't know about StorageConfig either. Is this a ZEO config file? I have no experience setting up a ZEO server which serves out more than one storage. That said, if this file gets run, you're creating a file named "SessionData.fs" somewhere that might stomp on the *other* SessionData.fs that you've created in the external method. I actually don't see why you need to.. wait.. actually, here's the thing. The "SleepyStorage" that is served out of ZEO needs to be communicated through via a ZEO ClientStorage. So instead of this in your exmount external method: from ZODB.BerkeleyStorage import SleepyStorage def createSessionDB(): return ZODB.DB(SleepyStorage('/home/nnle/ZEO_FARM/SessionData.fs')) do something like: from ZEO.ClientStorage import ClientStorage def createSessionDB(): return ZODB.DB(ClientStorage([options])) (Note that I don't know the "options" above, and I'm punting on it. ;-) In any case, I *think* this is what you're trying to do. You want to share a BerkeleyStorage between many ZEO clients for sessioning. The only way to do this is to use a ZEO ClientStorage, so the clients shouldn't know anything about SleepyStorage. Instead, they should know about ClientStorage, and the ZEO server is the only thing that should know about SleepyStorage. Huh. I guess I should revise the docs. I don't go into this much detail in them.
The ZEO stuff works really fine without CST, but I have grandiose plans for that, so I'm *really* loathe to lose it.
Tell me if I'm being a pain and I'll go lurk again...
Nah, you're just trying to use the stuff as if it actually *worked*. ;-)
participants (3)
-
Chris McDonough -
Chris McDonough -
Tony McDonald