[Zope-dev] CoreSession, ZEO and BerkeleyStorage =CorruptedDataError

[Chris McDonough]

> I knew having a 'big brain' on it would help.

That's gotta be yours, because it's not mine.  ;-)

> Your comments made me look again at the script (in Extensions) that
creates
> the session storage. This is used by ExternalMount and looks like this
>
> from ZODB.BerkeleyStorage import SleepyStorage
> def createSessionDB():
>     return ZODB.DB(SleepyStorage('/home/nnle/ZEO_FARM/SessionData.fs'))
>
> Whereas what I had was
> from ZODB.FileStorage import FileStorage
> def createDB():
>     return ZODB.DB(FileStorage('/home/nnle/ZEO_FARM/SessionData.fs'))
>
> Ie not good
>
> Now I can start my ZEO cluster with
> % zctl.py start_zeo
> And then cd to the other instances of my Zclients and do
> % zctl.py start
>
> A netstat -a | grep LISTEN shows the ZEO server and the clients running.
>
> One thing: I find that if SessionData.fs is *not there* when I start up my
> ZEO server with
> % zctl.py start_zeo
> Then it is created anyhow.

Uh huh.  This is the default behavior of FileStorage.

> I find that when I start ZEO server several __db.* files are created
> (__db.001 .. __db.005) in the var directory of that Instance.

Hmm.  I dunno why... these are Berkeley files (but you knew that).

> However, when I start my ZClients these files are created in the var
> directory of *that* Instance as well. *They* are the files that are
changed
> with CST activity. The file /home/nnle/ZEO_FARM/SessionData.fs shows no
> activity at all. A clue methinks?

Maybe.  But the corrupted data error points at the filestorage.  So
something was trying to read or write to it at some point?

FWIW, bsddb3Storage is available.  If I were doing this, I'd probably use it
instead of Ty Sarna's Berkeley Storage... particularly the Packless variant.
See http://www.zope.org/Products/bsddb3Storage .  It's based on BerkeleyDB
3.X instead of 1.X, so you'll need a new bsddb3 and PyBSDDB3.  Not really an
imperative to use one over the other, but the "packless" properties of the
Packless bsddb3Storage variant make it ideal for a session storage.

> Part of my zope.conf is
> ZEO['ZEO_STORAGES'] = {}
> ZEO['ZEO_STORAGES']['main'] = 'StorageConfig:main_storage'
> ZEO['ZEO_STORAGES']['session'] = 'StorageConfig:session_storage'
>
> And this is repeated for each Instance (InstanceHome .. InstanceHome5)

OK, I'm not familiar with this idiom.

> My custom_zodb.py is
> import ZEO.ClientStorage, os, string
>
> host=os.environ.get('ZEO_SERVER_NAME', '')
> port=string.atoi(os.environ['ZEO_SERVER_PORT'])
>
> Storage=ZEO.ClientStorage.ClientStorage(
>     (host, port), name='ZEO eGuides', storage='main')
>
> (there's no mention of the SessionData.fs here, should there be?)

No.. custom_zodb.py need only define the "main" storage.

> And StorageConfig in lib/python/ZEO is:
> import ZODB.FileStorage, ZODB.BerkeleyStorage
> # Change the filenames to suit your system
> main_storage =
> ZODB.FileStorage.FileStorage('/home/nnle/InstanceHome/var/Data.fs')
> session_storage =
> ZODB.BerkeleyStorage.SleepyStorage('/home/nnle/ZEO_FARM/SessionData.fs')

Geez... I don't know about StorageConfig either.  Is this a ZEO config file?
I have no experience setting up a ZEO server which serves out more than one
storage.

That said, if this file gets run, you're creating a file named
"SessionData.fs" somewhere that might stomp on the *other* SessionData.fs
that you've created in the external method.  I actually don't see why you
need to.. wait.. actually, here's the thing.  The "SleepyStorage" that is
served out of ZEO needs to be communicated through via a ZEO ClientStorage.
So instead of this in your exmount external method:

from ZODB.BerkeleyStorage import SleepyStorage
def createSessionDB():
     return ZODB.DB(SleepyStorage('/home/nnle/ZEO_FARM/SessionData.fs'))

do something like:

from ZEO.ClientStorage import ClientStorage
def createSessionDB():
     return ZODB.DB(ClientStorage([options]))

(Note that I don't know the "options" above, and I'm punting on it.  ;-)  In
any case, I *think* this is what you're trying to do.  You want to share a
BerkeleyStorage between many ZEO clients for sessioning.  The only way to do
this is to use a ZEO ClientStorage, so the clients shouldn't know anything
about SleepyStorage.  Instead, they should know about ClientStorage, and the
ZEO server is the only thing that should know about SleepyStorage.

Huh.  I guess I should revise the docs.  I don't go into this much detail in
them.

> The ZEO stuff works really fine without CST, but I have grandiose plans
for
> that, so I'm *really* loathe to lose it.
>
> Tell me if I'm being a pain and I'll go lurk again...

Nah, you're just trying to use the stuff as if it actually *worked*. ;-)