[Zope-dev] Re: sessions in the presence of conflicts

[Dennis Allison]

On Thu, 15 Dec 2005, Chris McDonough wrote:

> Trimmed zodb-dev off the cc list.
> 
> On Dec 15, 2005, at 2:24 PM, Dennis Allison wrote:
> 
> > The systems are running a Zope/ZEO combination with a store  
> > configuration
> > of:
> >
> > #
> >  <zodb_db temporary>
> >      # Temporary storage database (for sessions)
> >      <temporarystorage>
> >        name temporary storage for sessioning
> >      </temporarystorage>
> >      mount-point /temp_folder
> >      container-class Products.TemporaryFolder.TemporaryContainer
> >  </zodb_db>
> 
> OK, that's good.  Having a nonzeo session database is a good thing.
> >
> 
> > Although the connection to ZEO is via a network port, it runs on  
> > the same
> > physical hardware.
> >
> > TemporaryStorage is not transactional.  Does it need to be under MVCC?
> 
> I thought we sewed this one up. ;-)  It is indeed transactional.  See  
> our email conversations around 11/15/2005 and beyond.

**** That's what I thought as well, but I did not see any transaction
rollback mechanism.  loadBefore() give access to earlier transactions (as
long as they are not garbage collected) as needed for MVCC.  Ooops,
it inherits from BaseStorage that has the transaction management stuff.
I see it now.
 
> >
> > TemporaryStorage does provide a conflict cache to do "rudimentary  
> > conflict
> > resolution".  There are several timing and scaling parameters that  
> > need to
> > be considered:
> >
> > CONFLICT_CACHE_MAXAGE = 60  (seconds)
> > CONFLICT_CACHE_GCEVERY = 60 (seconds)
> > RECENTLY_GC_OIDS_LEN = 200
> >
> > Entries in the recently gc's oids list are those which may be  
> > resolvable
> > by a retry.
> >
> > These numbers may be too small given the loads we see and the  
> > number of
> > accesses made to the session variables.  I plan to incrase them to see
> > if there is any impact.
> 
> That's fine.
> >
> > MAYBE CONFLICTS AND THEIR RESOLUTION ARE NOT THE ROOT CAUSE OF THE  
> > SESSION
> > VARIABLE PROBLEM.
> 
> That's possible.
> 
> 
> > either all the session variables
> > are gone and only the container remains or most of the session  
> > variables
> > are gone and a few remain.
> 
> The former sounds like the session may have expired.  The second  
> sounds like it could be due to conflict resolution.  You don't say  
> where the KeyError is raised from, that's important.

****KeyErrors are raised by getSessionVariable(), code below.
> 
> >
> > 74769573A2H7SIH2AKo=id: 11343269231636975299, token:  
> > 74769573A2H7SIH2AKo,
> > contents keys: ['currentTab', 'calendarPage', 'currentCourse',
> > 'currentTextbook']
> >
> > and
> >
> > 77307574A2HTTdXCYYg=id: 11343267811075063138, token:  
> > 77307574A2HTTdXCYYg,
> > contents keys: []
> 
> Note that these are actually two different session objects (see the  
> "token").  I don't know if this is meaningful.
> 
> >
> > Access to the session variables are almost alwsys through a pair of
> > Scripts(Python).  Occasionally a session variable is read with an
> > expression of the form REQUEST['SESSION']['key'].
> >
> > ## Script (Python) "getSessionVariable"
> > ##bind container=container
> > ##bind context=context
> > ##bind namespace=
> > ##bind script=script
> > ##bind subpath=traverse_subpath
> > ##parameters=varname
> > ##title=
> > ##
> > request=container.REQUEST
> > session=request['SESSION']
> > return session[varname]
> 
> 
> How about session.get('varname') instead?  What happens when the  
> session expires?  Do you deal with that in your code in some way?
> 
**** session.get('varname') would not work as the variable being accessed 
is a parameter.  Are you questioning the syntax or the sematics?

**** sessions do not expire or rather have lifetimes that can be ignored.
We support and expect multi-hour sessions.   Other constraints ensure that 
users sessions terminate before they time out.

> >
> > # Script (Python) "setSessionVariable"
> > ##bind container=container
> > ##bind context=context
> > ##bind namespace=
> > ##bind script=script
> > ##bind subpath=traverse_subpath
> > ##parameters=var, val
> > ##title=
> > ##
> > request = container.REQUEST
> > RESPONSE =  request.RESPONSE
> > session=request['SESSION']
> > session[var]=val
> > request.set( 'SESSION', session )
> >
> > This all seems right to me.  Any suggestions as to how to localized  
> > when
> > the session variables get lost?  That might help localize the root  
> > cause.
> >
> 
> What is request.set('SESSION', session) supposed to do?  Whatever  
> it's supposed to be doing is probably not what you think it's doing.   
> I actually don't know quite what the impact of doing that would be;  
> it would depend on your other code.  Remove that, it can't do  
> anything good. ;-)  If you're trying to make sure the session is "re- 
> stored" in the ZODB, you needn't.  You only need to do this for  
> mutable variables *in* the session, eg.:
> 
> request.SESSION['a'] = {}
> a = request.SESSION['a']
> a['b'] = 'c'
> # here it comes
> request.SESSION['a'] = a

**** Hmmm... you may have hit upon the problem.  

The setSetSessionVariable code's intent was to copy out the entire session
object, mutate it, and then write it back into REQUEST, the approved way
blessed in some of the ancient documentation for the initial 
implementation of sessions in Zope 2.5.1.  You are saying that this is
not necessary at the level of the SessionObject and only really needed 
when the object being stored in the session is itself mutable (e.g., a 
list or a dictionary).  And the problems I have been seeing are likely to 
be due to an intereaction between the persistence mechanism and the way I 
was managing persistence.

I'll rewrite the setSessionVariable() routine and see if that resolves
the lost session variable problem.   

> 
> I also enumerated other knobs to you in previous emails that could  
> help reduce sessioning conflicts (like turning off inband  
> housekeeping -- see the Transience.py code for that -- and bumping up  
> transient object container timeout resolution via session-timeout- 
> resolution in zope.conf in your current config).  Those are the  
> things I meant when I said that you hadn't told us whether you had  
> taken steps based on recommendations.  I won't enumerate them here  
> again here as it should be reasonably easy to go look in the archives  
> or your mail history for those recommendations.  I'd suggest trying  
> these recommendations before playing around with conflict resolution  
> code.  Dunny also suggested trying to mess with the WRITEGRANULIARITY  
> in TransientObject.py, FWIW.
> 
I have with little effect on the primary problem.  

Thanks for the perceptive comments and sharp eyes.

--