[Zope-dev] Re: Sessions and long requests = conflicts? Mcdutils?

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Maciej Wisniowski wrote:
> Hi
> 
> Currently I have some problems with our application (Zope2.8.4)
> and with Conflict Errors in sessions.
> In general if we have few concurrent requests that are running
> sometimes for 3-4 minutes (and they're touching session inside)
> I get a lot of conflict errors with Inceraser, OOBTree, Length2 etc.
> These are errors like:
> 
> serial starded with xxx serial currently commited xxxx
> 
> Of course I know that it is best to move such long processes to
> something external with Async, lovely.Remotetask or built
> in Oracle jobs, but so far I have to deal with this state of
> our application (as I'm not the author ot this but rather
> something like an zope admin/supporter for this app).
> 
> I think that ConflictErrors are caused because of sessions
> implementation, especially: timeslices, "current" bucket etc.
> as written in Transience/HowTransienceWorks.stx.
> Changing session_resolution_seconds to big value
> helps here. By default we had session_resolution_seconds
> set to 300.
> 
> I wonder how this happens. If I have two requests that at the beginning
> modify it's sessions like: session.set('aa', 1), and then call long
> running ZSQLMethods then (if meanwhile timeslice has changed because
> of too short session_resolution_seconds):
> 
> - first request that finishes finds that there is new 'current' bucket
>   and moves it's session object and second request's session object to
>   new 'current' bucket and commits this
> 
> - second request finishes and finds that it's session object is not
>   the same as it was at the beginning (because it was moved to 'current'
>   bucket)??
> 
> 
> Anybody can say if I'm right here?
> 
> 
> I also tried Faster product to manage sessions but this behaves in a
> similiar way (I mean causes conflict errors in such situation).
> 
> I found 'mcdutils' too. Tres Seaver said on zope-dev (a long time ago)
> that it is supposed to have no conflict errors:
> http://mail.zope.org/pipermail/zope-dev/2006-May/027555.html
> 
> Mcdutils:
> http://agendaless.com/Members/tseaver/software/mcdutils
> 
> There is only 0.1 version. What is it's current state? Seems to
> be dead? Can I take a look at this or is this better to not
> even touch that?

'mcdutils' is in a "worked for me under light load" state;  I never
ended up deploying it in production.  Because it is not based on the
ZODB, it is certainly not going to raise any ZODB conflict errors.
However, using memcache as the only backing store for your data is *not*
a recommended practice by the memcache developers:  they designed it as
a *cache*, not an atomic storage.  I dropped further development on it
once I evaluated the cost of having session data disappear (or become
inaccessible) when new memcache servers were added, or old ones removed.

I *do* use 'faster' in production, with the session storage mounted
across ZEO (a configuration which nobody in their right mind would do
with the standard session storage).  Some conflicts are possible in the
default configuration, although I never see them in practice at the user
level.  However, that applicaiton does *not* set up multiple
long-running transactions which attempt to mutate the same session data.

If you are *sharing* mutable session data between multiple long-running
requests, and expect to have no conflicts, you are in for a
disappointment:  unless your application can supply resolution logic,
you *want* conflict errors in such cases.

If you believe that 'faster' is raising conflicts due its own internal
data structures (OOBTree bucket splits), rather than in the
application-dveined session data, there is a conflict-free alternative
available:  we found that it was slower than the other, and therefore
didn't scale as well, even given the possibility of conflicts.

To enable the conflict-free storage, you need to patch the
'_BUCKET_TYPE' class-level variable of the storage to use
'AppendOnlyDict' rather than 'OOBTree'.  E.g.:

  from Products.faster.sessiondata import CBSessionDataContainer
  from Products.faster.appendict import AppendOnlyDict
  CBSessionDataContainer._BUCKET_TYPE = AppendOnlyDict

I had actually meant to make that a zope.conf-tweakable setting, but
never got around to it.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.2.2 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFGHbFO+gerLs4ltQ4RAk2UAJ9mFgY3OX4KZAK4ztGQwLMJX9MIEQCeJMfW
88/fKc3zcm4gi3efM15rJmg=
=fRak
-----END PGP SIGNATURE-----