[Zope] session variables in the presence of conflicts

[Chris McDonough]

On Sun, 2005-11-13 at 01:39 -0800, Dennis Allison wrote:
> Zope 2.8.4
> ZEO 3.4.2
> ZODB 3.4.2
> Python 2.4.2 or 2.3.5 
> MySQL 4.0.20
> MySQL-Python 1.2.0
> MYSQLDA 2.0.9
> 
> We have just moved from Zope 2.7.6 to Zope 2.8.4 motivated, in part, but 
> the ability to avoid read conflicts under ZODB 3.4.2.  We have been having 
> a lot of problems: more conflict errors, release of unreleased locks in 
> MySQL transactions, loss of session variables, and others.

That sounds unfortunate.

> We use session variables and MySQL very heavily.  Our MySQL tables are
> MyISAM and are not transactional.

That also sounds unfortunate. ;-)

> ZODB3 3.4.1 introduce some changes to the conflict resolution which
           ^ 3.4.2 I think you mean.

> appears to be a logical improvement but also appears to have acerbated
> our problems.  In the ZODB# 3.4.2 system, if a commit fails, an exception
> is raised and the transaction is aborted.  Zope then attempts to reschedule
> the transaction.  The abort needs to roll back MySQL transactions that are
> part of the conflicting ZODB and the  Session Variable Store so the 
> restart is clean.

Yes.  Of course with MyISAM tables, this not really possible as you note
below.

> In our case, we have been using MyISAM tables, which are not
> transactional.  The rollbacks fail and the locking in the database adapter
> gets confused. Changing the table type from MyISAM to Innodb, which is
> transactional, should resolve the MySQL problems, although it has yet to
> be tested.

FWIW, InnoDB tables are a good idea in any case.  But I'm suprised that
a rollback which includes MyISAM tables actually causes failure.  In my
experience with the most recent version of Andy's MySQL database
adapter, rollbacks don't actually do anything but they don't cause any
failures either.  This is also with a fairly recent MySQL
(4.1.something).

> Conflict errors also arise through the session variables.  Since we have a
> multi-Zope system configuration sharing a single ZEO, the session
> variables are stored in a Temporary Storage object on the ZEO host.  When
> a conflict occurs, a number of things, none good, apparently can happen.
> Session variable transactions cannot be rolled back (as the temporary
> storage mechanism is not transactional).

That is actually not true.  The tempstorage mechanism is transactional,
meaning that changes to a tempstorage will be committed or thrown out at
transaction boundaries depending on the result of the participating data
manager vote which determines whether a transaction should be committed
or aborted.  It is not undoable like FileStorage is, meaning you can't
go back and selectively choose states to roll back to past the current
database state, but it is indeed transactional.

>   And, at a different level,
> session variable persistence requires that transactions require a
> copy-out/copy-in mechanism to identify those changes which have been
> changed and need to persisted; an aborted transaction may not be marked as
> persistent and so will disappear at some apparently random time in the
> future when the temporary store is packed and cruft removed.

The temporary storage does not need to be packed unless there are cycles
in the data.  It is a "packless" storage in this regard.  However, the
sessioning machinery does do its own garbage collection of old data,
which removes old data from the storage.  But this machinery operates
(logically) independently of whatever storage implementation you happen
to be using to keep your session data manager.

If you see session data disappear "randomly", this is a bug.  However,
if it can be explained by the inability of a particular transaction to
commit due to bugs in participating data managers, it's not.

> Some refactoring of the programming can help minimize the potential for 
> session variable conflicts, but they cannot be eliminated.
> 
> If systems stability in the face of session variable conflicts is 
> important, 
> 
>   1.  Does the store for session variable need to be transactional 
>   and support rollbacks?

It already is.

> 
> and 
> 
>   2.  What Temporary Storage object provides the right mechanism to 
>   manage session variables with rollback efficiently?

There is only one, and it is coded in such a way that it should provide
this.  You might need to provide some additional information about what
kinds of errors you're experiencing; optimally you could provide a
reproducible test case but experience has shown that with sessioning
this is not always easy.

Before doing much more investigation on the Zope end, you may want to
get yourself to a place where you have recent versions of MySQL software
and Andy's DA.

- C