On Sun, 2005-11-13 at 01:39 -0800, Dennis Allison wrote:
Zope 2.8.4 ZEO 3.4.2 ZODB 3.4.2 Python 2.4.2 or 2.3.5 MySQL 4.0.20 MySQL-Python 1.2.0 MYSQLDA 2.0.9
We have just moved from Zope 2.7.6 to Zope 2.8.4 motivated, in part, but the ability to avoid read conflicts under ZODB 3.4.2. We have been having a lot of problems: more conflict errors, release of unreleased locks in MySQL transactions, loss of session variables, and others.
That sounds unfortunate.
We use session variables and MySQL very heavily. Our MySQL tables are MyISAM and are not transactional.
That also sounds unfortunate. ;-)
ZODB3 3.4.1 introduce some changes to the conflict resolution which ^ 3.4.2 I think you mean.
appears to be a logical improvement but also appears to have acerbated our problems. In the ZODB# 3.4.2 system, if a commit fails, an exception is raised and the transaction is aborted. Zope then attempts to reschedule the transaction. The abort needs to roll back MySQL transactions that are part of the conflicting ZODB and the Session Variable Store so the restart is clean.
Yes. Of course with MyISAM tables, this not really possible as you note below.
In our case, we have been using MyISAM tables, which are not transactional. The rollbacks fail and the locking in the database adapter gets confused. Changing the table type from MyISAM to Innodb, which is transactional, should resolve the MySQL problems, although it has yet to be tested.
FWIW, InnoDB tables are a good idea in any case. But I'm suprised that a rollback which includes MyISAM tables actually causes failure. In my experience with the most recent version of Andy's MySQL database adapter, rollbacks don't actually do anything but they don't cause any failures either. This is also with a fairly recent MySQL (4.1.something).
Conflict errors also arise through the session variables. Since we have a multi-Zope system configuration sharing a single ZEO, the session variables are stored in a Temporary Storage object on the ZEO host. When a conflict occurs, a number of things, none good, apparently can happen. Session variable transactions cannot be rolled back (as the temporary storage mechanism is not transactional).
That is actually not true. The tempstorage mechanism is transactional, meaning that changes to a tempstorage will be committed or thrown out at transaction boundaries depending on the result of the participating data manager vote which determines whether a transaction should be committed or aborted. It is not undoable like FileStorage is, meaning you can't go back and selectively choose states to roll back to past the current database state, but it is indeed transactional.
And, at a different level, session variable persistence requires that transactions require a copy-out/copy-in mechanism to identify those changes which have been changed and need to persisted; an aborted transaction may not be marked as persistent and so will disappear at some apparently random time in the future when the temporary store is packed and cruft removed.
The temporary storage does not need to be packed unless there are cycles in the data. It is a "packless" storage in this regard. However, the sessioning machinery does do its own garbage collection of old data, which removes old data from the storage. But this machinery operates (logically) independently of whatever storage implementation you happen to be using to keep your session data manager. If you see session data disappear "randomly", this is a bug. However, if it can be explained by the inability of a particular transaction to commit due to bugs in participating data managers, it's not.
Some refactoring of the programming can help minimize the potential for session variable conflicts, but they cannot be eliminated.
If systems stability in the face of session variable conflicts is important,
1. Does the store for session variable need to be transactional and support rollbacks?
It already is.
and
2. What Temporary Storage object provides the right mechanism to manage session variables with rollback efficiently?
There is only one, and it is coded in such a way that it should provide this. You might need to provide some additional information about what kinds of errors you're experiencing; optimally you could provide a reproducible test case but experience has shown that with sessioning this is not always easy. Before doing much more investigation on the Zope end, you may want to get yourself to a place where you have recent versions of MySQL software and Andy's DA. - C