session variables in the presence of conflicts
Zope 2.8.4 ZEO 3.4.2 ZODB 3.4.2 Python 2.4.2 or 2.3.5 MySQL 4.0.20 MySQL-Python 1.2.0 MYSQLDA 2.0.9 We have just moved from Zope 2.7.6 to Zope 2.8.4 motivated, in part, but the ability to avoid read conflicts under ZODBÂ 3.4.2. We have been having a lot of problems: more conflict errors, release of unreleased locks in MySQL transactions, loss of session variables, and others. We use session variables and MySQL very heavily. Our MySQL tables are MyISAM and are not transactional. ZODB3 3.4.1 introduce some changes to the conflict resolution which appears to be a logical improvement but also appears to have acerbated our problems. In the ZODB# 3.4.2 system, if a commit fails, an exception is raised and the transaction is aborted. Zope then attempts to reschedule the transaction. The abort needs to roll back MySQL transactions that are part of the conflicting ZODB and the Session Variable Store so the restart is clean. In our case, we have been using MyISAM tables, which are not transactional. The rollbacks fail and the locking in the database adapter gets confused. Changing the table type from MyISAM to Innodb, which is transactional, should resolve the MySQL problems, although it has yet to be tested. Conflict errors also arise through the session variables. Since we have a multi-Zope system configuration sharing a single ZEO, the session variables are stored in a Temporary Storage object on the ZEO host. When a conflict occurs, a number of things, none good, apparently can happen. Session variable transactions cannot be rolled back (as the temporary storage mechanism is not transactional). And, at a different level, session variable persistence requires that transactions require a copy-out/copy-in mechanism to identify those changes which have been changed and need to persisted; an aborted transaction may not be marked as persistent and so will disappear at some apparently random time in the future when the temporary store is packed and cruft removed. Some refactoring of the programming can help minimize the potential for session variable conflicts, but they cannot be eliminated. If systems stability in the face of session variable conflicts is important, 1. Does the store for session variable need to be transactional and support rollbacks? and 2. What Temporary Storage object provides the right mechanism to manage session variables with rollback efficiently?
Have you considered upgrading to PostgreSQL? Things might have changed since the last time I tried mysql but I'm pretty sure Postgresql and psycopg deals much better with transactions in Zope. Admittedly, this doesn't explain why you're getting into trouble just because you've upgraded to zodb 3.4.2 On 11/13/05, Dennis Allison <allison@shasta.stanford.edu> wrote:
Zope 2.8.4 ZEO 3.4.2 ZODB 3.4.2 Python 2.4.2 or 2.3.5 MySQL 4.0.20 MySQL-Python 1.2.0 MYSQLDA 2.0.9
We have just moved from Zope 2.7.6 to Zope 2.8.4 motivated, in part, but the ability to avoid read conflicts under ZODB3.4.2. We have been having a lot of problems: more conflict errors, release of unreleased locks in MySQL transactions, loss of session variables, and others.
We use session variables and MySQL very heavily. Our MySQL tables are MyISAM and are not transactional.
ZODB3 3.4.1 introduce some changes to the conflict resolution which appears to be a logical improvement but also appears to have acerbated our problems. In the ZODB# 3.4.2 system, if a commit fails, an exception is raised and the transaction is aborted. Zope then attempts to reschedule the transaction. The abort needs to roll back MySQL transactions that are part of the conflicting ZODB and the Session Variable Store so the restart is clean.
In our case, we have been using MyISAM tables, which are not transactional. The rollbacks fail and the locking in the database adapter gets confused. Changing the table type from MyISAM to Innodb, which is transactional, should resolve the MySQL problems, although it has yet to be tested.
Conflict errors also arise through the session variables. Since we have a multi-Zope system configuration sharing a single ZEO, the session variables are stored in a Temporary Storage object on the ZEO host. When a conflict occurs, a number of things, none good, apparently can happen. Session variable transactions cannot be rolled back (as the temporary storage mechanism is not transactional). And, at a different level, session variable persistence requires that transactions require a copy-out/copy-in mechanism to identify those changes which have been changed and need to persisted; an aborted transaction may not be marked as persistent and so will disappear at some apparently random time in the future when the temporary store is packed and cruft removed.
Some refactoring of the programming can help minimize the potential for session variable conflicts, but they cannot be eliminated.
If systems stability in the face of session variable conflicts is important,
1. Does the store for session variable need to be transactional and support rollbacks?
and
2. What Temporary Storage object provides the right mechanism to manage session variables with rollback efficiently?
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
We are using MySQL but are fully transactional using innodb. The sort of problems we are seeing are (cruft removed) are things like: 2005-11-18T12:50:16 ERROR txn.3075 Error in tpc_abort() on manager <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820> 2005-11-18T12:50:16 ERROR txn.3075 Error in tpc_abort() on manager <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x41487acc> at 1190774252> 2005-11-18T12:50:16 INFO ZODB conflict error at /courses/topics/navigation_box (55 conflicts since startup at 2005-11-18T12:13:38) 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort object: Surrogate oid=<object object at 0x4009d470> 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820> 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort object: Surrogate oid=<object object at 0x4009d470> 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x41487acc> at 1190774252> all of which appear to be the result of the the single conflict error. The problems occur when the site is under load. To date I have not been able to find a simple instance that can reproduce the problem. Debugging in situ with a live system is not simple. In one instance, we have observed a loop where the conflict error generated resolution created another conflict error -- or so it appeared. On Sun, 13 Nov 2005, Peter Bengtsson wrote:
Have you considered upgrading to PostgreSQL? Things might have changed since the last time I tried mysql but I'm pretty sure Postgresql and psycopg deals much better with transactions in Zope. Admittedly, this doesn't explain why you're getting into trouble just because you've upgraded to zodb 3.4.2
On 11/13/05, Dennis Allison <allison@shasta.stanford.edu> wrote:
Zope 2.8.4 ZEO 3.4.2 ZODB 3.4.2 Python 2.4.2 or 2.3.5 MySQL 4.0.20 MySQL-Python 1.2.1c3 (although we see the same problems with 1.2.0) MYSQLDA 2.0.9
We have just moved from Zope 2.7.6 to Zope 2.8.4 motivated, in part, but the ability to avoid read conflicts under ZODB3.4.2. We have been having a lot of problems: more conflict errors, release of unreleased locks in MySQL transactions, loss of session variables, and others.
We use session variables and MySQL very heavily. Our MySQL tables are MyISAM and are not transactional.
ZODB3 3.4.1 introduce some changes to the conflict resolution which appears to be a logical improvement but also appears to have acerbated our problems. In the ZODB# 3.4.2 system, if a commit fails, an exception is raised and the transaction is aborted. Zope then attempts to reschedule the transaction. The abort needs to roll back MySQL transactions that are part of the conflicting ZODB and the Session Variable Store so the restart is clean.
In our case, we have been using MyISAM tables, which are not transactional. The rollbacks fail and the locking in the database adapter gets confused. Changing the table type from MyISAM to Innodb, which is transactional, should resolve the MySQL problems, although it has yet to be tested.
Conflict errors also arise through the session variables. Since we have a multi-Zope system configuration sharing a single ZEO, the session variables are stored in a Temporary Storage object on the ZEO host. When a conflict occurs, a number of things, none good, apparently can happen. Session variable transactions cannot be rolled back (as the temporary storage mechanism is not transactional). And, at a different level, session variable persistence requires that transactions require a copy-out/copy-in mechanism to identify those changes which have been changed and need to persisted; an aborted transaction may not be marked as persistent and so will disappear at some apparently random time in the future when the temporary store is packed and cruft removed.
Some refactoring of the programming can help minimize the potential for session variable conflicts, but they cannot be eliminated.
If systems stability in the face of session variable conflicts is important,
1. Does the store for session variable need to be transactional and support rollbacks?
and
2. What Temporary Storage object provides the right mechanism to manage session variables with rollback efficiently?
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-- Peter Bengtsson, work www.fry-it.com home www.peterbe.com hobby www.issuetrackerproduct.com
--
Dennis Allison wrote at 2005-11-18 13:15 -0800:
We are using MySQL but are fully transactional using innodb. 2005-11-18T12:50:16 ERROR txn.3075 Error in tpc_abort() on manager <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820>
Thus, you should find out why "tpc_abort()" fails on the "ZMySQLDA.db.DB" instance...
... 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820>
same here.... -- Dieter
Dennis Allison wrote:
We are using MySQL but are fully transactional using innodb.
The sort of problems we are seeing are (cruft removed) are things like:
2005-11-18T12:50:16 ERROR txn.3075 Error in tpc_abort() on manager <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820>
There should be a traceback here, or if there's none you should patch _transaction.py to make it produced. Seeing that traceback is important and would point to an error somewhere in MySQLDA's code I expect.
2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820> 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort object: Surrogate oid=<object object at 0x4009d470> 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x41487acc> at 1190774252>
Same here, the traceback is important. Florent -- Florent Guillaume, Nuxeo (Paris, France) Director of R&D +33 1 40 33 71 59 http://nuxeo.com fg@nuxeo.com
Florent, There were, of course, tracebacks. Any assist you can provide would be appreciated. -D ------ 2005-11-18T12:50:16 ERROR txn.3075 Error in tpc_abort() on manager <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/_transaction.py", line 462, in _cleanup rm.tpc_abort(self) File "/usr/local/src/zope/Zope2.8/lib/python/transaction/_transaction.py", line 548, in tpc_abort self.manager.tpc_abort(txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------ 2005-11-18T12:50:16 ERROR txn.3075 Error in tpc_abort() on manager <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x41487acc> at 1190774252> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/_transaction.py", line 462, in _cleanup rm.tpc_abort(self) File "/usr/local/src/zope/Zope2.8/lib/python/transaction/_transaction.py", line 548, in tpc_abort self.manager.tpc_abort(txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------ 2005-11-18T12:50:16 INFO ZODB conflict error at /courses/topics/navigation_box (55 conflicts since startup at 2005-11-18T12:13:38) ------ 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort object: Surrogate oid=<object object at 0x4009d470> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/_transaction.py", line 562, in abort self.manager.abort(o, txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------ 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/_transaction.py", line 489, in abort rm.abort(self) File "/usr/local/src/zope/Zope2.8/lib/python/transaction/_transaction.py", line 562, in abort self.manager.abort(o, txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------ 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort object: Surrogate oid=<object object at 0x4009d470> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/_transaction.py", line 562, in abort self.manager.abort(o, txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------ 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x41487acc> at 1190774252> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/_transaction.py", line 489, in abort rm.abort(self) File "/usr/local/src/zope/Zope2.8/lib/python/transaction/_transaction.py", line 562, in abort self.manager.abort(o, txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------ On Sat, 19 Nov 2005, Florent Guillaume wrote:
Dennis Allison wrote:
We are using MySQL but are fully transactional using innodb.
The sort of problems we are seeing are (cruft removed) are things like:
2005-11-18T12:50:16 ERROR txn.3075 Error in tpc_abort() on manager <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820>
There should be a traceback here, or if there's none you should patch _transaction.py to make it produced. Seeing that traceback is important and would point to an error somewhere in MySQLDA's code I expect.
2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820> 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort object: Surrogate oid=<object object at 0x4009d470> 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x41487acc> at 1190774252>
Same here, the traceback is important.
Florent
--
Dennis Allison wrote:
Florent, There were, of course, tracebacks. Any assist you can provide would be appreciated. -D
------ 2005-11-18T12:50:16 ERROR txn.3075 Error in tpc_abort() on manager <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/_transaction.py", line 462, in _cleanup rm.tpc_abort(self) File "/usr/local/src/zope/Zope2.8/lib/python/transaction/_transaction.py", line 548, in tpc_abort self.manager.tpc_abort(txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock
Well undoubtedly the bug is in ZMySQLDA... Florent -- Florent Guillaume, Nuxeo (Paris, France) Director of R&D +33 1 40 33 71 59 http://nuxeo.com fg@nuxeo.com
That was my conclusion long ago, but I have not been able to track it down. It only occurs under heavy load when backing out of a conflict error. I have not been successful at creating a test set that triggers it. I suppose it is time to again read the ZMySQLDA code... There are a few comments on the net about problems with MySQLDA which may also be related. On Sun, 20 Nov 2005, Florent Guillaume wrote:
Dennis Allison wrote:
Florent, There were, of course, tracebacks. Any assist you can provide would be appreciated. -D
------ 2005-11-18T12:50:16 ERROR txn.3075 Error in tpc_abort() on manager <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/_transaction.py", line 462, in _cleanup rm.tpc_abort(self) File "/usr/local/src/zope/Zope2.8/lib/python/transaction/_transaction.py", line 548, in tpc_abort self.manager.tpc_abort(txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock
Well undoubtedly the bug is in ZMySQLDA...
Florent
--
Hi Dennis, I notice that line 389 of db.py of the most recent ZMySQLDA (2.0.8) doesn't match the traceback you show. No mutex locking at all is done in the 2.0.8 version of that module (or any other module in that product). Are you using an older version? - C On Nov 20, 2005, at 9:57 AM, Dennis Allison wrote:
Florent, There were, of course, tracebacks. Any assist you can provide would be appreciated. -D
------ 2005-11-18T12:50:16 ERROR txn.3075 Error in tpc_abort() on manager <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 462, in _cleanup rm.tpc_abort(self) File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 548, in tpc_abort self.manager.tpc_abort(txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/ TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------ 2005-11-18T12:50:16 ERROR txn.3075 Error in tpc_abort() on manager <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x41487acc> at 1190774252> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 462, in _cleanup rm.tpc_abort(self) File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 548, in tpc_abort self.manager.tpc_abort(txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/ TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------ 2005-11-18T12:50:16 INFO ZODB conflict error at /courses/topics/ navigation_box (55 conflicts since startup at 2005-11-18T12:13:38) ------ 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort object: Surrogate oid=<object object at 0x4009d470> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 562, in abort self.manager.abort(o, txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/ TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------ 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 489, in abort rm.abort(self) File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 562, in abort self.manager.abort(o, txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/ TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------ 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort object: Surrogate oid=<object object at 0x4009d470> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 562, in abort self.manager.abort(o, txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/ TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------ 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x41487acc> at 1190774252> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 489, in abort rm.abort(self) File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 562, in abort self.manager.abort(o, txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/ TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------
On Sat, 19 Nov 2005, Florent Guillaume wrote:
Dennis Allison wrote:
We are using MySQL but are fully transactional using innodb.
The sort of problems we are seeing are (cruft removed) are things like:
2005-11-18T12:50:16 ERROR txn.3075 Error in tpc_abort() on manager <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820>
There should be a traceback here, or if there's none you should patch _transaction.py to make it produced. Seeing that traceback is important and would point to an error somewhere in MySQLDA's code I expect.
2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820> 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort object: Surrogate oid=<object object at 0x4009d470> 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x41487acc> at 1190774252>
Same here, the traceback is important.
Florent
--
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Sorry, I noticed in another email you said that you're using 2.0.9 (which appears to be in beta) and I confirmed that the traceback matches that source. The note in the product's CHANGES.txt about this is: Wrap queries with a lock to prevent multiple threads from using the connection simultaneously (this may or may not be happening). If transactional, then there is an additional transaction lock, acquired at the beginning of the transaction and released when either finished or aborted. The last sentence is the meaningful one in your case. I'm not sure if this is a bug fix but you may want to try 2.0.8 if you haven't already because it doesn't use a transaction lock (or any other lock). It seems as if the logic in 2.0.9b to obtain the transaction lock fails to take some corner case into account, although it's not obvious from the code what that is. On Nov 20, 2005, at 1:55 PM, Chris McDonough wrote:
Hi Dennis,
I notice that line 389 of db.py of the most recent ZMySQLDA (2.0.8) doesn't match the traceback you show. No mutex locking at all is done in the 2.0.8 version of that module (or any other module in that product). Are you using an older version?
- C
On Nov 20, 2005, at 9:57 AM, Dennis Allison wrote:
Florent, There were, of course, tracebacks. Any assist you can provide would be appreciated. -D
------ 2005-11-18T12:50:16 ERROR txn.3075 Error in tpc_abort() on manager <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 462, in _cleanup rm.tpc_abort(self) File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 548, in tpc_abort self.manager.tpc_abort(txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/ TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------ 2005-11-18T12:50:16 ERROR txn.3075 Error in tpc_abort() on manager <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x41487acc> at 1190774252> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 462, in _cleanup rm.tpc_abort(self) File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 548, in tpc_abort self.manager.tpc_abort(txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/ TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------ 2005-11-18T12:50:16 INFO ZODB conflict error at /courses/topics/ navigation_box (55 conflicts since startup at 2005-11-18T12:13:38) ------ 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort object: Surrogate oid=<object object at 0x4009d470> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 562, in abort self.manager.abort(o, txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/ TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------ 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 489, in abort rm.abort(self) File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 562, in abort self.manager.abort(o, txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/ TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------ 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort object: Surrogate oid=<object object at 0x4009d470> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 562, in abort self.manager.abort(o, txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/ TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------ 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x41487acc> at 1190774252> Traceback (most recent call last): File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 489, in abort rm.abort(self) File "/usr/local/src/zope/Zope2.8/lib/python/transaction/ _transaction.py", line 562, in abort self.manager.abort(o, txn) File "/usr/local/src/zope/Zope2.8/lib/python/Shared/DC/ZRDB/ TM.py", line 64, in abort try: self._abort() File "/opt/zope/zproducts/standard/ZMySQLDA/db.py", line 389, in _abort self._tlock.release() error: release unlocked lock ------
On Sat, 19 Nov 2005, Florent Guillaume wrote:
Dennis Allison wrote:
We are using MySQL but are fully transactional using innodb.
The sort of problems we are seeing are (cruft removed) are things like:
2005-11-18T12:50:16 ERROR txn.3075 Error in tpc_abort() on manager <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820>
There should be a traceback here, or if there's none you should patch _transaction.py to make it produced. Seeing that traceback is important and would point to an error somewhere in MySQLDA's code I expect.
2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x450431cc> at 1190763820> 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort object: Surrogate oid=<object object at 0x4009d470> 2005-11-18T12:50:16 ERROR txn.3075 Failed to abort resource manager: <MultiObjectResourceAdapter for <Products.ZMySQLDA.db.DB instance at 0x41487acc> at 1190774252>
Same here, the traceback is important.
Florent
--
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
On Sun, 2005-11-13 at 01:39 -0800, Dennis Allison wrote:
Zope 2.8.4 ZEO 3.4.2 ZODB 3.4.2 Python 2.4.2 or 2.3.5 MySQL 4.0.20 MySQL-Python 1.2.0 MYSQLDA 2.0.9
We have just moved from Zope 2.7.6 to Zope 2.8.4 motivated, in part, but the ability to avoid read conflicts under ZODB 3.4.2. We have been having a lot of problems: more conflict errors, release of unreleased locks in MySQL transactions, loss of session variables, and others.
That sounds unfortunate.
We use session variables and MySQL very heavily. Our MySQL tables are MyISAM and are not transactional.
That also sounds unfortunate. ;-)
ZODB3 3.4.1 introduce some changes to the conflict resolution which ^ 3.4.2 I think you mean.
appears to be a logical improvement but also appears to have acerbated our problems. In the ZODB# 3.4.2 system, if a commit fails, an exception is raised and the transaction is aborted. Zope then attempts to reschedule the transaction. The abort needs to roll back MySQL transactions that are part of the conflicting ZODB and the Session Variable Store so the restart is clean.
Yes. Of course with MyISAM tables, this not really possible as you note below.
In our case, we have been using MyISAM tables, which are not transactional. The rollbacks fail and the locking in the database adapter gets confused. Changing the table type from MyISAM to Innodb, which is transactional, should resolve the MySQL problems, although it has yet to be tested.
FWIW, InnoDB tables are a good idea in any case. But I'm suprised that a rollback which includes MyISAM tables actually causes failure. In my experience with the most recent version of Andy's MySQL database adapter, rollbacks don't actually do anything but they don't cause any failures either. This is also with a fairly recent MySQL (4.1.something).
Conflict errors also arise through the session variables. Since we have a multi-Zope system configuration sharing a single ZEO, the session variables are stored in a Temporary Storage object on the ZEO host. When a conflict occurs, a number of things, none good, apparently can happen. Session variable transactions cannot be rolled back (as the temporary storage mechanism is not transactional).
That is actually not true. The tempstorage mechanism is transactional, meaning that changes to a tempstorage will be committed or thrown out at transaction boundaries depending on the result of the participating data manager vote which determines whether a transaction should be committed or aborted. It is not undoable like FileStorage is, meaning you can't go back and selectively choose states to roll back to past the current database state, but it is indeed transactional.
And, at a different level, session variable persistence requires that transactions require a copy-out/copy-in mechanism to identify those changes which have been changed and need to persisted; an aborted transaction may not be marked as persistent and so will disappear at some apparently random time in the future when the temporary store is packed and cruft removed.
The temporary storage does not need to be packed unless there are cycles in the data. It is a "packless" storage in this regard. However, the sessioning machinery does do its own garbage collection of old data, which removes old data from the storage. But this machinery operates (logically) independently of whatever storage implementation you happen to be using to keep your session data manager. If you see session data disappear "randomly", this is a bug. However, if it can be explained by the inability of a particular transaction to commit due to bugs in participating data managers, it's not.
Some refactoring of the programming can help minimize the potential for session variable conflicts, but they cannot be eliminated.
If systems stability in the face of session variable conflicts is important,
1. Does the store for session variable need to be transactional and support rollbacks?
It already is.
and
2. What Temporary Storage object provides the right mechanism to manage session variables with rollback efficiently?
There is only one, and it is coded in such a way that it should provide this. You might need to provide some additional information about what kinds of errors you're experiencing; optimally you could provide a reproducible test case but experience has shown that with sessioning this is not always easy. Before doing much more investigation on the Zope end, you may want to get yourself to a place where you have recent versions of MySQL software and Andy's DA. - C
participants (5)
-
Chris McDonough -
Dennis Allison -
Dieter Maurer -
Florent Guillaume -
Peter Bengtsson