ZODB Conflict, Thunked_TM and transaction hoseups
I have an irritated customer whose data driven zope website is screwing up, seemingly due to ZODB Conflicts on StyleSheet. Description: Zope 2.5.1 binary on linux Interbase, gvibDA Page Templates Use PageTemplates and lots of javascript with xml-rpc requests to pythonscripts to interace through ZSQL methods to talk to interbase via gvibDA. I also use SQLDict, maintaining a pool of SQLDict instances (they're expensive to create) in a Queue (stored in an external method) After pulling an SQLDict from the queue, I attach a gvibDA connection to sqldict by calling the Interbase DA object (to get a db instance), then (from an external method) I call db._register() and then use the connection object from that thread. This works rather well. However, as soon as I get a ZODB conflict, all subsequent transactions fail. That is, I am unable to alter the back-end (interbase) database at all. It's as if every single transaction after that point gets a permanent _rollback(). I have no idea why I'm getting conflicts on ZStyleSheet. I even tried creating a cached DTML method that caches the stylesheet, then serving that. But now I get conflicts on that object as well. 2002-07-12T18:45:38 INFO(0) ZODB conflict error at /Smoking/CachedStyleSheet (1 conflicts since startup at 2002-07-12T18:42:15) ------ 2002-07-12T18:47:02 INFO(0) ZODB conflict error at /Smoking/CachedStyleSheet (2 conflicts since startup at 2002-07-12T18:42:15) So .. it seems that my method of hooking into gvibDA/db.py, calling _register on it (it subclasses THUNKED_TM) is not entirely proper, because once a transaction error occurs, I seem to be blocking some proper rollback. But, why am I getting a conflict error in the first place? It makes no sense. . Just like getting conflicts on helpsys. Brad Clements, bkc@murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax AOL-IM: BKClements
your doing lots of very idiosyncratic things, so its hard to give much by way of advice or analysis. but the problems your seeing are exactly almost exactly the symptons of some txn and transaction bugs that have been fixed in 2.5.1. i did a write up of the details at. http://www.zope.org/Members/k_vertigo/ShortStories/Transactions/TxnProblems.... i find it strange that you might have run into them as your running 2.5.1, but its possible if you're trying to directly call the tm interface instead of letting the transaction drive the interaction beyond the intiial _register(). one question, are you using any sub transactions as well? its also likely from the description that, there are other problems (stylesheet conflicts). hth, kapil On Friday 12 July 2002 12:20 pm, Brad Clements wrote:
I have an irritated customer whose data driven zope website is screwing up, seemingly due to ZODB Conflicts on StyleSheet.
Description:
Zope 2.5.1 binary on linux
Interbase, gvibDA
Page Templates
Use PageTemplates and lots of javascript with xml-rpc requests to pythonscripts to interace through ZSQL methods to talk to interbase via gvibDA.
I also use SQLDict, maintaining a pool of SQLDict instances (they're expensive to create) in a Queue (stored in an external method)
After pulling an SQLDict from the queue, I attach a gvibDA connection to sqldict by calling the Interbase DA object (to get a db instance), then (from an external method) I call db._register() and then use the connection object from that thread.
This works rather well.
However, as soon as I get a ZODB conflict, all subsequent transactions fail. That is, I am unable to alter the back-end (interbase) database at all. It's as if every single transaction after that point gets a permanent _rollback().
I have no idea why I'm getting conflicts on ZStyleSheet. I even tried creating a cached DTML method that caches the stylesheet, then serving that. But now I get conflicts on that object as well.
2002-07-12T18:45:38 INFO(0) ZODB conflict error at /Smoking/CachedStyleSheet (1 conflicts since startup at 2002-07-12T18:42:15) ------ 2002-07-12T18:47:02 INFO(0) ZODB conflict error at /Smoking/CachedStyleSheet (2 conflicts since startup at 2002-07-12T18:42:15)
So .. it seems that my method of hooking into gvibDA/db.py, calling _register on it (it subclasses THUNKED_TM) is not entirely proper, because once a transaction error occurs, I seem to be blocking some proper rollback.
But, why am I getting a conflict error in the first place? It makes no sense. . Just like getting conflicts on helpsys.
Brad Clements, bkc@murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax AOL-IM: BKClements
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
On 13 Jul 2002 at 8:03, kapil thangavelu wrote:
but the problems your seeing are exactly almost exactly the symptons of some txn and transaction bugs that have been fixed in 2.5.1. i did a write up of the details at. http://www.zope.org/Members/k_vertigo/ShortStories/Transactions/TxnProblems....
I've looked at your patch, and compared it to the 2.5.1 source. Note that I'm using Thunked_TM, which doesn't appear to work exactly like TM. I've changed my Thunked_TM to use _finalize and also changed the order of thunk_lock.release to be after all changes to _registered (except for the initial register) However problem still remains. Basically what is happening is multiple db's are being created. I think this is the crux of the problem. I'm getting multiple connections to the database when there should be only exactly one. gvibDA's db is: class DB(Shared.DC.ZRDB.THUNK.THUNKED_TM): However I'm getting multiple instances created. As soon as a 2nd instance appears, the application fails in that the two different connections don't see each other's sql transactions. It appears that commit() is no longer being called on one of the connections. So . I need to find out why I get additional db's created, and how it is that I'm getting a "dead" one. I get the DB() object using an external method: def GetDB(self): """Return database""" global dbPool daobject = getattr(self,gvibObjectName) dbobject = daobject() gvibObjectName is the Zope id of my Interbase connection
i find it strange that you might have run into them as your running 2.5.1, but its possible if you're trying to directly call the tm interface instead of letting the transaction drive the interaction beyond the intiial _register().
I am only calling _register()
one question, are you using any sub transactions as well?
no
its also likely from the description that, there are other problems (stylesheet conflicts).
I don't know why these appear, but they do not appear to be related to the problem. That's a red herring. Brad Clements, bkc@murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax AOL-IM: BKClements
Brad Clements writes:
I have an irritated customer whose data driven zope website is screwing up, seemingly due to ZODB Conflicts on StyleSheet.
... I also use SQLDict, maintaining a pool of SQLDict instances (they're expensive to create) in a Queue (stored in an external method)
After pulling an SQLDict from the queue, I attach a gvibDA connection to sqldict by calling the Interbase DA object (to get a db instance), then (from an external method) I call db._register() and then use the connection object from that thread. I do not know SQLDict, but when it is a persistent object (or contains persistent objects), your approach can give strange errors because persistent objects read in one request are accessed in a different request and maybe in a different connection.
This can lead to mysterious errors. Dieter
On 15 Jul 2002 at 18:29, Dieter Maurer wrote:
Brad Clements writes:
I also use SQLDict, maintaining a pool of SQLDict instances (they're expensive to create) in a Queue (stored in an external method)
I do not know SQLDict, but when it is a persistent object (or contains persistent objects), your approach can give strange errors because persistent objects read in one request are accessed in a different request and maybe in a different connection.
I believe the problem is how THUNKED_TM handles it's lock and changing it's instance variables. Plus, TM.TM is now using _finalize and THUNKED_TM knows nothing about it. Here's my suggested patch for THUNKED_TM, which *appears* to have resolved the issue. Ideally, however, I want only one DB instance shared between ALL threads, rather than only "one active instance" which is what THUNKED_TM gets me. (I actually get one DB for each thread, which means one database connection per thread, which isn't really safe with Interbase) *** /usr/local/Zope/Zope-2.5.1-linux2-x86/lib/python/Shared/DC/ZRDB/THUNK.py Wed Nov 28 10:51:13 2001 --- THUNK.py Mon Jul 15 13:16:58 2002 *************** *** 22,46 **** def _register(self): if not self._registered: thunk_lock.acquire() ! try: ! get_transaction().register(Surrogate(self)) ! self._begin() ! except: ! thunk_lock.release() ! raise ! else: ! self._registered=1 def tpc_finish(self, *ignored): ! if self._registered: try: self._finish() finally: - thunk_lock.release() self._registered=0 def abort(self, *ignored): - if self._registered: try: self._abort() finally: - thunk_lock.release() self._registered=0 --- 22,40 ---- def _register(self): if not self._registered: thunk_lock.acquire() ! TM.TM._register(self) def tpc_finish(self, *ignored): ! if self._finalize: try: self._finish() finally: self._registered=0 + thunk_lock.release() def abort(self, *ignored): try: self._abort() finally: self._registered=0 + thunk_lock.release() + + tpc_abort = abort # match what's in TM.TM Brad Clements, bkc@murkworks.com (315)268-1000 http://www.murkworks.com (315)268-9812 Fax AOL-IM: BKClements
participants (3)
-
Brad Clements -
Dieter Maurer -
kapil thangavelu