Question about _v_ attributes and database connections
Hi all. We're trying to tackle a problem in hanging connections in ZPsycopg. However, I would like to have some information about the functioning of the _v_ attributes, and the cache management. In particular: 1) where is the code which handles the _v_ attributes (I like to look at the source code, you know ;)) 2) where is the code which handles the caching of the objects? 3) is it possible to disable caching of some object? 4) when an _v_ attribute is removed, what code of the object/attribute itself is called? If possible, please answer also directly to me (I'm not on this list). Regards Marco -- Marco Bizzarri - Amministratore Delegato - Icube S.r.l. Sede: Via Ridolfi 15 - 56124 Pisa (PI), Italia E-mail: m.bizzarri@icube.it WWW: www.icube.it Tel: (+39) 050 97 02 07 Fax: (+39) 050 31 36 588
Marco Bizzarri wrote:
Hi all.
We're trying to tackle a problem in hanging connections in ZPsycopg. However, I would like to have some information about the functioning of the _v_ attributes, and the cache management. In particular:
1) where is the code which handles the _v_ attributes (I like to look at the source code, you know ;))
2) where is the code which handles the caching of the objects?
3) is it possible to disable caching of some object?
4) when an _v_ attribute is removed, what code of the object/attribute itself is called?
I'm not sure which caching you mean -- _v_ variables aren't cached afaik. Theres plenty source code to look at, for example lib/python/ZODB/cPersistence.c -- there is some special handling of _v_ attributes there. Search for the Per__getstate__, ghostify / unghostify functions
If possible, please answer also directly to me (I'm not on this list).
Regards Marco
------------------------------------------------------------------------
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Hi Marco, Marco Bizzarri wrote:
We're trying to tackle a problem in hanging connections in ZPsycopg.
Hehe, I fixed this for ZOracleDA a couple of months ago. You may want to look at the thread pool code in there to see how I did it, gimme a nod if you need a URL (it's on a CVS branch in cvs.zope.org)
However, I would like to have some information about the functioning of the _v_ attributes, and the cache management. In particular:
1) where is the code which handles the _v_ attributes (I like to look at the source code, you know ;))
Never bothered finding it, the sematics are pretty well documented.
2) where is the code which handles the caching of the objects?
The ZODB cache. It's big and complicated, trust me, you DON'T need to go there...
3) is it possible to disable caching of some object?
No. The fact that you want to means you're mis-using _v_ attributes ;-)
4) when an _v_ attribute is removed, what code of the object/attribute itself is called?
None. I believe it gets lost at the object is ghosted.
If possible, please answer also directly to me (I'm not on this list).
Are you maintaining ZPsychopg? If not, the maintainers might be able to help... cheers, Chris - who learnt a whole lot more about ZODB and ZODB caching when solving this kind of problem in ZOracleDA ;-) -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On Fri, Mar 05, 2004 at 11:16:37AM +0000, Chris Withers wrote:
Hi Marco,
Hi.
Marco Bizzarri wrote:
We're trying to tackle a problem in hanging connections in ZPsycopg.
Hehe, I fixed this for ZOracleDA a couple of months ago. You may want to look at the thread pool code in there to see how I did it, gimme a nod if you need a URL (it's on a CVS branch in cvs.zope.org)
I took a look at that, the connections.CTM, and the other. The simple question is if this is a known problem with *ALL* db adapters, and, in this case, it could be possible to devise a general solution, or if this has something to do with psycopg. I've a scenario where I can produce hanging connections in ZPsycopg: 1) use page with sql methods 2) refresh a product 3) repeat 1) 4) refresh a product if you have 4 threads, you should have now 5 connections. Obviously here there is something wrong with the management of _v_ attributes, in particular the _v_database_connection in the DA.py. The psycopg code is quite similar to the basic DA.py code, so I was looking for other ways of approaching the problem.
However, I would like to have some information about the functioning of the _v_ attributes, and the cache management. In particular:
1) where is the code which handles the _v_ attributes (I like to look at the source code, you know ;))
Never bothered finding it, the sematics are pretty well documented.
Yes, actually.
2) where is the code which handles the caching of the objects?
The ZODB cache. It's big and complicated, trust me, you DON'T need to go there...
:(
3) is it possible to disable caching of some object?
No. The fact that you want to means you're mis-using _v_ attributes ;-)
My feeling was that the hanging connection problem was related to the cache... so the question (I'm not using _v_ attributes in any particular way)
4) when an _v_ attribute is removed, what code of the object/attribute itself is called?
None. I believe it gets lost at the object is ghosted.
If possible, please answer also directly to me (I'm not on this list).
Are you maintaining ZPsychopg? If not, the maintainers might be able to help...
No, actually we've a db application which makes a lot of use of zpsycopg for insert/updates, not only for select, and we stumbled on this problem. We're working with Federico Di Gregorio (psycopg mantainer) in order to solve the problem.
cheers,
Chris - who learnt a whole lot more about ZODB and ZODB caching when solving this kind of problem in ZOracleDA ;-)
I suspect I will have to learn something in the hard way too :)
-- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Thanks for your help Regards Marco -- Marco Bizzarri - Amministratore Delegato - Icube S.r.l. Sede: Via Ridolfi 15 - 56124 Pisa (PI), Italia E-mail: m.bizzarri@icube.it WWW: www.icube.it Tel: (+39) 050 97 02 07 Fax: (+39) 050 31 36 588
Marco Bizzarri wrote at 2004-3-5 09:18 +0100:
We're trying to tackle a problem in hanging connections in ZPsycopg. However, I would like to have some information about the functioning of the _v_ attributes, and the cache management. In particular:
1) where is the code which handles the _v_ attributes (I like to look at the source code, you know ;))
in "ZODB/cPersistence.c".
2) where is the code which handles the caching of the objects?
in "ZODB/cPickleCache.c"
3) is it possible to disable caching of some object?
No, but you can (in most cases) flush it again by calling its "_p_deactivate" method. You can always do it by assigning "None" to "_p_changed", but this is highly unsafe. You should not do it (unless you really know what you are doing).
4) when an _v_ attribute is removed, what code of the object/attribute itself is called?
The object's "__del__" method (if defined). This means the object that was bound to the "_v_" attribute not the object that had the "_v_attribute". -- Dieter
On Fri, 2004-03-05 at 16:57, Dieter Maurer wrote:
3) is it possible to disable caching of some object?
No, but you can (in most cases) flush it again by calling its "_p_deactivate" method.
You can always do it by assigning "None" to "_p_changed", but this is highly unsafe. You should not do it (unless you really know what you are doing).
Another was of saying this is that the cache performs an important role in maintaining object identity. It's not possible to use a persistent object without using a cached version of it. If you think you need to disable caching, you should probably say something more about what you are trying to do.
4) when an _v_ attribute is removed, what code of the object/attribute itself is called?
The object's "__del__" method (if defined). This means the object that was bound to the "_v_" attribute not the object that had the "_v_attribute".
The setattr hook in cPersistence.c is called when an attribute is removed (del obj.attr). An __del__() method would only be called if the last reference to the object was also deleted. And __del__ is to be avoided whenever possible. Jeremy
On Mon, Mar 08, 2004 at 09:34:28AM -0500, Jeremy Hylton wrote:
On Fri, 2004-03-05 at 16:57, Dieter Maurer wrote:
3) is it possible to disable caching of some object?
No, but you can (in most cases) flush it again by calling its "_p_deactivate" method.
You can always do it by assigning "None" to "_p_changed", but this is highly unsafe. You should not do it (unless you really know what you are doing).
Another was of saying this is that the cache performs an important role in maintaining object identity. It's not possible to use a persistent object without using a cached version of it. If you think you need to disable caching, you should probably say something more about what you are trying to do.
4) when an _v_ attribute is removed, what code of the object/attribute itself is called?
The object's "__del__" method (if defined). This means the object that was bound to the "_v_" attribute not the object that had the "_v_attribute".
The setattr hook in cPersistence.c is called when an attribute is removed (del obj.attr). An __del__() method would only be called if the last reference to the object was also deleted. And __del__ is to be avoided whenever possible.
What I would like to do is to have an hook to the removal of a _v_ attribute, so that I can perform some action *BEFORE* losing any reference to the object itself. I look at the setattr code in cPersistence.c and tried also to write my own __setattr__, but I do not know what should be called *AFTER* my code. What I would do is: def __setattr__(self, key, value): if key[0:3] == '_v_': doMyStuff() doStandardStuff() but I do not know what doStandardStuff actually is. Regards Marco
Jeremy
-- Marco Bizzarri - Amministratore Delegato - Icube S.r.l. Sede: Via Ridolfi 15 - 56124 Pisa (PI), Italia E-mail: m.bizzarri@icube.it WWW: www.icube.it Tel: (+39) 050 97 02 07 Fax: (+39) 050 31 36 588
Marco Bizzarri wrote:
What I would like to do is to have an hook to the removal of a _v_ attribute, so that I can perform some action *BEFORE* losing any reference to the object itself. I look at the setattr code in cPersistence.c and tried also to write my own __setattr__,
__setattr__ isn't the hook you're looking for.. This is better done in a transcation-hooking type way, a la CTM.py in ZOracleDA... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On Wed, Mar 10, 2004 at 02:56:33PM +0000, Chris Withers wrote:
Marco Bizzarri wrote:
What I would like to do is to have an hook to the removal of a _v_ attribute, so that I can perform some action *BEFORE* losing any reference to the object itself. I look at the setattr code in cPersistence.c and tried also to write my own __setattr__,
__setattr__ isn't the hook you're looking for..
Correct, I discovered that after some digging.
This is better done in a transcation-hooking type way, a la CTM.py in ZOracleDA...
cheers,
Chris
I'm trying something inspired to that at the moment... I succeeded in installing an hack, very inspired from your code (but at a much simpler level) and some code in the SecurityManager, where I mantain a dictionary of connections, indexed by the result of thread.get_ident(). Once a thread needs a new connection, either it is present in the dictionary or not. If it is present, give it to the thread and assign that to the _v_database_connection attribute. Otherwise, create a new one, put in the dictionary, and go on as before. Of course this is a (horrible) hack, because if you have more than one ZPsycopg objects you will end with mixed connections (aaaaargh!) As a proof of concept however it works, and it is resistent to the refreshing of other products (not the ZPsycopg one). Regards Marco
-- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
-- Marco Bizzarri - Amministratore Delegato - Icube S.r.l. Sede: Via Ridolfi 15 - 56124 Pisa (PI), Italia E-mail: m.bizzarri@icube.it WWW: www.icube.it Tel: (+39) 050 97 02 07 Fax: (+39) 050 31 36 588
Marco Bizzarri wrote:
I'm trying something inspired to that at the moment... I succeeded in installing an hack, very inspired from your code (but at a much simpler level) and some code in the SecurityManager,
?!
where I mantain a dictionary of connections, indexed by the result of thread.get_ident().
:-(
Once a thread needs a new connection, either it is present in the dictionary or not. If it is present, give it to the thread and assign that to the _v_database_connection attribute. Otherwise, create a new one, put in the dictionary, and go on as before.
When do these connections get closed?
Of course this is a (horrible) hack, because if you have more than one ZPsycopg objects you will end with mixed connections (aaaaargh!)
Indeed. cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On Thu, Mar 11, 2004 at 11:49:34AM +0000, Chris Withers wrote:
Marco Bizzarri wrote:
I'm trying something inspired to that at the moment... I succeeded in installing an hack, very inspired from your code (but at a much simpler level) and some code in the SecurityManager,
?!
Look in AccessControl/SecurityManagement.py, the getSecurityManager()... it does in that way.
where I mantain a dictionary of connections, indexed by the result of thread.get_ident().
:-(
I know, it is an hack :(
Once a thread needs a new connection, either it is present in the dictionary or not. If it is present, give it to the thread and assign that to the _v_database_connection attribute. Otherwise, create a new one, put in the dictionary, and go on as before.
When do these connections get closed?
Never, for the moment. -- Marco Bizzarri - Amministratore Delegato - Icube S.r.l. Sede: Via Ridolfi 15 - 56124 Pisa (PI), Italia E-mail: m.bizzarri@icube.it WWW: www.icube.it Tel: (+39) 050 97 02 07 Fax: (+39) 050 31 36 588
fwiw, i think this is a viable solution (although you should not only scope connections to thread local but to da paths in the module level global) as there are problems with zope and das going away because of their volatiles.. it can lead to strange behavior, like say your in the midst of a long txn, the da conn is involved gets registered w/ the txn. during the course of the txn the da persistent object is ghosted, latter it comes back but it will do so with a different connection if used again, this is bad from an rdb txn isolation pov, because assuming any isolation it can't see the changes done with the other as yet uncommitted connection (still alive cause its ref indirectly from the zope txn itself). unfortunately this is arch. problem with zope and current rdb/da integration and the only real solution that i see is via module level global storage for conns. as for shutdown of these conns, easiest might be to register an atexit function, or a cleaner sweeper thread with last usage times and in txn status on the otherwise thread local conn storage. cheers, -kapil On Thu, 2004-03-11 at 06:56, Marco Bizzarri wrote:
On Thu, Mar 11, 2004 at 11:49:34AM +0000, Chris Withers wrote:
Marco Bizzarri wrote:
I'm trying something inspired to that at the moment... I succeeded in installing an hack, very inspired from your code (but at a much simpler level) and some code in the SecurityManager,
?!
Look in AccessControl/SecurityManagement.py, the getSecurityManager()... it does in that way.
where I mantain a dictionary of connections, indexed by the result of thread.get_ident().
:-(
I know, it is an hack :(
Once a thread needs a new connection, either it is present in the dictionary or not. If it is present, give it to the thread and assign that to the _v_database_connection attribute. Otherwise, create a new one, put in the dictionary, and go on as before.
When do these connections get closed?
Never, for the moment.
participants (6)
-
Chris Withers -
Dieter Maurer -
emmebi＠icube.it -
Jeremy Hylton -
Kapil Thangavelu -
Peter Sabaini