MySQL thread connection question
Mornin, We use ZEO and two public clients on Linux along with Squid and POUND on one box. All this is attached to MySQL4 on the same box. For reasons unknown the clients will thread-hang for long periods and all the threads will fill up and make the client unreachable. POUND will still try to direct requests to the full client and so cause our 'www' to become unreachable. When this happens we manually pull the rug from the unresponsive client and restart it. Eveything is fine once again. However, in watching the threads in MySQL, there will be many sleeping threads that have lived for MUCH longer than any of the running zope clients. We are assuming that the zope client connections are established when the client comes up and are maintained whether used or not. When we kill the client those threads stay there and new connections are made to MySQL. As I write this I think its turning into more of a MySQL topic than zope but is there anything that can be done to kill off the old sleeping MySQL threads or have the zope client use the ones that are there already?? Sort of confusing but thought I would throw this out. Thanks Allen
aschmidt@fredericksburg.com wrote: Hello
Mornin,
We use ZEO and two public clients on Linux along with Squid and POUND on one box. All this is attached to MySQL4 on the same box. Nearly the same configuration, i think the same Problem!
For reasons unknown the clients will thread-hang for long periods and all the threads will fill up and make the client unreachable. POUND will still try to direct requests to the full client and so cause our 'www' to become unreachable.
When this happens we manually pull the rug from the unresponsive client and restart it. Eveything is fine once again. We do the same, unfortunately! Without ZEO it seams we don not have the problem. Or not so often.
However, in watching the threads in MySQL, there will be many sleeping threads that have lived for MUCH longer than any of the running zope clients. We are assuming that the zope client connections are established when the client comes up and are maintained whether used or not. When we kill the client those threads stay there and new connections are made to MySQL. How do you see these treads?
As I write this I think its turning into more of a MySQL topic than zope but is there anything that can be done to kill off the old sleeping MySQL threads or have the zope client use the ones that are there already??
Sort of confusing but thought I would throw this out.
Thanks
Allen
Christian
cklinger wrote:
However, in watching the threads in MySQL, there will be many sleeping
How do you see these treads?
We use a tool on Windows called MySQLFront (freeware) and you can see the threads and sort them by connection, database, time, etc. Very useful tool. I think it is ZEO that keeps them alive because if we kill and restart ZEO along with the clients the the number of mysql threads drops to almost nothing and then starts building back up as the clients reconnect each Zope client thread to each database connection. Has anyone seen "too many" mysql connections from Zope? What if we add more ZEO clients on the same or different servers? And each with 6 or 7 Zope threads...and each have the same number of database connections? Hundreds of MySQL connections... That seems it can only cause more problems. Just not sure. Allen
Allen Schmidt wrote:
I think it is ZEO that keeps them alive because if we kill and restart ZEO along with the clients the the number of mysql threads drops to almost nothing and then starts building back up as the clients reconnect each Zope client thread to each database connection.
ZEO has nothing to do with MySQL threads ;-) Only the ZEO clients and not the storage server will open connections to MySQL. When you restart the client it should drop all the connections...
Has anyone seen "too many" mysql connections from Zope?
Not MySQL, but look through the archives of zope-db@zope.org for lots of discussion of this problem..
What if we add more ZEO clients on the same or different servers? And each with 6 or 7 Zope threads...and each have the same number of database connections?
Well, then you have a helluva lot of MySQL connections ;-) It largely depends how ZMySQLDA manages its connections to MySQL, and you'd have to ask Andy Dustman about that...
Hundreds of MySQL connections... That seems it can only cause more problems. Just not sure.
Yep, 'cos if it works like most DA's, only a fraction of those will ever get used, and the rest will just hang around doing nothing :-( cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On Thu, 12 Aug 2004 13:22:43 +0100, Chris Withers <chris@simplistix.co.uk> wrote:
Allen Schmidt wrote:
What if we add more ZEO clients on the same or different servers? And each with 6 or 7 Zope threads...and each have the same number of database connections?
Well, then you have a helluva lot of MySQL connections ;-) It largely depends how ZMySQLDA manages its connections to MySQL, and you'd have to ask Andy Dustman about that...
Hundreds of MySQL connections... That seems it can only cause more problems. Just not sure.
Yep, 'cos if it works like most DA's, only a fraction of those will ever get used, and the rest will just hang around doing nothing :-(
I would say, it works like most DA's, i.e. ZMySQLDA doesn't add any connection pooling features itself. I suspect you should have one connection per Zope thread (after a sufficient number of requests). -- Computer interfaces should never be made of meat.
Andy Dustman wrote:
I would say, it works like most DA's, i.e. ZMySQLDA doesn't add any connection pooling features itself. I suspect you should have one connection per Zope thread (after a sufficient number of requests).
Waaaagh! *sigh* I'm starting to feel like I should see about migrating the changes I made for ZOracleDA into the ZOpe core and have all DA's use that way of doing things... What do people think? Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
In one of the systems we run we have O(150) static MySQL connections only 20 or so of which are active at any given time. We keep the number of connections low by changing databases with almost every query. Currently we have the connection limit set to O(200) to give us a little headroom. The number of connections is important as it multiples the per thread buffer requirements. +1 for making the adaptor pool connections, etc. +1 for migration into the core. On Fri, 13 Aug 2004, Chris Withers wrote:
Andy Dustman wrote:
I would say, it works like most DA's, i.e. ZMySQLDA doesn't add any connection pooling features itself. I suspect you should have one connection per Zope thread (after a sufficient number of requests).
Waaaagh!
*sigh*
I'm starting to feel like I should see about migrating the changes I made for ZOracleDA into the ZOpe core and have all DA's use that way of doing things...
What do people think?
Chris
-- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Dennis Allison wrote:
+1 for making the adaptor pool connections, etc. +1 for migration into the core.
Okay, I guess this is getting a little tight for Zope 2.8, maybe 2.9? Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote at 2004-8-13 10:41 +0200:
... *sigh*
I'm starting to feel like I should see about migrating the changes I made for ZOracleDA into the ZOpe core and have all DA's use that way of doing things...
What do people think?
I never felt that the current way is wrong in any way. You may get a few extra connections but unless you litter your site with redundant DA instances, that should not make a big problem. -- Dieter
Dieter, What if a site runs multiple independent Zopes but shares the database? Each of the Zopes needs its own connection(s). On Fri, 13 Aug 2004, Dieter Maurer wrote:
Chris Withers wrote at 2004-8-13 10:41 +0200:
... *sigh*
I'm starting to feel like I should see about migrating the changes I made for ZOracleDA into the ZOpe core and have all DA's use that way of doing things...
What do people think?
I never felt that the current way is wrong in any way.
You may get a few extra connections but unless you litter your site with redundant DA instances, that should not make a big problem.
-- Dieter _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Dennis Allison wrote at 2004-8-14 15:19 -0700:
What if a site runs multiple independent Zopes but shares the database? Each of the Zopes needs its own connection(s).
Sure, if they are truely independent -- unless you use SQLRelay (or something similar). -- Dieter
Dieter, we've been using the profiler and it works wonderfully well. Thanks. -d
Dieter (or anyone): have you had experience wiht SQLRelay? Using it would provide the equivalent of connection pooling. Currently, in our site, most queries are done in two parts--a "select database" query followed by the desired sql query--this shares the connection, but at some cost. Anyone have any idea about the tradeoffs? On Sun, 15 Aug 2004, Dieter Maurer wrote:
Dennis Allison wrote at 2004-8-14 15:19 -0700:
What if a site runs multiple independent Zopes but shares the database? Each of the Zopes needs its own connection(s).
Sure, if they are truely independent -- unless you use SQLRelay (or something similar).
-- Dieter
Dennis Allison wrote at 2004-8-15 14:46 -0700:
Dieter (or anyone): have you had experience wiht SQLRelay?
I used SQLRelay to connect to an MS SQL server. But just for a very short time: first the project was postponed and then the MS SQL server replaced by Postgres. As I expressed several times: we do not have problems with Zope's build in connection pooling. Thus, after the MS SQL server was gone, there was no longer any need to use SQLRelay (any intermediate actor slows things down). -- Dieter
Agred. Thanks/ On Mon, 16 Aug 2004, Dieter Maurer wrote:
Dennis Allison wrote at 2004-8-15 14:46 -0700:
Dieter (or anyone): have you had experience wiht SQLRelay?
I used SQLRelay to connect to an MS SQL server. But just for a very short time: first the project was postponed and then the MS SQL server replaced by Postgres.
As I expressed several times: we do not have problems with Zope's build in connection pooling. Thus, after the MS SQL server was gone, there was no longer any need to use SQLRelay (any intermediate actor slows things down).
-- Dieter
A long time ago, I made a very simple Pool class (based on Queue.Queue) which was primary for things like connection pooling. I never found the need to incorporate it into ZMySQLDA, but if you want to try it out, it's here http://dustman.net/andy/python/Pool It should even work on some antique versions of Python (1.5.2). Another thing you can try is the ping() method on the database connection. This tests to see if the connection is still open, and if not, it re-opens it. You can also kill threads on the server side. SHOW PROCESSLIST will show all active connections. Anything with Command='Sleep' and Time>some number of seconds you can kill with KILL id. For example, you could have a job that runs periodically and kills any connection by a certain user that has been sleeping longer than your timeout value. If you do this, you will definitely want to have ping() incorporated into ZMySQLDA, most likely in the begin() method; it should probably be in there anyway. -- Computer interfaces should never be made of meat.
In the ZMI I use Last Modified all the time to see what had been chnaged recently. It used to sort both ways on 2.6.2. Now it only sorts ascending... Another click still does not change the direction. Most annoying on 2.7.2. Anyone see this? Or is it not that big a deal to most? Allen
Allen Schmidt wrote:
In the ZMI I use Last Modified all the time to see what had been chnaged recently. It used to sort both ways on 2.6.2. Now it only sorts ascending... Another click still does not change the direction. Most annoying on 2.7.2.
Anyone see this? Or is it not that big a deal to most?
Fixed. Just a missing semicolon. <http://cvs.zope.org/Zope/lib/python/OFS/dtml/Attic/main.dtml.diff?r1=1.20.2.2&r2=1.20.2.3> Cheers, Yuppie
Guess I shoulda dug for it a bit. I use the skey and rkey sorts for many of the things I have built. Thanks! Allen yuppie wrote:
Allen Schmidt wrote:
In the ZMI I use Last Modified all the time to see what had been chnaged recently. It used to sort both ways on 2.6.2. Now it only sorts ascending... Another click still does not change the direction. Most annoying on 2.7.2.
Anyone see this? Or is it not that big a deal to most?
Fixed. Just a missing semicolon. <http://cvs.zope.org/Zope/lib/python/OFS/dtml/Attic/main.dtml.diff?r1=1.20.2.2&r2=1.20.2.3>
Cheers, Yuppie
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
We use about 10 total database connections. We have 2 public clients and one 'admin' client. The 2 public clients use 7 zope threads and the admin one uses 4. So (10 x 7) x 2 plus (10 x 4) total mysql connections each time the whole thing starts up. Then if we kill off a dead client it leaves the old ones sleeping and connects new ones all over. We do that several times and our da connections of mostly all sleeping is a few hundred. The numbers may be off a bit but pretty close. This is what we are seeing. Allen Dieter Maurer wrote:
Chris Withers wrote at 2004-8-13 10:41 +0200:
... *sigh*
I'm starting to feel like I should see about migrating the changes I made for ZOracleDA into the ZOpe core and have all DA's use that way of doing things...
What do people think?
I never felt that the current way is wrong in any way.
You may get a few extra connections but unless you litter your site with redundant DA instances, that should not make a big problem.
Allen Schmidt wrote at 2004-8-14 22:04 -0400:
We use about 10 total database connections.
Why do you have 10 database connections to the same server? Almost surely, I would have a single one.
... Then if we kill off a dead client it leaves the old ones sleeping and connects new ones all over.
Looks like a bug -- probably in either your operating system or the MySQL server. BTW, we do not see this problem: when we restart a Zope client, MySQL connections are closed correctly. We run on Linux.
We do that several times and our da connections of mostly all sleeping is a few hundred.
I doubt that a connection pool would help you: There probably is a reason that you allow seven threads. When the connection pool is not ready to allocate seven connections, you will get less concurrency than you (apparently) want. *If* there is a reason that you have more than 1 DA instance, then the connection pool cannot reuse connections for different DAs. Otherwise, you, too, could live with a single DA instance! This means, the worst case number of connections is the same in both schemes (unless you are ready to reduce concurrency). Using a connection pool will not fix your "sleeping connection" problem when you restart a client. Implementing a connection pool correctly is tricky (when one request requests two connections from the same DA, it *MUST* get the same connection). -- Dieter
Dieter Maurer wrote:
Why do you have 10 database connections to the same server?
10 different users with different levels of access?
We do that several times and our da connections of mostly all sleeping is a few hundred.
I doubt that a connection pool would help you:
I have real-world evidence to the contrary ;-)
There probably is a reason that you allow seven threads. When the connection pool is not ready to allocate seven connections, you will get less concurrency than you (apparently) want.
I don't understand what you're trying to say here...
*If* there is a reason that you have more than 1 DA instance, then the connection pool cannot reuse connections for different DAs. Otherwise, you, too, could live with a single DA instance!
Yep, but rather than creating one connection for each Zope thread, it only creates as many connections as are concurrently needed, and that's almost always a MUCH lower number...
This means, the worst case number of connections is the same in both schemes (unless you are ready to reduce concurrency).
With a connection pool, the worst case number of connections is the largest number of concurrent requests. Without one, it is the number of Zope threads (plus some for those leaked by _v_ variables), regardless of the load on the server.
Using a connection pool will not fix your "sleeping connection" problem when you restart a client.
That's true.
Implementing a connection pool correctly is tricky (when one request requests two connections from the same DA, it *MUST* get the same connection).
I'd be interested to hear your thoughts on the code in ZOracleDA on the branch I created... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
In an attempt to clear this a bit: (or make it cloudier) ZEO / 2 public and 1 admin Zope262 clients. Squid in front of all 3 and POUND in front of the 2 public clients. MySQL 4.0.17 All on one RH 8 Linux box We have other boxes we are considering adding more clients to but at the current rate we would be increasing our MySQL threads even more. We have multiple MySQL databases for the various things we run on our site(s). It is my understanding that if we have a Real Estate database with 30 tables then there is one connection. An Automotive database with many tables, there is another DA. Classified Ads takes another and so on... we have about 10 different databases all holding different types of things. If I refresh a page that references a specific database connection only a few of the 11 connections refresh....older ones stay sleeping. Does that shed any light on our setup? Thanks --Allen Chris Withers wrote:
Dieter Maurer wrote:
Why do you have 10 database connections to the same server?
10 different users with different levels of access?
We do that several times and our da connections of mostly all sleeping is a few hundred.
Allen Schmidt wrote:
Does that shed any light on our setup?
Yep, means you're likely to suffer from the problems I'm describing.. Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Allen Schmidt wrote at 2004-8-16 09:21 -0400:
... We have multiple MySQL databases for the various things we run on our site(s). It is my understanding that if we have a Real Estate database with 30 tables then there is one connection. An Automotive database with many tables, there is another DA. Classified Ads takes another and so on... we have about 10 different databases all holding different types of things.
You cannot pool connections to different databases. Chris' connection pool may however give you more control over the time period an unused connection stays in the pool.
If I refresh a page that references a specific database connection only a few of the 11 connections refresh....older ones stay sleeping.
Refreshing a page has little to do with connections. When you want to flush the connections in the ZODB pool, you must press "Minimize" in "Control_Panel --> Database Management --> Cache". When the connections do not disappear, then there is probably a problem in your network: We see occational hanging connections to our Postgres database server. It is caused by a malfunctioning firewall: it drops connections after sufficiently long inactivity in a way that the endpoints are not informed about the droppage. Consequence: the server waits indefinitely for incoming requests. There is a "keep alive" mechanism that should prevent this. -- Dieter
Dieter, your Zope Profiler has been a great help! I have had a bit of a problem understanding one part of the output -- sometimes the counts are written in a number/number notation (d.h, 4/2). How is that to be interpreted?
Dennis Allison wrote at 2004-8-23 11:12 -0700:
Dieter, your Zope Profiler has been a great help! I have had a bit of a problem understanding one part of the output -- sometimes the counts are written in a number/number notation (d.h, 4/2). How is that to be interpreted?
I answered a very similar question recently, I think on "plone-users". Please search the archives. Search for "Dieter ZopeProfiler artefact recursive". -- Dieter
Chris Withers wrote at 2004-8-16 13:48 +0200:
...
There probably is a reason that you allow seven threads. When the connection pool is not ready to allocate seven connections, you will get less concurrency than you (apparently) want.
I don't understand what you're trying to say here...
The current Zope code uses connection pooling: each DA instance is a pool. Its capacity is min(number_of_threads, number_of_ZODB_connections). When another pool implementation has lower capacity, concurrency is restricted.
... Yep, but rather than creating one connection for each Zope thread, it only creates as many connections as are concurrently needed, and that's almost always a MUCH lower number...
Looks as if you close connections as fast as possible? You are aware that some database systems (Oracle seems to be) have long connection times?
... With a connection pool, the worst case number of connections is the largest number of concurrent requests. Without one, it is the number of Zope threads (plus some for those leaked by _v_ variables),
There are no "_v_variable" leaks.
regardless of the load on the server.
The current pool is not so stupid. When it opens n connections than the server was hit by at least n concurrent requests. It is true, that not all of these concurrent requests need to access the database.
...
Implementing a connection pool correctly is tricky (when one request requests two connections from the same DA, it *MUST* get the same connection).
I'd be interested to hear your thoughts on the code in ZOracleDA on the branch I created...
I am not so keen to participate in code reviews. We are (apparently) happy with the current ZOracleDA code (the standard one, not that on your branch). No need yet, to look at your code. This will change, however, when your pooling enters the DA core in Zope -- and our Postgres interaction should no longer work correctly (I use Postgres, colleagues fight Oracle). -- Dieter
Dieter Maurer wrote:
The current Zope code uses connection pooling:
Yes, it uses ZODB's connection pool, which has very different requirements to the RDB connection pool in some common cases.
Yep, but rather than creating one connection for each Zope thread, it only creates as many connections as are concurrently needed, and that's almost always a MUCH lower number...
Looks as if you close connections as fast as possible?
No, it just doesn't open them unlesss it needs them.
You are aware that some database systems (Oracle seems to be) have long connection times?
Yes, which is why I don't recommend closing and opening connections all the time ;-)
With a connection pool, the worst case number of connections is the largest number of concurrent requests. Without one, it is the number of Zope threads (plus some for those leaked by _v_ variables),
There are no "_v_variable" leaks.
yeah there are, you describe them in your other mail :-)
The current pool is not so stupid. When it opens n connections than the server was hit by at least n concurrent requests.
Not true. Zope appears to randomly select ZODB connections to use from the available pool, default size 7...
It is true, that not all of these concurrent requests need to access the database.
...but this may also be the case, I'm happy to accept clarification in the form of line numbers in source files that do this work ;-)
I'd be interested to hear your thoughts on the code in ZOracleDA on the branch I created...
I am not so keen to participate in code reviews.
Well, that's a shame :-/
This will change, however, when your pooling enters the DA core in Zope -- and our Postgres interaction should no longer work correctly (I use Postgres, colleagues fight Oracle).
Heh, why do you think it won't work correctly? Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote at 2004-8-19 09:07 +0200:
...
The current pool is not so stupid. When it opens n connections than the server was hit by at least n concurrent requests.
Not true. Zope appears to randomly select ZODB connections to use from the available pool, default size 7...
Not true: Zope manages free connections in a stack and opens a new one only when the stack is empty.
It is true, that not all of these concurrent requests need to access the database.
...but this may also be the case, I'm happy to accept clarification in the form of line numbers in source files that do this work ;-)
"ZODB.DB.DB.open" and "ZODB.DB.DB._closeConnection". I cannot give you line numbers because my ZODB code is heavily changed.
... Heh, why do you think it won't work correctly?
Because it is not easy to implement a reliable connection pool (I explained this in a previous message) ;-) -- Dieter
Dieter Maurer wrote:
Not true: Zope manages free connections in a stack and opens a new one only when the stack is empty.
While I can believe that's the intent, the code is far from clear ;-)
...but this may also be the case, I'm happy to accept clarification in the form of line numbers in source files that do this work ;-)
"ZODB.DB.DB.open" and "ZODB.DB.DB._closeConnection".
I cannot give you line numbers because my ZODB code is heavily changed.
I now see why Jim says the version code complicates ZODB more than is needed ;-) As I said, I can believe the intent is that, but there's so much room for optionality that I'm not convinced that's what actually happens...
Because it is not easy to implement a reliable connection pool (I explained this in a previous message) ;-)
Getting lots of visibility on this would surely help it get better much more quickly. cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
[Dieter Maurer]
Not true: Zope manages free connections in a stack and opens a new one only when the stack is empty.
[Chris Withers]
While I can believe that's the intent, the code is far from clear ;-)
...
I now see why Jim says the version code complicates ZODB more than is needed ;-) As I said, I can believe the intent is that, but there's so much room for optionality that I'm not convinced that's what actually happens...
These are the only lines that matter <0.5 wink>: c=pool[-1] del pool[-1] in Db.open(). If an appropriate pool is available, a connection is reused from the end by exactly those two lines. When a connection is returned (_closeConnection()), it's added to the end of the appropriate pool: pool.append(connection) So all pools are stack-like, until they're exhausted. There are a couple of good proposals to drastically simplify the Connection maze (even while retaining versions, if necessary -- Dieter's fondness for them notwithstanding, the primary thing that's been keeping versions alive is that they *so* complicate so much of the code that it's dangerous to take them out): http://zope.org/Wikis/ZODB/NotesOnPossibleNewZODBConnectionPool http://zope.org/Wikis/ZODB/SimplifyConnectionManagement
Tim Peters wrote:
These are the only lines that matter <0.5 wink>:
c=pool[-1] del pool[-1]
in Db.open(). If an appropriate pool is available, a connection is reused from the end by exactly those two lines. When a connection is returned (_closeConnection()), it's added to the end of the appropriate pool:
pool.append(connection)
So all pools are stack-like, until they're exhausted.
*wry grinz*
Dieter's fondness for them notwithstanding, the primary thing that's been keeping versions alive is that they *so* complicate so much of the code that it's dangerous to take them out):
You peered into Shared.ZRDB recently? Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Dieter Maurer wrote:
I never felt that the current way is wrong in any way.
You may get a few extra connections but unless you litter your site with redundant DA instances, that should not make a big problem.
They're not redundant if you have a policy that means different departments use different subtrees of aZODB, and they each connect to different databases on different database servers. Zope's current method is a cheap attempt at connection pooling without doing real connection pooling, and one that uses _v_ variables in a very dodgy fashion... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote at 2004-8-16 13:42 +0200:
Dieter Maurer wrote:
I never felt that the current way is wrong in any way.
You may get a few extra connections but unless you litter your site with redundant DA instances, that should not make a big problem.
They're not redundant if you have a policy that means different departments use different subtrees of aZODB,
Maybe, they should comminicate how to use resources in their common Zope efficiently...
and they each connect to different databases on different database servers.
You cannot pool connections when they go to different databases or servers.
Zope's current method is a cheap attempt at connection pooling without doing real connection pooling,
It is reusing the existing ZODB connection pool. Reusing rather than reinventing is not so bad ;-)
and one that uses _v_ variables in a very dodgy fashion...
The only problem with "_v_" variables is that they can disappear at subtransaction boundaries. A long time ago, I posted a "_p_sticky" patch to "zodb-dev" that allows this to control. Unfortunately, the patch did not yet go into the ZODB core (and I fear, it will not happen at all). -- Dieter
Dieter Maurer wrote:
They're not redundant if you have a policy that means different departments use different subtrees of aZODB,
Maybe, they should comminicate how to use resources in their common Zope efficiently...
Maybe you should stop making facetious comments ;-)
and they each connect to different databases on different database servers.
You cannot pool connections when they go to different databases or servers.
true enough.
Zope's current method is a cheap attempt at connection pooling without doing real connection pooling,
It is reusing the existing ZODB connection pool. Reusing rather than reinventing is not so bad ;-)
...except when the pools have very differing requirements.
and one that uses _v_ variables in a very dodgy fashion...
The only problem with "_v_" variables is that they can disappear at subtransaction boundaries.
Yep, and with no attempt to close the RDB connection they represent, leaking DB connections... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote at 2004-8-19 09:09 +0200:
...
The only problem with "_v_" variables is that they can disappear at subtransaction boundaries.
Yep, and with no attempt to close the RDB connection they represent, leaking DB connections...
This is the responsibility of the class which encapulates the connection (its instances are used as values of the "_v_connection" attribute) if it does not happen automatically. It happens automatically for ZPsycopgDA. It happened automatically for an ancient ZOracleDA version (I used several years ago) after I had prevented the creation of a cyclic structure (which prevented automatic garbage collection). -- Dieter
Dieter Maurer wrote:
It happened automatically for an ancient ZOracleDA version (I used several years ago) after I had prevented the creation of a cyclic structure (which prevented automatic garbage collection).
Did those changes ever make it into a release? Are they still relevent now or does the cyclic garbage collection stuff solve the problem? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote at 2004-8-21 09:10 +0200:
Dieter Maurer wrote:
It happened automatically for an ancient ZOracleDA version (I used several years ago) after I had prevented the creation of a cyclic structure (which prevented automatic garbage collection).
Did those changes ever make it into a release?
I do not know.
Are they still relevent now or does the cyclic garbage collection stuff solve the problem?
I no longer work with Oracle. My colleagues (who work with Oracle) have not reported any connection leaks. Either, the problem is gone or they do not call Oracle stored procedures (the cycle was build only during stored procedure calls). -- Dieter
participants (9)
-
Allen Schmidt -
Andy Dustman -
aschmidt＠fredericksburg.com
-
Chris Withers -
cklinger -
Dennis Allison -
Dieter Maurer -
Tim Peters -
yuppie