I'm running 2.7.0 on Linux (Mandrake) with Python 2.3.4. I wanted to bump up the database connection pool limit so put the following custom_zodb.py in the instance home: import os import ZODB.FileStorage import ZODB.DB filename = os.path.join(INSTANCE_HOME, 'var', 'Data.fs') Storage = ZODB.FileStorage.FileStorage(filename) DB = ZODB.DB(Storage, pool_size=20, cache_size=2000) Zope starts fine w/ this (inasmuch as it creates a custom_zodb.pyc file) but the problem is that when I try to manage the database from the ZMI, it throws an exception complaining about not being able to lock the database. I've read elsewhere the using custom_zodb.py is deprecated by the new zope.conf file in Zope 2.7, but is there a way to specify the database connection pool size in zope.conf? TIA, John
John E. Barham wrote:
I'm running 2.7.0 on Linux (Mandrake) with Python 2.3.4.
I wanted to bump up the database connection pool limit so put the following custom_zodb.py in the instance home:
custom_zodb.py shouldn't be used in 2.7.0 AFAIK...
import os import ZODB.FileStorage import ZODB.DB
filename = os.path.join(INSTANCE_HOME, 'var', 'Data.fs') Storage = ZODB.FileStorage.FileStorage(filename) DB = ZODB.DB(Storage, pool_size=20, cache_size=2000)
Look at zope.conf in the /etc directory of your instance. Check out the ZODB section and see if there's a key that will do what you want. Also bear in mind that changing the pool size on it's own doesn't really do much. You need to bump up the number of Zope threads running, otherwise all you have is unused database connections lying about ;-) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
John E. Barham wrote:
I'm running 2.7.0 on Linux (Mandrake) with Python 2.3.4.
I wanted to bump up the database connection pool limit so put the following custom_zodb.py in the instance home:
custom_zodb.py shouldn't be used in 2.7.0 AFAIK...
So I'd gathered from reading the changelog for 2.7. However, Zope 2.7.0 still reads in the custom_zodb.py file if you create it but only complains when you try to manage it through the ZMI. IMHO it should either refuse to load custom_zodb.py (and print some message giving the recommended alternative) or else let you manage it from the ZMI.
import os import ZODB.FileStorage import ZODB.DB
filename = os.path.join(INSTANCE_HOME, 'var', 'Data.fs') Storage = ZODB.FileStorage.FileStorage(filename) DB = ZODB.DB(Storage, pool_size=20, cache_size=2000)
Look at zope.conf in the /etc directory of your instance. Check out the ZODB section and see if there's a key that will do what you want.
Also bear in mind that changing the pool size on it's own doesn't really do much. You need to bump up the number of Zope threads running, otherwise all you have is unused database connections lying about ;-)
To answer my own question, the Zope 2.7.0 way to increase the thread pool size is to set the pool-size parameter of the <zodb_db main> in etc/zope.conf like so: <zodb_db main> # Main FileStorage database <filestorage> path $INSTANCE/var/Data.fs </filestorage> mount-point / pool-size 15 </zodb_db> Bumping up the number of threads will also help as Chris says, but in my case I mainly wanted to be able to increase the number of possible external relational database connections (although they are distributed from general database connection pool). However, determining that this is the new way of increasing the database connections pool took an inordinate amount of time searching through the Zope source code and figuring out the new configuration system. What's the best way of documenting this so that other people don't have to go through the same process? John
On Wed, 2004-06-02 at 19:52, John E. Barham wrote:
However, determining that this is the new way of increasing the database connections pool took an inordinate amount of time searching through the Zope source code and figuring out the new configuration system. What's the best way of documenting this so that other people don't have to go through the same process?
You could submit to the Zope collector a patch to zope.conf.in (the file from which the default zope.conf is generated) with a description of the pool-size parameter. - C
John
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
John E. Barham wrote:
Bumping up the number of threads will also help as Chris says, but in my case I mainly wanted to be able to increase the number of possible external relational database connections (although they are distributed from general database connection pool).
I'm sorry? How does this setting relate to the number of possible external relational database connections?! Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
John E. Barham wrote:
Bumping up the number of threads will also help as Chris says, but in my case I mainly wanted to be able to increase the number of possible external relational database connections (although they are distributed from general database connection pool).
I'm sorry? How does this setting relate to the number of possible external relational database connections?!
Relational database connections are distributed from the general database connection pool. See: http://www.zope.org/Members/petrilli/ZSQLFAQ Elsewhere it's recommended that the database connection pool size be larger than the thread pool size.
John E. Barham wrote:
Relational database connections are distributed from the general database connection pool. See: http://www.zope.org/Members/petrilli/ZSQLFAQ
That article is old, unclear and misleading. The relationship between connections to a relational database and the DA object isn't as rigid as Chris implies. Take, for example, my branch of the ZOracleDA adapter, which has it's own pool of Oracle connections that it shares across all DA instances.
Elsewhere it's recommended that the database connection pool size be larger than the thread pool size.
Where and why? There's no benefit in having more database connections that publisher threads, unless the publisher is leaking database connections, which it doesn't and usually doesn't... Just occurred to me that you may be misunderstanding what a "database connection" is, since the zope terminology is horribly confused. The "SQL Connection" object you add through the ZMI is NOT a database connection, either relational or ZODB, and the number of those objects has nothing to do with the zodb pool size and not much to do with the number of connections from Zope to your relational database. BTW, what relational database and what database adapter are you using? Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
John E. Barham wrote:
Relational database connections are distributed from the general database connection pool. See: http://www.zope.org/Members/petrilli/ZSQLFAQ
That article is old, unclear and misleading.
Perhaps the original email from Jim Fulton on which the article is based is clearer: http://mail.zope.org/pipermail/zope/2000-January/101673.html It might be more than four years old but reading the Zope source code confirms that the default connection pool size is still 7.
Elsewhere it's recommended that the database connection pool size be larger than the thread pool size.
Where and why?
The Zope Book (2.6 Edition), chapter 24 (Maintaining Zope): """The "pool_size" parameter is the number of database connections. Note that the number of database connections should always be higher than the number of ZServer threads by a few (it doesn't make sense to have fewer database connections than threads).""" And Zope itself follows this recommendation in that the default database connection pool size is 7 but the default thread pool size is only 4.
Just occurred to me that you may be misunderstanding what a "database connection" is, since the zope terminology is horribly confused.
The "SQL Connection" object you add through the ZMI is NOT a database connection, either relational or ZODB, and the number of those objects has nothing to do with the zodb pool size and not much to do with the number of connections from Zope to your relational database.
I realize that although the terminology is admittedly confusing. Even if one were to create multiple database "connection" objects to the same relational database, I don't see how that would help since a Z SQL Method can only be associated w/ one database connection.
BTW, what relational database and what database adapter are you using?
PostgreSQL and psycopg on Linux. John
John E. Barham wrote:
Perhaps the original email from Jim Fulton on which the article is based is clearer: http://mail.zope.org/pipermail/zope/2000-January/101673.html
A cursory read suggests it backs up what I was saying ;-)
The Zope Book (2.6 Edition), chapter 24 (Maintaining Zope):
"""The "pool_size" parameter is the number of database connections. Note that the number of database connections should always be higher than the number of ZServer threads by a few (it doesn't make sense to have fewer database connections than threads)."""
And Zope itself follows this recommendation in that the default database connection pool size is 7 but the default thread pool size is only 4.
I can see no reason for this and remember recent mailing list discussions where this was agreed. But I might be hallucinating ;-) Explain to be how more than 4 ZODB connections can be used when there are only 4 publisher threads running, and none of them is leaking zodb connections...
I realize that although the terminology is admittedly confusing. Even if one were to create multiple database "connection" objects to the same relational database, I don't see how that would help since a Z SQL Method can only be associated w/ one database connection.
Can you explain what the problem is you're trying to solve please? Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
John E. Barham wrote at 2004-6-2 16:52 -0700:
... However, Zope 2.7.0 still reads in the custom_zodb.py file if you create it but only complains when you try to manage it through the ZMI. IMHO it should either refuse to load custom_zodb.py (and print some message giving the recommended alternative) or else let you manage it from the ZMI.
It is not always easy to strike the balance between cleanness and backward compatibility: Usually, when you use a "custom_zodb" you make the customazation in this file and *not* via the ZMI. Otherwise, it is unclear what customization should win. With Zope 2.7, a "custom_zodb" is almost in all cases no longer necessary. However, why force someone with a "custom_zodb" setup to switch to "zope.conf" customization and drop "custom_zodb"? I would have asked "why?" had Zope 2.7 told me I had to do that. Nowadays, I see that Zope has a bug with "custom_zodb" handling and I recognize that it is not worth to fix it. I dropped "custom_zodb". -- Dieter
participants (4)
-
Chris McDonough -
Chris Withers -
Dieter Maurer -
John E. Barham