if you're going to be connect to postgres outside of zope i recommend not using PoPy. Its regularly crashes python for me when i try to do things any heavy db work from outside zope. heavy constituting using bind variables, stored procs, checking rowcount, description, etc. in accessing postgres external to zope i'd recommend using pygresql (the native non-standard interface not the dbi wrapper) or pgsql.
while it still does not support direct call of stored procedures using callproc(), psycopg is pretty stable on bind variables and also offers a very nice type-casting system to convers postgres results to custom python types.
one thing to not about ZPygre is that it synchronizes all db access. if
this is bad, because you can't rollback...
you're making lots of small queries this is not the adaptor for you. if you've got some large queries you're better off testing both and seeing which one gives you better throughput (probably PoPy, buts it not a sure thing).
i am very interested in drivers performance: has anybody set up a test of some kind to test this stuff? we use a simple test with 3 threads doing simultaneous INSERTs and SELECTs, creating lots of cursors and psycopg seems pretty fast but i would better like some *real* test.
any others? psycopg mentions that it is designed for heavy multi threaded operations but popy seems more developed.
psycopg has interesting arch, but if you're using postgres its good to keep in mind that libpq (the c client interface to postgres) does not support multiple threads sharing a connection. psycopg does internal bookeeping in
that's why we came out with the "interesting arch" :) note that psycopg does it better when you create lots of short-lived threads all using the same connection and creating one or more cursor each.
the module itself after a connect to keep around a pool of connections. when it matures it should be a very good postgres adaptor for zope.
ok, we wrote it in about 4 weeks, but let me say that psycopg is already pretty stable. we know at least 1 company that switched its production site from another driver (i won't say what) to psycopg because they had stability problems... i am glad to say that the problems are now resolved (mmm... or, maybe, psycopg simply destroyed their entire production server, i can't say... :) jokes apart, a new product can't mature if nobody use it because it is "still young", so try it out and send bug reports, thanx! federico -- Federico Di Gregorio MIXAD LIVE Chief of Research & Technology fog@mixadlive.com Debian GNU/Linux Developer & Italian Press Contact fog@debian.org Try the Joy of TeX [http://www.tug.org] -- brought to you by One Line Spam
On Friday 09 March 2001 17:06, Federico Di Gregorio wrote:
if you're going to be connect to postgres outside of zope i recommend not using PoPy. Its regularly crashes python for me when i try to do things any heavy db work from outside zope. heavy constituting using bind variables, stored procs, checking rowcount, description, etc. in accessing postgres external to zope i'd recommend using pygresql (the native non-standard interface not the dbi wrapper) or pgsql.
while it still does not support direct call of stored procedures using callproc(), psycopg is pretty stable on bind variables and also offers a very nice type-casting system to convers postgres results to custom python types.
incidentally, my attempts at 'heavy use' were for some tenative work i was doing on a postgres storage, but i ended up having too many problems getting any of the postgres adaptors to properly handle binary data (not in lob in a text field), bind variables, and calling of procs (converted over the OracleStorage to pl/pgsql procs) all at the same time. between the fact that a db -backed storage wasn't very useful in the first place and the difficulty in getting things working i shelved it. i was wondering if there are any plans for psycopg to go beyond the db-api and support more of the functionality of libpq, such as asynch requests, and notifications. (yes, i know the line about contributions, but i'm more curious about your plans for psycopg).
one thing to not about ZPygre is that it synchronizes all db access. if
this is bad, because you can't rollback...
ok, i'll bite. why not?
you're making lots of small queries this is not the adaptor for you. if you've got some large queries you're better off testing both and seeing which one gives you better throughput (probably PoPy, buts it not a sure thing).
i am very interested in drivers performance: has anybody set up a test of some kind to test this stuff? we use a simple test with 3 threads doing simultaneous INSERTs and SELECTs, creating lots of cursors and psycopg seems pretty fast but i would better like some *real* test.
hmmm... i vaguely recall someone posting some comparisions between pygre and popy in the last month or two, you'll have to check the searchable archives.
any others? psycopg mentions that it is designed for heavy multi threaded operations but popy seems more developed.
psycopg has interesting arch, but if you're using postgres its good to keep in mind that libpq (the c client interface to postgres) does not support multiple threads sharing a connection. psycopg does internal bookeeping in
that's why we came out with the "interesting arch" :) note that psycopg does it better when you create lots of short-lived threads all using the same connection and creating one or more cursor each.
the module itself after a connect to keep around a pool of connections. when it matures it should be a very good postgres adaptor for zope.
ok, we wrote it in about 4 weeks, but let me say that psycopg is already pretty stable. we know at least 1 company that switched its production site from another driver (i won't say what) to psycopg because they had stability problems... i am glad to say that the problems are now resolved (mmm... or, maybe, psycopg simply destroyed their entire production server, i can't say... :)
jokes apart, a new product can't mature if nobody use it because it is "still young", so try it out and send bug reports, thanx!
true, i'll try my attempts on a postgres storage on it and see what happens. although for something like a zodb storage the multiple connection thing is not useful since the zodb will synchronize access to the storage layer. i'll see if i can convert my experiments into some unittests. cheers kapil
Scavenging the mail folder uncovered ender's letter:
while it still does not support direct call of stored procedures using callproc(), psycopg is pretty stable on bind variables and also offers a very nice type-casting system to convers postgres results to custom python types.
incidentally, my attempts at 'heavy use' were for some tenative work i was doing on a postgres storage, but i ended up having too many problems getting any of the postgres adaptors to properly handle binary data (not in lob in a text field), bind variables, and calling of procs (converted over the OracleStorage to pl/pgsql procs) all at the same time. between the fact that a db -backed storage wasn't very useful in the first place and the difficulty in getting things working i shelved it.
psycopg should have no problems with binary data in a string. is uses *only* python strings, that are 8-bits clean and can also contain \0. as i said we still don't support callproc() but you can always execute("SELECT proc()") and fetch the results...
i was wondering if there are any plans for psycopg to go beyond the db-api and support more of the functionality of libpq, such as asynch requests, and notifications. (yes, i know the line about contributions, but i'm more curious about your plans for psycopg).
async requests are imho, a must. we'll begin to add extra stuff after 1.0. goals for 1.0 are *complete* DBAPI compliance (plus some little extensions, like cursor commits and the type system) and stability. after that we'll start working on an extended api to support some of the stuff in libpq.
one thing to not about ZPygre is that it synchronizes all db access. if
this is bad, because you can't rollback...
ok, i'll bite. why not?
by "synchronizes" i intended that it works in "autocommit", i.e., does not do a BEGIN for you. that way every execute() is automatically commited to the db and you can't rollback. pretty bad in zope, where one of the niciest things is the server doing a rollback (if i read the code right) when an error occurs in the current page.
i am very interested in drivers performance: has anybody set up a test of some kind to test this stuff? we use a simple test with 3 threads doing simultaneous INSERTs and SELECTs, creating lots of cursors and psycopg seems pretty fast but i would better like some *real* test.
hmmm... i vaguely recall someone posting some comparisions between pygre and popy in the last month or two, you'll have to check the searchable archives.
ok. i'll go searching...
jokes apart, a new product can't mature if nobody use it because it is "still young", so try it out and send bug reports, thanx!
true, i'll try my attempts on a postgres storage on it and see what happens. although for something like a zodb storage the multiple connection thing is not useful since the zodb will synchronize access to the storage layer.
i'll see if i can convert my experiments into some unittests.
very nice. please report any problems and ask for features if you need particular ones. real-cases are always the best tests for software like drivers. thank you for your comment, federico -- Federico Di Gregorio MIXAD LIVE Chief of Research & Technology fog@mixadlive.com Debian GNU/Linux Developer & Italian Press Contact fog@debian.org Don't dream it. Be it. -- Dr. Frank'n'further
participants (2)
-
ender -
Federico Di Gregorio