Strangeness with Zope and Postgres
For months now, we have been noticing that every once in awhile zope just appears to freeze up. I go and look at the zope processes and they are all sleeping. The only way to fix the problem was to restart zope, hence I didn't think about the fact that it might be the database. I have been monitoring it and assumed it was something funny in Zope, but something that was mentioned on this list made me start thinking that it could be something funny happening with the database. We use postgres as our dbms and our site is heavily database instensive. Sure enough, when ever the site appears to freeze, I go and look at the database threads and one thread has a status of "UPDATE WAITING" and the the others are sitting in the normal idle in transaction status. The server is idle (2% or less cpu) and the drives are as normal, silent. The server is doing absolutely nothing. I also found out how to lock it up every time. I have a specific page that does some intensive database work including some updates to some preference tables as well as some reads of some large tables, etc. If I go to that page and then hit F5 twice in succession, it locks up the system every time requiring a zope restart. Now I am starting to think that we are running into a concurrency issue with postgres and transactions. I was wondering if anyone with a bit more experience might be able to shed some more light on the situation and/or where I might look to further debug the situation. My hunch is that the following happens: 1. First request comes in. 2. User presses F5 before the first transaction finished. 3. Browser drops first connection and reconnects. 4. Zope starts to write to the socket, notices that the browser is no longer connected. 5. Zope aborts the transaction. 6. BUT, the database never get's a rollback and simply waits in the transaction for ever. 7. Unfortunately, this must also cause some locks that don't allow any other thread to do any work. 8. Restarting Zope closes those db connections and postgres roles the transaction back. Does anyone have any further suggestions on how to debug this further or to stop the problem? -Chris RH 7.2 Zope 2.4.1 ZPoPyDA Postgres 7.1.3 ------------------------------ Chris Kratz chris.kratz@vistashare.com
Hi. We had this problem too. The thing is with PoPy. I think this is solved in a newer version of PoPy. Please mail me again if a newer version won't help, then i have to take a look how we solved this, but we did, so go on and ask me. Christian On Thu, Nov 01, 2001 at 04:43:10PM -0500, Chris Kratz wrote:
For months now, we have been noticing that every once in awhile zope just appears to freeze up. I go and look at the zope processes and they are all sleeping. The only way to fix the problem was to restart zope, hence I didn't think about the fact that it might be the database. I have been monitoring it and assumed it was something funny in Zope, but something that was mentioned on this list made me start thinking that it could be something funny happening with the database.
We use postgres as our dbms and our site is heavily database instensive. Sure enough, when ever the site appears to freeze, I go and look at the database threads and one thread has a status of "UPDATE WAITING" and the the others are sitting in the normal idle in transaction status. The server is idle (2% or less cpu) and the drives are as normal, silent. The server is doing absolutely nothing.
I also found out how to lock it up every time. I have a specific page that does some intensive database work including some updates to some preference tables as well as some reads of some large tables, etc. If I go to that page and then hit F5 twice in succession, it locks up the system every time requiring a zope restart.
Now I am starting to think that we are running into a concurrency issue with postgres and transactions. I was wondering if anyone with a bit more experience might be able to shed some more light on the situation and/or where I might look to further debug the situation.
My hunch is that the following happens: 1. First request comes in. 2. User presses F5 before the first transaction finished. 3. Browser drops first connection and reconnects. 4. Zope starts to write to the socket, notices that the browser is no longer connected. 5. Zope aborts the transaction. 6. BUT, the database never get's a rollback and simply waits in the transaction for ever. 7. Unfortunately, this must also cause some locks that don't allow any other thread to do any work. 8. Restarting Zope closes those db connections and postgres roles the transaction back.
Does anyone have any further suggestions on how to debug this further or to stop the problem?
-Chris
RH 7.2 Zope 2.4.1 ZPoPyDA Postgres 7.1.3 ------------------------------ Chris Kratz chris.kratz@vistashare.com
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
-- Christian Theune - ct@gocept.com gocept gmbh & co.kg - schalaunische strasse 6 - 06366 koethen/anhalt tel.+49 3496 3099112 - fax.+49 3496 3099118 mob. - 0178 48 33 981 reduce(lambda x,y:x+y,[chr(ord(x)^42) for x in 'zS^BED\nX_FOY\x0b'])
We have this problem also. We are using popy 2.0.7 with postgres 7.1.3. Where is no newer stable version available at the moment. Regards Lutz Steinborn On Fri, 2 Nov 2001 11:1 3:37 +0100 Christian Theune <ct@gocept.com> wrote:
Hi.
We had this problem too. The thing is with PoPy. I think this is solved in a newer version of PoPy.
Please mail me again if a newer version won't help, then i have to take a look how we solved this, but we did, so go on and ask me.
Christian
On Thu, Nov 01, 2001 at 04:43:10PM -0500, Chris Kratz wrote:
For months now, we have been noticing that every once in awhile zope just appears to freeze up. I go and look at the zope processes and they are all sleeping. The only way to fix the problem was to restart zope, hence I didn't think about the fact that it might be the database. I have been monitoring it and assumed it was something funny in Zope, but something that was mentioned on this list made me start thinking that it could be something funny happening with the database.
We use postgres as our dbms and our site is heavily database instensive. Sure enough, when ever the site appears to freeze, I go and look at the database threads and one thread has a status of "UPDATE WAITING" and the the others are sitting in the normal idle in transaction status. The server is idle (2% or less cpu) and the drives are as normal, silent. The server is doing absolutely nothing.
I also found out how to lock it up every time. I have a specific page that does some intensive database work including some updates to some preference tables as well as some reads of some large tables, etc. If I go to that page and then hit F5 twice in succession, it locks up the system every time requiring a zope restart.
Now I am starting to think that we are running into a concurrency issue with postgres and transactions. I was wondering if anyone with a bit more experience might be able to shed some more light on the situation and/or where I might look to further debug the situation.
My hunch is that the following happens: 1. First request comes in. 2. User presses F5 before the first transaction finished. 3. Browser drops first connection and reconnects. 4. Zope starts to write to the socket, notices that the browser is no longer connected. 5. Zope aborts the transaction. 6. BUT, the database never get's a rollback and simply waits in the transaction for ever. 7. Unfortunately, this must also cause some locks that don't allow any other thread to do any work. 8. Restarting Zope closes those db connections and postgres roles the transaction back.
Does anyone have any further suggestions on how to debug this further or to stop the problem?
-Chris
RH 7.2 Zope 2.4.1 ZPoPyDA Postgres 7.1.3 ------------------------------ Chris Kratz chris.kratz@vistashare.com
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
-- Christian Theune - ct@gocept.com gocept gmbh & co.kg - schalaunische strasse 6 - 06366 koethen/anhalt tel.+49 3496 3099112 - fax.+49 3496 3099118 mob. - 0178 48 33 981
reduce(lambda x,y:x+y,[chr(ord(x)^42) for x in 'zS^BED\nX_FOY\x0b'])
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
-- Lutz Steinborn mailto:l.steinborn@4c-ag.de 4c AG Tel +49 6092 999592 Stefan-George-Ring 22-24 Fax +49 89 99341 399 81929 München http://www.4c-shopping.de
We also had this problem. We moved to psycopg and these problems went away. However, it does occasionally happen when a long running zsql method is stopped and then reloaded from the browser. Psycopg performs much better in IMHO that popy because its thread transaction is much tighter. It seems that the commit and rollbacks are done from the zsql thread and then closed rather to let run until its default thread rollover time has passed. Using the detailed logging will show you more info. You'll notice when you come across a hung thread, the log shows a few transactions that were piped into the same thread. One way was to bring the thread rolltime (in z2.py) down to a second, but that seems pretty stupid. Regards, Paul Zwarts -----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org] On Behalf Of Lutz Steinborn Sent: Friday, November 02, 2001 11:42 AM To: Christian Theune Cc: zope@zope.org Subject: Re: [Zope] Strangeness with Zope and Postgres We have this problem also. We are using popy 2.0.7 with postgres 7.1.3. Where is no newer stable version available at the moment. Regards Lutz Steinborn On Fri, 2 Nov 2001 11:1 3:37 +0100 Christian Theune <ct@gocept.com> wrote:
Hi.
We had this problem too. The thing is with PoPy. I think this is solved in a newer version of PoPy.
Please mail me again if a newer version won't help, then i have to take a look how we solved this, but we did, so go on and ask me.
Christian
On Thu, Nov 01, 2001 at 04:43:10PM -0500, Chris Kratz wrote:
For months now, we have been noticing that every once in awhile zope just appears to freeze up. I go and look at the zope processes and they are all sleeping. The only way to fix the problem was to restart zope, hence I didn't think about the fact that it might be the database. I have been monitoring it and assumed it was something funny in Zope, but something that was mentioned on this list made me start thinking that it could be something funny happening with the database.
We use postgres as our dbms and our site is heavily database instensive. Sure enough, when ever the site appears to freeze, I go and look at the database threads and one thread has a status of "UPDATE WAITING" and the the others are sitting in the normal idle in transaction status. The server is idle (2% or less cpu) and the drives are as normal, silent. The server is doing absolutely nothing.
I also found out how to lock it up every time. I have a specific page that does some intensive database work including some updates to some preference tables as well as some reads of some large tables, etc. If I go to that page and then hit F5 twice in succession, it locks up the system every time requiring a zope restart.
Now I am starting to think that we are running into a concurrency issue with postgres and transactions. I was wondering if anyone with a bit more experience might be able to shed some more light on the situation and/or where I might look to further debug the situation.
My hunch is that the following happens: 1. First request comes in. 2. User presses F5 before the first transaction finished. 3. Browser drops first connection and reconnects. 4. Zope starts to write to the socket, notices that the browser is no longer connected. 5. Zope aborts the transaction. 6. BUT, the database never get's a rollback and simply waits in the transaction for ever. 7. Unfortunately, this must also cause some locks that don't allow any other thread to do any work. 8. Restarting Zope closes those db connections and postgres roles the transaction back.
Does anyone have any further suggestions on how to debug this further or to stop the problem?
-Chris
RH 7.2 Zope 2.4.1 ZPoPyDA Postgres 7.1.3 ------------------------------ Chris Kratz chris.kratz@vistashare.com
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
-- Christian Theune - ct@gocept.com gocept gmbh & co.kg - schalaunische strasse 6 - 06366 koethen/anhalt tel.+49 3496 3099112 - fax.+49 3496 3099118 mob. - 0178 48 33 981
reduce(lambda x,y:x+y,[chr(ord(x)^42) for x in 'zS^BED\nX_FOY\x0b'])
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
-- Lutz Steinborn mailto:l.steinborn@4c-ag.de 4c AG Tel +49 6092 999592 Stefan-George-Ring 22-24 Fax +49 89 99341 399 81929 München http://www.4c-shopping.de _______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Hello Christian, I went back to check the version we are using and it appears that we are using version 3.0 beta 1. Perhaps the 2.0.8 version is newer then the beta? Source forge [1] states that the updates since 3.0 have all been to the 2.0 version line and the latest is 2.0.7 and was updated on August 14, 2001. On zope.org [2], the latest is version 2.0.8 which was released on August 27. Do you have any idea what version you are using? -Chris [1] http://sourceforge.net/projects/popy [2] http://www.zope.org/Members/tm/ZPoPyDA ------------------------------ Chris Kratz chris.kratz@vistashare.com ----- Original Message ----- From: "Christian Theune" <ct@gocept.com> To: <zope@zope.org> Sent: Friday, November 02, 2001 5:13 AM Subject: Re: [Zope] Strangeness with Zope and Postgres Hi. We had this problem too. The thing is with PoPy. I think this is solved in a newer version of PoPy. Please mail me again if a newer version won't help, then i have to take a look how we solved this, but we did, so go on and ask me. Christian
participants (4)
-
Chris Kratz -
Christian Theune -
Lutz Steinborn -
Paul Zwarts