Re: [Zope] ZServer Must Be Continually Re-Started: my Method to stop zope dead:
Hello This thread is going around since several years now (and zope version). I am aware of this problem because I encounter it since zope 0.9.xxx (or was it 1.0.xx??) :-); and I would love to have a solution... I only use zope basic functionality: Zserver + one postgreSQL adapter (Zpopyda at this time) and use some external methods , also some mailhost object. No Zclass or fancy products. I am personally thinking that ONE of the problem is that there is no time-out on the Zserver socket, I think that if for some reason a request get stuck, the thread is never released. If you use all the thread pool, your server doesn't reply anymore. Exercise (It's just a silly test I have done 5 mins ago): - my Zserver is configured to have a pool of 4 threads - I create en external method which loop - I create an DTML (hang_test) method which call this external method, then I make 4 copy ( hang_test1,hang_test2, etc..) - from a browser I call 2 of them: the requests get stuck as expected: I use the debug page to check them: (I have 2 long running threads at this time) 1)Opened:Fri May 11 09:12:13 2001 (488.23s) HTTP_ACCEPT_ENCODING : 'deflate, gzip, x-gzip, identity,*;q=0'. SERVER_PORT : '8080'. etc..... 2)Opened:Fri May 11 09:19:54 2001 (27.81s)..... 3)debug request. 4)free. - I block the zserver by calling the other hang_test3, hang_test4 methods. - that it. My Zope server is now bloqued since 15 minutes, I will see if it reply again later on. - eventualy (if I have some time) I will redo the same test using Zserver with a timeout socket library to see if it act the same... please understand that this is just a demonstration, I know that doing infinite loop is silly. In real life, one blocking condition I saw(I am almost sure of that, just almost) was the mailhost object: In case of empty(or bad, I don't remember) mail recipient, the request was blocking when it was doing the sendmail. This server was dying every few days. - my zope server is still dead: I have requests in my browser dated from 33 mins. have someone a solution? Gilles -----Original Message----- From: Ben Ocean <zope@thewebsons.com> To: zope@zope.org <zope@zope.org> Date: Friday, May 11, 2001 1:50 AM Subject: [Zope] ZServer Must Be Continually Re-Started
Hi; I'm still having problems with ZServer quitting on me on a whim. Can someone help me figure out how to trouble-shoot this? I just installed the latest distro, so it's not the *old bug* that plagued earlier versions. TIA, BenO
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Gilles Lavaux wrote:
I am personally thinking that ONE of the problem is that there is no time-out on the Zserver socket, I think that if for some reason a request get stuck, the thread is never released. If you use all the thread pool, your server doesn't reply anymore.
I'd second that...
- eventualy (if I have some time) I will redo the same test using Zserver with a timeout socket library to see if it act the same...
I'd be very interested to see the results :-)
please understand that this is just a demonstration, I know that doing infinite loop is silly. In real life, one blocking condition I saw(I am almost sure of that, just almost) was the mailhost object: In case of empty(or bad, I don't remember) mail recipient, the request was blocking when it was doing the sendmail. This server was dying every few days.
That could be a problem with your mail server, ours just comes back straight away and Zope raises an error.
have someone a solution?
I hope so, that could be what's causing our server to hang since the system involved can have some very logn running methods... cheers, Chris ChrisM: I cc'ed you on this because I have a hunch that it's something along these lines that's causing the hangs I mentioned on zope-dev. That said, would the python process usage drop down to nothing in the above scenario?
Chris Withers wrote:
Gilles Lavaux wrote:
I am personally thinking that ONE of the problem is that there is no time-out on the Zserver socket, I think that if for some reason a request get stuck, the thread is never released. If you use all the thread pool, your server doesn't reply anymore.
I'd second that...
There evidently already is a timeout (according to Jim). I think it's set to 30 minutes. I'm not sure where it is. It's possible it needs to be lowered.
ChrisM: I cc'ed you on this because I have a hunch that it's something along these lines that's causing the hangs I mentioned on zope-dev. That said, would the python process usage drop down to nothing in the above scenario?
No... at least I don't think so... if you have a long-running method, and some joker reloads that method 5 times (because they think it'll speed things up), it will chew up five threads, and whatever processing is happening will be happening five times. It's likely that the Python process would be working hard in this scenario... unless of course the method was long-running because it was IO-bound instead of CPU-bound... did requestprofiler turn anything up for you?
Hi Chrises ;), as far I can see from code, in http_server.py there is code to clean up zombies after 30 mins timeout but only on channel creation time. If all threads are eaten up within 30 mins there is no chance to initialize a new one and so no chance to do the cleanup. May be a maintenance-thread will help a bit? Ok, this does not help on the python thread switcher get stuck on external c functions problem. Regards Tino --On Freitag, 11. Mai 2001 08:04 -0400 Chris McDonough <chrism@digicool.com> wrote:
Chris Withers wrote:
Gilles Lavaux wrote:
I am personally thinking that ONE of the problem is that there is no time-out on the Zserver socket, I think that if for some reason a request get stuck, the thread is never released. If you use all the thread pool, your server doesn't reply anymore.
I'd second that...
There evidently already is a timeout (according to Jim). I think it's set to 30 minutes. I'm not sure where it is. It's possible it needs to be lowered.
ChrisM: I cc'ed you on this because I have a hunch that it's something along these lines that's causing the hangs I mentioned on zope-dev. That said, would the python process usage drop down to nothing in the above scenario?
No... at least I don't think so... if you have a long-running method, and some joker reloads that method 5 times (because they think it'll speed things up), it will chew up five threads, and whatever processing is happening will be happening five times. It's likely that the Python process would be working hard in this scenario... unless of course the method was long-running because it was IO-bound instead of CPU-bound... did requestprofiler turn anything up for you?
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Hi Gilles, --On Freitag, 11. Mai 2001 10:57 +0200 Gilles Lavaux <gilles.lavaux@esrin.esa.it> wrote:
Hello
This thread is going around since several years now (and zope version). I am aware of this problem because I encounter it since zope 0.9.xxx (or was it 1.0.xx??) :-); and I would love to have a solution... I only use zope basic functionality: Zserver + one postgreSQL adapter (Zpopyda at this time) and use some external methods , also some mailhost object. No Zclass or fancy products.
I am personally thinking that ONE of the problem is that there is no time-out on the Zserver socket, I think that if for some reason a request get stuck, the thread is never released. If you use all the thread pool, your server doesn't reply anymore.
Thats true. Even worser if the request ends in an external c-library (like for database-access) I believe the whole thread engine get stuck at once since the calling python instruction never ends and this way no 120 (or whatever once might have configured) instrucktions on this particular thread can be executed to go over to the next. We therefore need a solution to handle this situation ansynchronously or can give the python interpreter a proper wait state for this thread so it can continues to process the other threads. Regards Tino
Exercise (It's just a silly test I have done 5 mins ago): - my Zserver is configured to have a pool of 4 threads - I create en external method which loop - I create an DTML (hang_test) method which call this external method, then I make 4 copy ( hang_test1,hang_test2, etc..) - from a browser I call 2 of them: the requests get stuck as expected: I use the debug page to check them: (I have 2 long running threads at this time) 1)Opened:Fri May 11 09:12:13 2001 (488.23s) HTTP_ACCEPT_ENCODING : 'deflate, gzip, x-gzip, identity,*;q=0'. SERVER_PORT : '8080'. etc..... 2)Opened:Fri May 11 09:19:54 2001 (27.81s)..... 3)debug request. 4)free.
- I block the zserver by calling the other hang_test3, hang_test4 methods. - that it. My Zope server is now bloqued since 15 minutes, I will see if it reply again later on. - eventualy (if I have some time) I will redo the same test using Zserver with a timeout socket library to see if it act the same...
please understand that this is just a demonstration, I know that doing infinite loop is silly. In real life, one blocking condition I saw(I am almost sure of that, just almost) was the mailhost object: In case of empty(or bad, I don't remember) mail recipient, the request was blocking when it was doing the sendmail. This server was dying every few days.
- my zope server is still dead: I have requests in my browser dated from 33 mins.
have someone a solution?
Gilles
-----Original Message----- From: Ben Ocean <zope@thewebsons.com> To: zope@zope.org <zope@zope.org> Date: Friday, May 11, 2001 1:50 AM Subject: [Zope] ZServer Must Be Continually Re-Started
Hi; I'm still having problems with ZServer quitting on me on a whim. Can someone help me figure out how to trouble-shoot this? I just installed the latest distro, so it's not the *old bug* that plagued earlier versions. TIA, BenO
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
At 10:57 AM 5/11/2001 +0200, you wrote:
Hello
This thread is going around since several years now (and zope version). I am aware of this problem because I encounter it since zope 0.9.xxx (or was it 1.0.xx??) :-); and I would love to have a solution... I only use zope basic functionality: Zserver + one postgreSQL adapter (Zpopyda at this time) and use some external methods , also some mailhost object. No Zclass or fancy products.
I am personally thinking that ONE of the problem is that there is no time-out on the Zserver socket, I think that if for some reason a request get stuck, the thread is never released. If you use all the thread pool, your server doesn't reply anymore.
Would this cause the server to quit, as if I had entered the command ./stop? Also, I could write a cron script that checks to see if Zope is alive and if not re-starts it, and I could run that script every minute if I wanted to, but that seems like a waste of resources and just plain silly. Any better ideas? BenO
Hi Ben, no this is completly different to your problem. Dont know why these threads are intermixed. The problem below is zope not responding to requests but still beeing running. Your problem is different: zope stops running instantly. So I expect something in your environment and/or python version. Do you have a plain install or did you install additionally products with zope? Regards Tino Wildenhain --On Freitag, 11. Mai 2001 05:12 -0700 Ben Ocean <zope@thewebsons.com> wrote:
At 10:57 AM 5/11/2001 +0200, you wrote:
Hello
This thread is going around since several years now (and zope version). I am aware of this problem because I encounter it since zope 0.9.xxx (or was it 1.0.xx??) :-); and I would love to have a solution... I only use zope basic functionality: Zserver + one postgreSQL adapter (Zpopyda at this time) and use some external methods , also some mailhost object. No Zclass or fancy products.
I am personally thinking that ONE of the problem is that there is no time-out on the Zserver socket, I think that if for some reason a request get stuck, the thread is never released. If you use all the thread pool, your server doesn't reply anymore.
Would this cause the server to quit, as if I had entered the command ./stop? Also, I could write a cron script that checks to see if Zope is alive and if not re-starts it, and I could run that script every minute if I wanted to, but that seems like a waste of resources and just plain silly. Any better ideas? BenO
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
At 03:25 PM 5/11/2001 +0200, you wrote:
Hi Ben,
zope stops running instantly.
So I expect something in your environment and/or python version. Do you have a plain install or did you install additionally products with zope?
Well, I've built a number of products into the Zope Products folder *after* the installation. I may be including some of the regularly installed products, but here's the list:
Chameleon/Chameleon Skin CookielessSessionManager FSSession Local File System MailHost PHP File Parsed XML Portal RAM Cache Manager Script (Python) Set Access Rule Squishdot Site Tiny Tables Plus User Folder Version Virtual Host Monster Vocabulary Wizard/ Wizard Page XML Document XML Widget/XML Widget Manager Z MySQL Database Connection Z SQL Method Z Search Interface <<< It could be the python installation. I'm running two versions: 1.5 and 1.5.2. When I type *which python* at a command prompt I get this: /usr/bin/python and within this same folder is python1.5, whereas *locate python* reveals these directories:
/apache/vhosts/Python-1.5.2 /apache/Zope-2.3.2-linux2-x86/lib/python1.5 /usr/bin/Python-1.5.2
When I type *python* at the command line I find the version I'm running is 1.5.2.
So, perhaps that's where the problem is. How would you suggest I trouble-shoot this? Are there error logs in python? I wouldn't mind killing a version of python, but I don't want to screw anything else up, either. I'm running STUPID_LOG_FILE but catching no errors. Also, with regard to my *environment*, how would you suggest I search for potential problems? I'm running RH6.2. The only other bugs I'm working on in my Linux environment are a problem with warm reboot not reading the httpd.conf file and noticing a bunch of httpsd restarts early in the morning hours when I run *ps aux* that turn out to all be dead processes. TIA, BenO
participants (5)
-
Ben Ocean -
Chris McDonough -
Chris Withers -
Gilles Lavaux -
Tino Wildenhain