RE: [Zope] Running ZopeHTTPServer and PCGI together = a bad thing ?
-----Original Message----- From: Tony McDonald [mailto:tony.mcdonald@ncl.ac.uk] Sent: Thursday, June 03, 1999 4:05 AM To: Michel Pelletier; Zope List Subject: RE: [Zope] Running ZopeHTTPServer and PCGI together = a bad thing ?
Yes there is some rather expensive forking overhead that must be done to get from Apache to the Zope long running process. This is startup overhead only, once the connection gets to Zope the 'time to complete' will be identical.
Is this a fork for *every* request, eg every link I select. In that case, isn't it pretty expensive overall?
A wafer-thin CGI program is forked for every request, this is used to connect up Apache and the Zope long running process. So yes, this expensive fork happens for every request. It's no different, however, then other CGI overhead. One of these days, someone is going to write a mod_pcgi module for Apache, and take that expensive fork out. For light loads, it's really not noticeable, but for heavy loads, one fork per request gets heavy. ZServer is noticably faster in this regard than Apache.
I've used bbb.py from the Zope 2 distribution to create XML transactions for 0.. 9211505 and 11844613... 11847812 (but that still leaves 9211505..11844613 in limbo)
unnh... so what are my options now?
Well, first realize that corruption is corruption, there may be no way to get some of your info back from the lala lands. If two process writes conflicted with each other, one wins, and one looses (often the winner however does not win completely, but leave a corrupted record behind).
1) Chop the database at location 11844613 (but then what about the junk at 9211505)
Hmm.. first off, does it crash your Zope when you try and use it correctly? I would try wacking off the last 3K or so bytes starting at the last corrupted record, and then try exporting some stuff out, look for the point in managing the site where it breaks, this is probably where it encouters the broken record.
2) export as much as possible to .bbe files and reimport into a clean database?
If you can export these objects, try this at least to preserve your information somewhere.
3) use the bbb.py from the Zope 2 distribution to get transactions in XML (but then how do I recreate the database - bbb.py doesn't mention creation of a database)?
Uh yeah, the whole XML thing isn't complete yet. The dump to XML is really for database inspection, not manipulation or transportation, for the moment. This will probably change. -Michel
4) give up? ... nope, that's not an option. Zope is just so damn useful.
I hate to say this, but HELP! Tone
Michel, Thanks for the reply...
A wafer-thin CGI program is forked for every request, this is used to connect up Apache and the Zope long running process. So yes, this expensive fork happens for every request. It's no different, however, then other CGI overhead. One of these days, someone is going to write a mod_pcgi module for Apache, and take that expensive fork out. For light loads, it's really not noticeable, but for heavy loads, one fork per request gets heavy. ZServer is noticably faster in this regard than Apache.
I didn't realise that the process was so lightweight, but as it seems that ZServer is the future (because ZopeHTTPServer is no longer being developed), then I'm off to ZServer land...
I've used bbb.py from the Zope 2 distribution to create XML transactions for 0.. 9211505 and 11844613... 11847812 (but that still leaves 9211505..11844613 in limbo)
unnh... so what are my options now?
Well, first realize that corruption is corruption, there may be no way to get some of your info back from the lala lands. If two process writes conflicted with each other, one wins, and one looses (often the winner however does not win completely, but leave a corrupted record behind).
That's fair enough. I consider this a 'battle scar' gained on the proving grounds of Object-Orientated web delivery and will wear it with pride.
1) Chop the database at location 11844613 (but then what about the junk at 9211505)
Hmm.. first off, does it crash your Zope when you try and use it correctly? I would try wacking off the last 3K or so bytes starting at the last corrupted record, and then try exporting some stuff out, look for the point in managing the site where it breaks, this is probably where it encouters the broken record.
Actually, it doesn't crash when I use Zope correctly (although we've had some long response times on the server recently).
2) export as much as possible to .bbe files and reimport into a clean database?
If you can export these objects, try this at least to preserve your information somewhere.
I'm doing this at the moment. I'm getting most of the data out and shifting it to another box - where I am NOT running PCGI at all (so no chance of doing the same mistake again). I've had some errors importing, but I'm breaking the (broken) site down into bits that do import and doing it that way. I'll then start chopping bits of the end of the database and get as much as possible out that way.
3) use the bbb.py from the Zope 2 distribution to get transactions in XML (but then how do I recreate the database - bbb.py doesn't mention creation of a database)?
Uh yeah, the whole XML thing isn't complete yet. The dump to XML is really for database inspection, not manipulation or transportation, for the moment. This will probably change.
I'd like to make a BIG request that you do consider it as a manipulation/transportation vehicle as it would certainly put some peoples' minds at rest about the safety of their data. Once again, many thanks for the reply - at least I know what went wrong... tone. ------ Dr Tony McDonald, FMCC, Networked Learning Environments Project The Medical School, Newcastle University Tel: +44 191 222 5888 Fingerprint: 3450 876D FA41 B926 D3DD F8C3 F2D0 C3B9 8B38 18A2
Michel Pelletier wrote:
Is this a fork for *every* request, eg every link I select. In that case, isn't it pretty expensive overall?
A wafer-thin CGI program is forked for every request, this is used to connect up Apache and the Zope long running process. So yes, this expensive fork happens for every request. It's no different, however, then other CGI overhead. One of these days, someone is going to write a mod_pcgi module for Apache, and take that expensive fork out. For light loads, it's really not noticeable, but for heavy loads, one fork per request gets heavy. ZServer is noticably faster in this regard than Apache.
I'm fighting with exactly this right now :-) I also get a lot of Zope zombie-processes when the server is under heavy load, but have no idea why?? BTW: When I restart Zope it happens that 4 "python Zope.."-processes start at the same time (I have to kill them ASAP and next time the Server starts just once. Kind of strange behavior... Is there somebody working on mod_pcgi or is this just a wish? Tom -- Tom Schwaller http://www.linux-magazin.de/
Tom Schwaller wrote:
I also get a lot of Zope zombie-processes when the server is under heavy load, but have no idea why?? BTW: When I restart Zope it happens that 4 "python Zope.."-processes start at the same time (I have to kill them ASAP and next time the Server starts just once. Kind of strange behavior...
Is there somebody working on mod_pcgi or is this just a wish?
It's definitely on the list. It has been agreed that launching processes should not be a *requirement* of pcgi, especially for mod_pcgi. Regards, Jeff Bauer Rubicon, Inc.
Michel Pelletier wrote:
(snip)
A wafer-thin CGI program is forked for every request, this is used to connect up Apache and the Zope long running process. So yes, this expensive fork happens for every request.
It's not *that* expensive. This is a very thin wrapper.
It's no different, however, then other CGI overhead.
It's reallu much lower than typical CGI overhead. Most CGI scripts are written using interpreters (e.g. sh, perl, Python). Cranking up an interpreter is much more expensive than cranking up the pcgi_wrapper.
One of these days, someone is going to write a mod_pcgi module for Apache, and take that expensive fork out. For light loads, it's really not noticeable, but for heavy loads, one fork per request gets heavy.
I think a site could sustain a million hits a day using pcgi on moderately fast hardware.
ZServer is noticably faster in this regard than Apache.
True. An an apache module would definately be a good thing. BTW, ZServer supports PCGI, so you *can* use Apache+PCGI+ZServer. This is useful, for example, if you want to use Apache to do SSL while using ZServer for other HTTP requests and FTP. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
participants (5)
-
Jeff Bauer -
Jim Fulton -
Michel Pelletier -
Tom Schwaller -
Tony McDonald