Zope + Apache on Quad Debian machine
Yellow, I'm using Zope+Apache on a 4 xeon's/4GB ram machine running Debian. I've noticed that the CPU's never go beyond 30% top occupation... but on rush hours the site takes too long to load... I've been reading about process affinity and how it could speed up everything by making zope run on 1 CPU, Apache on another and so on... Has any1 tried this before? Can you point me to some documentation? What's your experience? is it true that not doing this the 4 cpu's will not be 100% used? Cheers -- Hugo Ramos - ramosh@gmail.com http://www.orkut.com/Profile.aspx?uid=10082105466310142690
Hugo Ramos wrote:
Yellow,
I'm using Zope+Apache on a 4 xeon's/4GB ram machine running Debian. I've noticed that the CPU's never go beyond 30% top occupation... but on rush hours the site takes too long to load...
I've been reading about process affinity and how it could speed up everything by making zope run on 1 CPU, Apache on another and so on...
Has any1 tried this before? Can you point me to some documentation? What's your experience? is it true that not doing this the 4 cpu's will not be 100% used?
You can set up zope with zeo and then assign 3 zeo clients on different processors that shares the zeo server. That should get you a lot of the way. -- hilsen/regards Max M, Denmark http://www.mxm.dk/ IT's Mad Science Phone: +45 66 11 84 94 Mobile: +45 29 93 42 96
Yes... But I'm looking for more specific answers about how to create the affinity between 1 of the zope processes and the CPU n.2, for example... How to make that a permanent choice? Cheers Hugo On 3/15/06, Max M <maxm@mxm.dk> wrote:
Hugo Ramos wrote:
Yellow,
I'm using Zope+Apache on a 4 xeon's/4GB ram machine running Debian. I've noticed that the CPU's never go beyond 30% top occupation... but on rush hours the site takes too long to load...
I've been reading about process affinity and how it could speed up everything by making zope run on 1 CPU, Apache on another and so on...
Has any1 tried this before? Can you point me to some documentation? What's your experience? is it true that not doing this the 4 cpu's will not be 100% used?
You can set up zope with zeo and then assign 3 zeo clients on different processors that shares the zeo server.
That should get you a lot of the way.
--
hilsen/regards Max M, Denmark
http://www.mxm.dk/ IT's Mad Science
Phone: +45 66 11 84 94 Mobile: +45 29 93 42 96
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-- Hugo Ramos - ramosh@gmail.com http://www.orkut.com/Profile.aspx?uid=10082105466310142690 http://otugga.blogspot.com/ http://otuggapoesia.blogspot.com/
On 15 Mar 2006, at 21:11, Hugo Ramos wrote:
Yes... But I'm looking for more specific answers about how to create the affinity between 1 of the zope processes and the CPU n.2, for example... How to make that a permanent choice?
This question has nothing to do with Zope. You will need to find out from your operating system documentation or from a forum that deals with it. jens
"Has any1 tried this before? Can you point me to some documentation? What's your experience? is it true that not doing this the 4 cpu's will not be 100% used?" I think these questions are very ZOPE RELATED since I asked them in a Zope mailing list don't you think? I'm asking about Zope user/administrator experience related to Zope use on a quad machine!!! Of course if some1 tells me how to do it using OS related stuff this is STILL Zope related! Hey... but thank you Jens for spending all that time writing an email that doesn't help any1 at all...!!! Cheers Hugo On 3/15/06, Jens Vagelpohl <jens@dataflake.org> wrote:
On 15 Mar 2006, at 21:11, Hugo Ramos wrote:
Yes... But I'm looking for more specific answers about how to create the affinity between 1 of the zope processes and the CPU n.2, for example... How to make that a permanent choice?
This question has nothing to do with Zope. You will need to find out from your operating system documentation or from a forum that deals with it.
jens
-- Hugo Ramos - ramosh@gmail.com http://www.orkut.com/Profile.aspx?uid=10082105466310142690
Hugo Ramos schrieb:
"Has any1 tried this before? Can you point me to some documentation? What's your experience? is it true that not doing this the 4 cpu's will not be 100% used?"
I think these questions are very ZOPE RELATED since I asked them in a Zope mailing list don't you think? I'm asking about Zope user/administrator experience related to Zope use on a quad machine!!!
Of course if some1 tells me how to do it using OS related stuff this is STILL Zope related!
Hey... but thank you Jens for spending all that time writing an email that doesn't help any1 at all...!!!
Hahaha. And the time you are weeping would have been better spent on googling <your os> <cpu afinity> - and err. btw. also check google for "multiple use of exclamation marks". Btw. I dont believe the "cpu afinity" buys you anything in zope context. Try it and test it in your setup with your application. --Tino
On 3/16/06, Tino Wildenhain <tino@wildenhain.de> wrote:
Hey... but thank you Jens for spending all that time writing an email that doesn't help any1 at all...!!!
check google for "multiple use of exclamation marks".
Or look up Terry Pratchett quotes on www.lspace.org: '"Multiple exclamation marks," he went on, shaking his head, "are a sure sign of a diseased mind."' -- Eric "Five exclamation marks, the sure sign of an insane mind." -- Reaper Man "And all those exclamation marks, you notice? Five? A sure sign of someone who wears his underpants on his head." -- Maskerade Terry doesn't like multiple exclamation marks. -- Martijn Pieters
Martijn Pieters said the following on 2006-03-16 12:25:
On 3/16/06, Tino Wildenhain <tino@wildenhain.de> wrote:
Hey... but thank you Jens for spending all that time writing an email that doesn't help any1 at all...!!!
check google for "multiple use of exclamation marks".
Or look up Terry Pratchett quotes on www.lspace.org:
<snip> Martinj, Chris and others giving nonsential answers: Don't you guys have work to do instead of playing bullies on the newbies?. To be honest, I find arrogant and pointless answers like yours way more annoying than a whole bunch of newbies asking more or less cluess questions. I see a lot of this newbie bashing increasing; in fact it has been some years since the zope list was a good example on how to treat newbies politely and point them to the prope answers. Sure, we all have loads of work and are way too stressed, we don't get enough sleep, yada, yada, yada... I fail to see how that is a newbie's fault, so no need to take it out on them. For the benefit of all those that have tiny, tiny memory spans, the question of Zope's processes necessity to be CPU-bound is a horse that was resolved a couple of years ago (2002). It turns out that it is not a Zope problem ("Oh, the Horror. He's bringing a non-zope question to the list!"). It is pythons "fault", if you are grumpy old fart, or it is a "feature" if you are having a good day. The need for threaded python processes to be CPU bound (yes, Zope is threaded, in case you have forgotten) arises from the fact that python has the GIL (Global Interpreter Lock). For background see: http://www.zope.org/Members/glpb/solaris and http://www.zope.org/Members/glpb/solaris/report_ps As far as I know, the GIL is still there and won't go away any time soon. Today, the only change from 2002 is that there is a heck of a lot more multi-CPU machines than there were back in 2002. So, to make a long story short: on linux there is a userspace tools called taskset: http://linuxcommand.org/man_pages/taskset1.html http://aplawrence.com/Basics/taskset.html if there is no such command for your particular distro/kernelversion, look no further than: http://directory.fsf.org/all/schedutils.html http://rlove.org/ http://rlove.org/schedutils/ ftp://ftp.kernel.org/pub/linux/utils/util-linux/ I cannot tell you wich version to use, but I think you get the idea. I do believe that some distros allready include taskset in them. So you have to write a wrapper script around zopectl (you are running a newer versionof zope, aren't you?) where you do this. It has worked like a charm for us since 2002 (well, almost - since schedutils were new and cool back in 2002 we had regular fights with the sysadmins guys regarding the need to introduce "non-redhat-aproved-kernel-hacks" into the production systems. Nowadays taskset and related tools are included in the later RHE releases). Regarding all other advises you have gotten so far (get more memory, look at your disks, get a life, etc), I am sure that they have merits, but as far as I can see, they don't actually solve anything at all. The fact that the issue of the GIL is not more prominent in the Zope worlds, I think is because relatively few zopistas are aware that there is a problem; mostly, because not so many run multicpu-boxes in production, and also because of attitude, I suppose: "since I don't have a problem with it, I don't care about it, and I'll inform you so." /dario - yes... a bit annoyed at all the "help" some folks give... -- -- ------------------------------------------------------------------- Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech. Lyrics applied to programming & application design: "emancipate yourself from mental slavery" - redemption song, b. marley
Dario Lopez-Kästen schrieb: ...
The fact that the issue of the GIL is not more prominent in the Zope worlds, I think is because relatively few zopistas are aware that there is a problem; mostly, because not so many run multicpu-boxes in production, and also because of attitude, I suppose: "since I don't have a problem with it, I don't care about it, and I'll inform you so."
/dario - yes... a bit annoyed at all the "help" some folks give...
Otoh, I have yet to see the figures showing the CPU afinity buys you anything in reality. We know the GIL, thats for sure but I never saw a measureable difference binding a process to a CPU (which is also highly depending on the OS scheduler) I can imagine usefull applications of this - and thats for tiny computing intense applications where you can assume the CPU afinity keeps the program in the CPU cache. Zope isnt by far a "tiny" application compared to the cache size of actual CPUs. ++Tino PS: yes and I dont think newbs or anybody should easily get angry if they dont get the answer they would like to read...
Tino Wildenhain said the following on 2006-03-21 14:51:
Otoh, I have yet to see the figures showing the CPU afinity buys you anything in reality. We know the GIL, thats for sure but I never saw a measureable difference binding a process to a CPU (which is also highly depending on the OS scheduler)
for us, it makes all the difference between "zope sucks, why do we bother with this non-sense, non-standard, butt-slow appserver, and use Java or PHP instead" and "nice, zope based solutions are really nice, not only feature wise, but also speedy. And they are clusterable too, neat!" /dario -- -- ------------------------------------------------------------------- Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech. Lyrics applied to programming & application design: "emancipate yourself from mental slavery" - redemption song, b. marley
On 3/21/06, Dario Lopez-Kästen <dario@ita.chalmers.se> wrote:
Tino Wildenhain said the following on 2006-03-21 14:51:
Otoh, I have yet to see the figures showing the CPU afinity buys you anything in reality. We know the GIL, thats for sure but I never saw a measureable difference binding a process to a CPU (which is also highly depending on the OS scheduler)
for us, it makes all the difference between "zope sucks, why do we bother with this non-sense, non-standard, butt-slow appserver, and use Java or PHP instead" and "nice, zope based solutions are really nice, not only feature wise, but also speedy. And they are clusterable too, neat!"
/dario
Dario, Do you have any kind of comparison numbers of using CPU affinity vs not for your particular case? Also, are you using ZEO or not? It's not that I don't believe you when you say it matters a lot for you. I do believe you. Like Tino, I'm just generally interested in how much it matters in measurable terms. I can imagine there are a number of factors determining how much it matters, like Zope app/workload as well as the underlying hardware architecture (how big of a penalty is it to synchronize cache pages between CPUs) and the OS CPU scheduler as Tino mentioned. Jeff D p.s. I agree with the rest of your sentiments about newbie bashing. Unfortunately, it seems to be a popular past time among some of the l33ts on a bunch of the lists I monitor these days.
Jeff Donsbach said the following on 2006-03-21 17:26:
Dario, Do you have any kind of comparison numbers of using CPU affinity vs not for your particular case? Also, are you using ZEO or not? It's not that I don't believe you when you say it matters a lot for you. I do believe you. Like Tino, I'm just generally interested in how much it matters in measurable terms. I can imagine there are a number of factors determining how much it matters, like Zope app/workload as well as the underlying hardware architecture (how big of a penalty is it to synchronize cache pages between CPUs) and the OS CPU scheduler as Tino mentioned.
To be honest, we have never had the inclination to do much of research in this area. Our situation has mostly been such that we experience horrible delays in zope-responsiveness in testing that vanish as soon as we use taskset. This is on both Solaris (with the equivalent of taskset there) and Linux . Since then (and for other reasons), we have migrated all of the central servers we manage from Solaris to Linux, so I cannot give any more input about Solaris. Yes, we use ZEO almost exclusively, it is realtively easy to setup even fpr development and we don't deploy without ZEO anymore. It makes a fair bit of improvement as well. We have an app (the one that tooks us on the road to zope) that for various reasons has not updated properly since 2002 when we first had our multicpu experiences. This particular app receives quite a bit of load, and since it is built entirely thru the web with by now age-old DTML and ye-olde-zope-techniques, it is not the speediest in the world. Add to that the fact that we use DCOracle2(*) to do Oracle queries, and we sometimes expericence hangups on the ZEO clients. For this app, we have "successfully" delayed it's demise and avoided total chaos by adding more ZEO clients (at the moment, these clients runt on two machines, four processes on one and two processes on the other). Well, we also have scripts that restart the nodes when the leak memory too much, and so on. BUT: the speed of the app has increased with each node we add. I am sorry that I cannot give more numbers - I hear from the traffic on the list that there are other factors involved nowadays that may in some way obsolete the need to bind to a particular CPU, however I do not understand how this can have an impact on the GIL. Let me be the first to admit my total lack of knowledge of kernel task schedulers, but generally speaking, unless the scheduler makes sure that a threaded python process never ever gets distributed over two processors simultaniously, then the GIL *will* be an issue. In any event, I'd love to be proved wrong about the need for taskset (especially if someone comes up with measurable data to that effect) because it means that my sysadmins can simplify their setup for managing Zope (making Zope even more acceptable :-). Cheers, /dario (*) We have not been able to use the versions of DCOracle2 that ChrisW has worked on, because we expericend nother set of problems with them and we never had the opportinuty to really spend time chasing those bugs. I believe ChrisW's DCO2 does solve some of the issues that the original DCO2 has Please note that the problems we had with it, may very well becasue of our particular setup (Oracle 8, bad code in our app, even worse sql, etc). I have heard that for other people Chris's DCOracle2 versions work better than the non-modified DCO2, so YMMV. -- -- ------------------------------------------------------------------- Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech. Lyrics applied to programming & application design: "emancipate yourself from mental slavery" - redemption song, b. marley
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Dario Lopez-Kästen wrote: <snip>
Let me be the first to admit my total lack of knowledge of kernel task schedulers, but generally speaking, unless the scheduler makes sure that a threaded python process never ever gets distributed over two processors simultaniously, then the GIL *will* be an issue.
The hypothesis is that the scheduler "notices" that threads in a single Python process are always blocking on a single kernel resource (the semaphore / mutex which is the GIL), and hence migrates them all to the same CPU (over time). I could imagine that this is like the "hand-rolled assembly vs. compiler-generated code" case: older kernels (like the one your old app was dpeloyed atop, perhaps?) didn't have such smarts, but newsr ones do.
In any event, I'd love to be proved wrong about the need for taskset (especially if someone comes up with measurable data to that effect) because it means that my sysadmins can simplify their setup for managing Zope (making Zope even more acceptable :-).
WHile at ZC, one of the other engineers and I did some testing on SMB boxes, and found that "one appserver per CPU" gave us near linear scaling of the application, without any explicit affinity set. I don't have the numbers (we were using stock Dell 1U dual CPU boxes, I think), but the win seemed clear enough that we quit invesigating taskset. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEIVAK+gerLs4ltQ4RAnb9AKC7qw38+BqNdAbY79bqPR4/G7USCwCbBmit UX+GcftjNQ5fUKajALsEbSk= =+E5c -----END PGP SIGNATURE-----
On Wed, 2006-03-22 at 08:24 -0500, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
WHile at ZC, one of the other engineers and I did some testing on SMB boxes, and found that "one appserver per CPU" gave us near linear scaling of the application, without any explicit affinity set. I don't have the numbers (we were using stock Dell 1U dual CPU boxes, I think), but the win seemed clear enough that we quit invesigating taskset.
The other engineer was me, and it was close enough to double the load per instance, per cpu. If he needs to test it, the numbers are easy to get. Run a single instance on a SMP Server, and hit it with load. You will get Y. Run another instance and spread that load across both instances and you will get nearly 2xY. Take a single cpu server and run 1 instance and load test it, and you'll get Y. Take that same single cpu server and run 2 instance, and load test it across both instances, and you'll get Y. These tests were done on both Linux and FreeBSD and both were approximately the same. The conclusion was, no further complexity is required to get a substantial performance benfit by running SMP Servers with multiple Zeo Clients instances, where the # of CPUS == the # of Zeo Client instances. This has been posted to the lists serveral times BTW. The only snafu is to make sure you have enough RAM to run several zeo client instances. If you've got a 4way server, I'd recommend 16GBs of RAM if it's a big site. I personally like to put as much ram into a server as it can physically handle. Andrew Sawyers
Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org
iD8DBQFEIVAK+gerLs4ltQ4RAnb9AKC7qw38+BqNdAbY79bqPR4/G7USCwCbBmit UX+GcftjNQ5fUKajALsEbSk= =+E5c -----END PGP SIGNATURE-----
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Andrew Sawyers wrote:
The other engineer was me, and it was close enough to double the load per instance, per cpu.
tangential question: I'm assuming you were using LVS in front of these boxes? If so, how'd you get the both clients on the box accessible by LVS? 2 ip addresses? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chris Withers wrote:
Andrew Sawyers wrote:
The other engineer was me, and it was close enough to double the load per instance, per cpu.
tangential question: I'm assuming you were using LVS in front of these boxes? If so, how'd you get the both clients on the box accessible by LVS? 2 ip addresses?
Nope, different port numbere. LVS is configured per-service, rather than per-IP. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEIo3E+gerLs4ltQ4RAmBaAKCwiU6E/cnJNRQBTSNDk32j+hKRkACgjhgn awM/z7gLOzscAeCzKY5Aexk= =0HJ9 -----END PGP SIGNATURE-----
Dario Lopez-Kästen wrote:
<snip>
Martinj, Chris and others giving nonsential answers: Don't you guys have work to do instead of playing bullies on the newbies?.
Dario, I actually think your comment here is a bit out of order if you're referring to this post of mine: http://mail.zope.org/pipermail/zope/2006-March/165574.html All I've provided there is information on a better forum to ask about cpu affinity and some other sources of possible slowness which might well be the case here. I also don't think it's exactly fair to lump Hugo in as a newbie. He's been at this enough years now that he should know better...
I see a lot of this newbie bashing increasing; in fact it has been some years since the zope list was a good example on how to treat newbies politely and point them to the prope answers.
In all fairness, the quality of newbies was better back then. Too many people come to this list nowadays asking about stuff without bothering to do any research and often asking about Plohn, whereas their own lists would be much better... They then get arsey because people won't bend over backwards to help them answer the same question they asked before, even though they're not even paying the people they're expecting to help them.
enough sleep, yada, yada, yada... I fail to see how that is a newbie's fault, so no need to take it out on them.
It's a two way street...
The need for threaded python processes to be CPU bound (yes, Zope is threaded, in case you have forgotten) arises from the fact that python has the GIL (Global Interpreter Lock).
For background see:
http://www.zope.org/Members/glpb/solaris
and
Having chatted with both the author and the researcher of that paper, I don't remember the results being as clear-cut as you imply ;-) Still, had Hugo bothered to do even some cursory googling, would he not have found all that information? http://www.google.co.uk/search?q=zope+multi+cpu
Regarding all other advises you have gotten so far (get more memory, look at your disks, get a life, etc), I am sure that they have merits, but as far as I can see, they don't actually solve anything at all.
I think you're stepping outside the bounds of reasonable argument here... The other advice, in fact, more often than not, has more impact on more zope installations...
The fact that the issue of the GIL is not more prominent in the Zope worlds, I think is because relatively few zopistas are aware that there is a problem; mostly, because not so many run multicpu-boxes in production, and also because of attitude, I suppose: "since I don't have a problem with it, I don't care about it, and I'll inform you so."
Actually, most people run multiple zeo clients on multi-processor boxes and let the native task scheduler "do the right thing". For "most people" this seems to work just fine... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers said the following on 2006-03-22 01:36:
Dario Lopez-Kästen wrote:
Dario, I actually think your comment here is a bit out of order if you're referring to this post of mine:
yes, it was and I apologise for it, you did point to relevant information and I was unfair towards you.
In all fairness, the quality of newbies was better back then. Too many people come to this list nowadays asking about stuff without bothering to do any research and often asking about Plohn, whereas their own lists would be much better...
They then get arsey because people won't bend over backwards to help them answer the same question they asked before, even though they're not even paying the people they're expecting to help them.
Unfortunaltey, that is the ways things work, and I think we all have to prepare to be nice to those newbies too. Back then, when we are a select "few" that used zope and zope was not so on-topic as it is today, the ones that were on the list were interested in Zope-the-technology and thus could ask sensible questions. Generally speaking, with the growing poularity of Zope-solutions, where Plone and CPS being popular solutions more or less "hide" the technology behind, we get a bunch of people that do not necessarily care about the technology behind plone/cps, they just wnat things to work. Also, especially having the "marketing zope" discussions happening on other lists, we need to come to terms with the fact that zope does not exist in isolation from it's environment. Questions on the general Zope list about issues and successes in deploying Zope, Zope performance *ARE* legitimate questions on the list. Even if the real answer is "it's a python thing". If folks don't care for politeness and a will to help users, then they should see it as a technical issue. Zope is probably the largest Python Threaded app there is at the moment (perhaps the largest Python app, period?) - Zope has lead to bugs being discovered in Python that get fixed in newer Python releases. So, questions and issues about threaded Pytohn apps are very likely to be related to Zope, and shoudl not be dismissed ad-hoc-ly, in spite of the answer being "google is your friend: zope+multi+cpu"
enough sleep, yada, yada, yada... I fail to see how that is a newbie's fault, so no need to take it out on them.
It's a two way street...
Indeed, and in hindsight, I guess I reverted to this behaviour as well :P. Sorry about that.
Having chatted with both the author and the researcher of that paper, I don't remember the results being as clear-cut as you imply ;-)
having experienced this first hand, before the paper arrived (and it did help me find the solution), I think I disagree :-)
Still, had Hugo bothered to do even some cursory googling, would he not have found all that information?
I guess he was under the wrong impression that the zope-list was a friendly and safe place to ask questions and get community support :) Joke aside, yes, he would have found the answer probably, but we cannot expect all netizens to be civilised and know proper behaviour. I think Checkov (or someone) said something along the lines of "good table manners is not to not spill sauce on the table, but to not notice when someone else does". I think we could do with something similar about netiquette.
Regarding all other advises you have gotten so far (get more memory, look at your disks, get a life, etc), I am sure that they have merits, but as far as I can see, they don't actually solve anything at all.
I think you're stepping outside the bounds of reasonable argument here...
The other advice, in fact, more often than not, has more impact on more zope installations...
I'd love to see more data and less opinion about this - and no, I am not being sarcastic here. If we can avoid taskset then the setup is *way* simpler for our sysadmins, so I have a real interest here.
Actually, most people run multiple zeo clients on multi-processor boxes and let the native task scheduler "do the right thing". For "most people" this seems to work just fine...
See above. /dario -- -- ------------------------------------------------------------------- Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech. Lyrics applied to programming & application design: "emancipate yourself from mental slavery" - redemption song, b. marley
Dario Lopez-Kästen wrote:
Unfortunaltey, that is the ways things work, and I think we all have to prepare to be nice to those newbies too.
I think we have a right to expect those newbies to be nice back ;-)
Generally speaking, with the growing poularity of Zope-solutions, where Plone and CPS being popular solutions more or less "hide" the technology behind, we get a bunch of people that do not necessarily care about the technology behind plone/cps, they just wnat things to work.
If that's the case, they should either be prepared to pay someone.
Also, especially having the "marketing zope" discussions happening on other lists, we need to come to terms with the fact that zope does not exist in isolation from it's environment. Questions on the general Zope list about issues and successes in deploying Zope, Zope performance *ARE* legitimate questions on the list.
Yes, but when you point out more helpful forums to people and they just whine back, they deserve all the abuse they get, I'm afraid...
So, questions and issues about threaded Pytohn apps are very likely to be related to Zope, and shoudl not be dismissed ad-hoc-ly, in spite of the answer being "google is your friend: zope+multi+cpu"
Not sure I agree. If people are expecting free help, they should at least have the courtesy to do the ground work.
Having chatted with both the author and the researcher of that paper, I don't remember the results being as clear-cut as you imply ;-)
having experienced this first hand, before the paper arrived (and it did help me find the solution), I think I disagree :-)
I think we can agree that this isn't consistent for everyone, which kindof proves the point that other options might be best explored first ;-)
I guess he was under the wrong impression that the zope-list was a friendly and safe place to ask questions and get community support :)
For the vast majority of people, it is. If you're rude and lazy, then maybe it shouldn't be ;-)
Checkov (or someone) said something along the lines of "good table manners is not to not spill sauce on the table, but to not notice when someone else does".
Visiting someone's house without an invitation and throwing sauce in their face is not something I think covered by that phrase... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Many things were said here... Lots of Kb's spent going around the world... The main question remained unexplained! Maybe Chris didn't know the answer and just directed this thread to another direction where he could be much more exposed as the "zope@zope.org father"??? I don't know. What I do know is that I'm not a newbie AND I did my homework (google+other sources) before coming here! Maybe I'm a newbie regarding the use of Zope on top of 4 CPU's but then again... Everybody is a bit newbie to something on this Earth! I got to know Chris while I was living in London (working with Zope) and he seemed a very nice guy having a pint of beer in front of him... But I also know him from this mailing list... So... When it comes to Chris answers I just ignore 80% of what he says and try to learn the other 20% (the knowledge itself). Having said this... Cheers! Hugo On 3/22/06, Chris Withers <chris@simplistix.co.uk> wrote:
Dario Lopez-Kästen wrote:
Unfortunaltey, that is the ways things work, and I think we all have to prepare to be nice to those newbies too.
I think we have a right to expect those newbies to be nice back ;-)
Generally speaking, with the growing poularity of Zope-solutions, where Plone and CPS being popular solutions more or less "hide" the technology behind, we get a bunch of people that do not necessarily care about the technology behind plone/cps, they just wnat things to work.
If that's the case, they should either be prepared to pay someone.
Also, especially having the "marketing zope" discussions happening on other lists, we need to come to terms with the fact that zope does not exist in isolation from it's environment. Questions on the general Zope list about issues and successes in deploying Zope, Zope performance *ARE* legitimate questions on the list.
Yes, but when you point out more helpful forums to people and they just whine back, they deserve all the abuse they get, I'm afraid...
So, questions and issues about threaded Pytohn apps are very likely to be related to Zope, and shoudl not be dismissed ad-hoc-ly, in spite of the answer being "google is your friend: zope+multi+cpu"
Not sure I agree. If people are expecting free help, they should at least have the courtesy to do the ground work.
Having chatted with both the author and the researcher of that paper, I don't remember the results being as clear-cut as you imply ;-)
having experienced this first hand, before the paper arrived (and it did help me find the solution), I think I disagree :-)
I think we can agree that this isn't consistent for everyone, which kindof proves the point that other options might be best explored first ;-)
I guess he was under the wrong impression that the zope-list was a friendly and safe place to ask questions and get community support :)
For the vast majority of people, it is. If you're rude and lazy, then maybe it shouldn't be ;-)
Checkov (or someone) said something along the lines of "good table manners is not to not spill sauce on the table, but to not notice when someone else does".
Visiting someone's house without an invitation and throwing sauce in their face is not something I think covered by that phrase...
cheers,
Chris
-- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-- Hugo Ramos - ramosh@gmail.com http://www.orkut.com/Profile.aspx?uid=10082105466310142690
Hugo Ramos wrote:
I don't know. What I do know is that I'm not a newbie AND I did my homework (google+other sources) before coming here!
Then how come you managed to miss Paul Browning and Matt Hamilton's extensive documentation of the subject? Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Maybe I just misspelled the word "affinity" and wrote it with just 1 f "afinity"... Hey Chris.. Cheer up dude! Try not be this hard to people that just want to learn a bit more... Specially because back in 2000 I had you as a Zope icon in Europe. I'm working in Brazil now... The entire Brazilian Federal Government decided to move into Open Source.. More specifically Linux+Zope+Plone. Here I'm learning something called respect for others. The rest of the world has a lot to learn from Brazilian people about social relations. Something I didn't even taste while living in England... Anyway... There's a lot of jobs for good professionals like you! Salary is much more than the average Brazilian makes... Maybe this country could be good for you! :-) Cheers Hugo On 3/23/06, Chris Withers <chris@simplistix.co.uk> wrote:
Hugo Ramos wrote:
I don't know. What I do know is that I'm not a newbie AND I did my homework (google+other sources) before coming here!
Then how come you managed to miss Paul Browning and Matt Hamilton's extensive documentation of the subject?
Chris
-- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
+-------[ Hugo Ramos ]---------------------- | | Anyway... There's a lot of jobs for good professionals like you! | Salary is much more than the average Brazilian makes... Maybe this | country could be good for you! :-) Don't worry Hugo, if Chris keeps going the way he's going, eventually he'll have to hide out in South America... d8) -- Andrew Milton akm@theinternet.com.au
Hugo Ramos wrote:
I think these questions are very ZOPE RELATED since I asked them in a Zope mailing list don't you think?
No, we don't. This has nothing specific to do with Zope. Process affinity is an OS-specific thing, and if you care about it you should as on an OS-specific forum. In your case, you're probably going to get help much more quickly and usefully on #debian on irc.freenode.net. The guys there are really clued up. However, as an aside, how have you established that processor affinity is the problem here? I'd suggest looking at your disk and memory usage patterns. i/o wait or swap death could quite easily see your processors only hitting 30% even if the affinity is locked down exactly as you'd like... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
Hugo Ramos wrote:
However, as an aside, how have you established that processor affinity is the problem here? I'd suggest looking at your disk and memory usage patterns. i/o wait or swap death could quite easily see your processors only hitting 30% even if the affinity is locked down exactly as you'd like...
With a 4 GB machine that is probably not a problem. You can usually run quite a few sites with that much memory. -- hilsen/regards Max M, Denmark http://www.mxm.dk/ IT's Mad Science Phone: +45 66 11 84 94 Mobile: +45 29 93 42 96
Hugo Ramos wrote:
However, as an aside, how have you established that processor affinity is the problem here? I'd suggest looking at your disk and memory usage patterns. i/o wait or swap death could quite easily see your processors only hitting 30% even if the affinity is locked down exactly as you'd like...
google for taskset and debian. that'll help tie a process to a cpu hth -- http://myzope.kedai.com.my - my-zope org
Hugo Ramos wrote:
"Has any1 tried this before? Can you point me to some documentation? What's your experience? is it true that not doing this the 4 cpu's will not be 100% used?"
I think these questions are very ZOPE RELATED since I asked them in a Zope mailing list don't you think?
Eh ... it doesn't become zope related because you ask it in a Zope group. Do you also ask how to write books on a Microsoft Word list?
I'm asking about Zope user/administrator experience related to Zope use on a quad machine!!!
Of course if some1 tells me how to do it using OS related stuff this is STILL Zope related!
Assigning a process to a cpu is a general os problem, and not really related to zope. Zope is just another process. eg: http://www.google.com/search?q=cpu+affinity plone.org is running on a dual cpu box, with one zeo and a zeo client, so clearly it can be done.
Hey... but thank you Jens for spending all that time writing an email that doesn't help any1 at all...!!!
Sarkasm against a long time, and helpfull, list user will not help you any at all. -- hilsen/regards Max M, Denmark http://www.mxm.dk/ IT's Mad Science Phone: +45 66 11 84 94 Mobile: +45 29 93 42 96
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hugo Ramos wrote:
Yes... But I'm looking for more specific answers about how to create the affinity between 1 of the zope processes and the CPU n.2, for example... How to make that a permanent choice?
That question is not Zope specific -- you would need to modify the start scripts for your appservers to make whatever kernel / libc-specific calls are needed (on Solaris, it would be invoking the 'pbind' command). In the worst case, you would end up writing a wrapper binary which set the affinity mask and then exec'ed Python. I don't know what facilities the kernel in your Linux provides for doing that (and haven't tried on any recent kernels of my own, either). Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEGLOR+gerLs4ltQ4RAhMQAJ0bYJ+0Acnbpg011jVsBBivFF3zJACg2hIK Nw1h0QVMbFr8WqjY/v/v558= =a1fo -----END PGP SIGNATURE-----
On 3/15/06, Tres Seaver <tseaver@palladion.com> wrote:
That question is not Zope specific -- you would need to modify the start scripts for your appservers to make whatever kernel / libc-specific calls are needed (on Solaris, it would be invoking the 'pbind' command). In the worst case, you would end up writing a wrapper binary which set the affinity mask and then exec'ed Python.
I don't know what facilities the kernel in your Linux provides for doing that (and haven't tried on any recent kernels of my own, either).
On 2.6 kernels (and possibly some patched/backported 2.4 kernels) with recent glibc versions, the API call to use would be sched_setaffinity() ( see <sched.h> ) . I'm not aware of any existing userland command on Linux equivalent to "pbind" of Solaris or "mpsched" or "runon" of other UNIX systems. It would be a pretty easy little program to write though. You should be able to set the processor affinity on your running Zope instance too, assuming appropriate privledges. - Jeff D
On Wed, Mar 15, 2006 at 05:45:23PM -0300, Hugo Ramos wrote:
Yellow,
I'm using Zope+Apache on a 4 xeon's/4GB ram machine running Debian. I've noticed that the CPU's never go beyond 30% top occupation... but on rush hours the site takes too long to load...
I've been reading about process affinity and how it could speed up everything by making zope run on 1 CPU, Apache on another and so on...
Has any1 tried this before? Can you point me to some documentation? What's your experience? is it true that not doing this the 4 cpu's will not be 100% used?
Have you considered that your setup might be IO bound? -- __________________________________________________ "Nothing is as subjective as reality" Reinoud van Leeuwen reinoud.v@n.leeuwen.net http://www.xs4all.nl/~reinoud __________________________________________________
You can do something like: /usr/bin/taskset -c 0 zeo/bin/zeoctl start /usr/bin/taskset -c 1 zope.a/bin/zopectl start /usr/bin/taskset -c 2 zope.b/bin/zopectl start /usr/bin/taskset -c 3 zope.c/bin/zopectl start You then need to load-balance across the ZOPE instances with (eg Pound or Squid). AFAIK it's only Python-based programs that need binding to a particular CPU, so Apache, Pound and Squid are fine without setting any affinity. Since ZEO is not very CPU intensive, you can share it's CPU with another ZOPE instance should you wish. We're running twin ZOPE instances and ZEO on a dual CPU Debian box, load balancing with Pound and caching and URL rewriting with Apache. The feeling these days seems to be that load balancing and caching with Squid gives better performance. Steven Steven Hayles - Computer Systems Developer, sh23@le.ac.uk Learning Technology Section, Computer Centre, University of Leicester, University Rd, Leicester, LE1 7RH Fax (0/+44)116 2525027 WWW <URL:http://www.le.ac.uk/home/sh23> On Wed, 15 Mar 2006, Hugo Ramos wrote:
Yellow,
I'm using Zope+Apache on a 4 xeon's/4GB ram machine running Debian. I've noticed that the CPU's never go beyond 30% top occupation... but on rush hours the site takes too long to load...
I've been reading about process affinity and how it could speed up everything by making zope run on 1 CPU, Apache on another and so on...
Has any1 tried this before? Can you point me to some documentation? What's your experience? is it true that not doing this the 4 cpu's will not be 100% used?
Cheers -- Hugo Ramos - ramosh@gmail.com http://www.orkut.com/Profile.aspx?uid=10082105466310142690 _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
a) Your system may be I/O bound - not CPU bound - so you will never see the CPU max out because the limiting factors are memory and disk/ network access. b) Make sure to tune Python's checkinterval. While you should *always* do that, it is especially important on multi-processor/multi- core hardware. The rule of thumb is to run the pystone benchmark, divide the "pystones" your machine benchmarks at by 50, and put the result into etc/zope.conf. e.g: python-check-interval 1000 Once you get the checkinterval right, playing with processor affinity becomes a pointless exercise, in my experience. c) Always use profiling tools for performance analysis. "Guessing" rarely leads to effective optimizations. ZopeProfiler and PTProfiler come to mind. HTH, Stefan On 15. Mär 2006, at 21:45, Hugo Ramos wrote:
Yellow,
I'm using Zope+Apache on a 4 xeon's/4GB ram machine running Debian. I've noticed that the CPU's never go beyond 30% top occupation... but on rush hours the site takes too long to load...
I've been reading about process affinity and how it could speed up everything by making zope run on 1 CPU, Apache on another and so on...
Has any1 tried this before? Can you point me to some documentation? What's your experience? is it true that not doing this the 4 cpu's will not be 100% used?
-- Anything that happens, happens. --Douglas Adams
Thank you all !!!!!!!!!! (very insane multiple exclamation marks) I've been trying different scenarios and Zope performance increased 10 to 11 times faster... Cheers Hugo On 3/16/06, Stefan H. Holek <stefan@epy.co.at> wrote:
a) Your system may be I/O bound - not CPU bound - so you will never see the CPU max out because the limiting factors are memory and disk/ network access.
b) Make sure to tune Python's checkinterval. While you should *always* do that, it is especially important on multi-processor/multi- core hardware. The rule of thumb is to run the pystone benchmark, divide the "pystones" your machine benchmarks at by 50, and put the result into etc/zope.conf.
e.g: python-check-interval 1000
Once you get the checkinterval right, playing with processor affinity becomes a pointless exercise, in my experience.
c) Always use profiling tools for performance analysis. "Guessing" rarely leads to effective optimizations. ZopeProfiler and PTProfiler come to mind.
HTH, Stefan
On 15. Mär 2006, at 21:45, Hugo Ramos wrote:
Yellow,
I'm using Zope+Apache on a 4 xeon's/4GB ram machine running Debian. I've noticed that the CPU's never go beyond 30% top occupation... but on rush hours the site takes too long to load...
I've been reading about process affinity and how it could speed up everything by making zope run on 1 CPU, Apache on another and so on...
Has any1 tried this before? Can you point me to some documentation? What's your experience? is it true that not doing this the 4 cpu's will not be 100% used?
-- Anything that happens, happens. --Douglas Adams
-- Hugo Ramos - ramosh@gmail.com http://www.orkut.com/Profile.aspx?uid=10082105466310142690 http://otugga.blogspot.com/ http://otuggapoesia.blogspot.com/
Hugo Ramos wrote:
Thank you all !!!!!!!!!! (very insane multiple exclamation marks)
I've been trying different scenarios and Zope performance increased 10 to 11 times faster...
Any chance you could share what you actually did to accomplish this? /Aton
Well... I just followed some advices ppl gave me in the mailing list and also a feeling I had about using process affinity. I used taskset when starting Zope + Apache like this: cpu0: all other processes cpu1: 10 Apache (mod_ssl,mod_rewrite,mod_proxy) cpu2: 10 Apache (mod_ssl,mod_rewrite,mod_proxy) cpu3: Zope I also fine tunned Python check interval to the specific server using stones.py I've created a nice caching policy at Zope level and also at Apache level. Other stuff I usually did before already like regular ZODB packs and so on... After all this I got 64 requests/sec on ab. Before it was 5 or 6 requests/sec Cheers On 3/31/06, Anton Stonor <anton@headnet.dk> wrote:
Hugo Ramos wrote:
Thank you all !!!!!!!!!! (very insane multiple exclamation marks)
I've been trying different scenarios and Zope performance increased 10 to 11 times faster...
Any chance you could share what you actually did to accomplish this?
/Aton
-- Hugo Ramos - ramosh@gmail.com http://www.orkut.com/Profile.aspx?uid=10082105466310142690 http://otugga.blogspot.com/ http://otuggapoesia.blogspot.com/
On Thu, Mar 16, 2006 at 02:38:04PM +0100, Stefan H. Holek wrote:
a) Your system may be I/O bound - not CPU bound - so you will never see the CPU max out because the limiting factors are memory and disk/ network access.
On that note, if running linux on a box with IDE disks, make sure that the disks have dma turned on. This can have a big impact on overall system responsiveness. man hdparm for details. -PW -- Paul Winkler http://www.slinkp.com
Hugo Ramos wrote at 2006-3-15 17:45 -0300:
I'm using Zope+Apache on a 4 xeon's/4GB ram machine running Debian. I've noticed that the CPU's never go beyond 30% top occupation... but on rush hours the site takes too long to load...
I've been reading about process affinity and how it could speed up everything by making zope run on 1 CPU, Apache on another and so on...
A further, more promissing option, is to run several Zope processes on the same machine (using different ports) with a load balancer in front feeding them with work. -- Dieter
Hi Hugo, I've got a similar setup, Compaq DL580 on a Quad Xeon with 3gb of ram on Debian. To overcome the GIL and avoid dealing with CPU affiinity and to optimize server utilization, I started using Xen virtualization to create multiple virtual servers. It's open source and works great. Check out http://www.cl.cam.ac.uk/Research/SRG/netos/xen/ or http://www.xensource.com. For performance see http://www.cl.cam.ac.uk/Research/SRG/netos/xen/performance.html This tutorial worked for me: http://www.debian-administration.org/articles/320 I also recommend installing xen-tools http://www.steve.org.uk/Software/xen-tools/ It's very easy to create new virtual servers: xen-create-image --host=apache xen-create-image --host=zclient1 xen-create-image --host=zclient2 xen-create-image --host=zeo To create and attach to a virtual server, just do "xm create zclient2.cfg -c" log in and install python, Zope and whatever else you need. So instead of running 1 plone instance at 22pps, you can have serveral plone instances running at 22pps each and no GIL problem. HTH, Kevin Smith -- View this message in context: http://www.nabble.com/Zope-%2B-Apache-on-Quad-Debian-machine-t1287183.html#a... Sent from the Zope - General forum at Nabble.com.
Thanks Kevin, It's a very good architecture option. I'm afraid a bit too late to test.. I'm already in production with the new architecture and CPU affinity. It's running smoothly though.. :-) Regards Hugo On 3/22/06, ksmith99 <ksmith93940-dev@yahoo.com> wrote:
Hi Hugo,
I've got a similar setup, Compaq DL580 on a Quad Xeon with 3gb of ram on Debian. To overcome the GIL and avoid dealing with CPU affiinity and to optimize server utilization, I started using Xen virtualization to create multiple virtual servers. It's open source and works great.
Check out http://www.cl.cam.ac.uk/Research/SRG/netos/xen/ or http://www.xensource.com. For performance see http://www.cl.cam.ac.uk/Research/SRG/netos/xen/performance.html
This tutorial worked for me: http://www.debian-administration.org/articles/320 I also recommend installing xen-tools http://www.steve.org.uk/Software/xen-tools/
It's very easy to create new virtual servers: xen-create-image --host=apache xen-create-image --host=zclient1 xen-create-image --host=zclient2 xen-create-image --host=zeo
To create and attach to a virtual server, just do "xm create zclient2.cfg -c" log in and install python, Zope and whatever else you need.
So instead of running 1 plone instance at 22pps, you can have serveral plone instances running at 22pps each and no GIL problem.
HTH,
Kevin Smith
-- View this message in context: http://www.nabble.com/Zope-%2B-Apache-on-Quad-Debian-machine-t1287183.html#a... Sent from the Zope - General forum at Nabble.com.
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
-- Hugo Ramos - ramosh@gmail.com http://www.orkut.com/Profile.aspx?uid=10082105466310142690 http://otugga.blogspot.com/ http://otuggapoesia.blogspot.com/
participants (19)
-
Andrew Milton -
Andrew Sawyers -
Anton Stonor -
Bakhtiar A Hamid -
Chris Withers -
Dario Lopez-Kästen -
Dieter Maurer -
Hugo Ramos -
Jeff Donsbach -
Jens Vagelpohl -
ksmith99 -
Martijn Pieters -
Max M -
Paul Winkler -
Reinoud van Leeuwen -
S.Hayles -
Stefan H. Holek -
Tino Wildenhain -
Tres Seaver