On 15 Mar 2006, at 21:11, Hugo Ramos wrote:

Yes... But I'm looking for more specific answers about how to create the affinity between 1 of the zope processes and the CPU n.2, for example... How to make that a permanent choice?

This question has nothing to do with Zope. You will need to find out from your operating system documentation or from a forum that deals with it. jens

Hugo Ramos schrieb:

"Has any1 tried this before? Can you point me to some documentation? What's your experience? is it true that not doing this the 4 cpu's will not be 100% used?"

I think these questions are very ZOPE RELATED since I asked them in a Zope mailing list don't you think? I'm asking about Zope user/administrator experience related to Zope use on a quad machine!!!

Of course if some1 tells me how to do it using OS related stuff this is STILL Zope related!

Hey... but thank you Jens for spending all that time writing an email that doesn't help any1 at all...!!!

Hahaha. And the time you are weeping would have been better spent on googling <your os> <cpu afinity> - and err. btw. also check google for "multiple use of exclamation marks". Btw. I dont believe the "cpu afinity" buys you anything in zope context. Try it and test it in your setup with your application. --Tino

Martijn Pieters said the following on 2006-03-16 12:25:

On 3/16/06, Tino Wildenhain <tino@wildenhain.de> wrote:

...

...

Hey... but thank you Jens for spending all that time writing an email that doesn't help any1 at all...!!!

check google for "multiple use of exclamation marks".

Or look up Terry Pratchett quotes on www.lspace.org:

<snip> Martinj, Chris and others giving nonsential answers: Don't you guys have work to do instead of playing bullies on the newbies?. To be honest, I find arrogant and pointless answers like yours way more annoying than a whole bunch of newbies asking more or less cluess questions. I see a lot of this newbie bashing increasing; in fact it has been some years since the zope list was a good example on how to treat newbies politely and point them to the prope answers. Sure, we all have loads of work and are way too stressed, we don't get enough sleep, yada, yada, yada... I fail to see how that is a newbie's fault, so no need to take it out on them. For the benefit of all those that have tiny, tiny memory spans, the question of Zope's processes necessity to be CPU-bound is a horse that was resolved a couple of years ago (2002). It turns out that it is not a Zope problem ("Oh, the Horror. He's bringing a non-zope question to the list!"). It is pythons "fault", if you are grumpy old fart, or it is a "feature" if you are having a good day. The need for threaded python processes to be CPU bound (yes, Zope is threaded, in case you have forgotten) arises from the fact that python has the GIL (Global Interpreter Lock). For background see: http://www.zope.org/Members/glpb/solaris and http://www.zope.org/Members/glpb/solaris/report_ps As far as I know, the GIL is still there and won't go away any time soon. Today, the only change from 2002 is that there is a heck of a lot more multi-CPU machines than there were back in 2002. So, to make a long story short: on linux there is a userspace tools called taskset: http://linuxcommand.org/man_pages/taskset1.html http://aplawrence.com/Basics/taskset.html if there is no such command for your particular distro/kernelversion, look no further than: http://directory.fsf.org/all/schedutils.html http://rlove.org/ http://rlove.org/schedutils/ ftp://ftp.kernel.org/pub/linux/utils/util-linux/ I cannot tell you wich version to use, but I think you get the idea. I do believe that some distros allready include taskset in them. So you have to write a wrapper script around zopectl (you are running a newer versionof zope, aren't you?) where you do this. It has worked like a charm for us since 2002 (well, almost - since schedutils were new and cool back in 2002 we had regular fights with the sysadmins guys regarding the need to introduce "non-redhat-aproved-kernel-hacks" into the production systems. Nowadays taskset and related tools are included in the later RHE releases). Regarding all other advises you have gotten so far (get more memory, look at your disks, get a life, etc), I am sure that they have merits, but as far as I can see, they don't actually solve anything at all. The fact that the issue of the GIL is not more prominent in the Zope worlds, I think is because relatively few zopistas are aware that there is a problem; mostly, because not so many run multicpu-boxes in production, and also because of attitude, I suppose: "since I don't have a problem with it, I don't care about it, and I'll inform you so." /dario - yes... a bit annoyed at all the "help" some folks give... -- -- ------------------------------------------------------------------- Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech. Lyrics applied to programming & application design: "emancipate yourself from mental slavery" - redemption song, b. marley

Tino Wildenhain said the following on 2006-03-21 14:51:

Otoh, I have yet to see the figures showing the CPU afinity buys you anything in reality. We know the GIL, thats for sure but I never saw a measureable difference binding a process to a CPU (which is also highly depending on the OS scheduler)

for us, it makes all the difference between "zope sucks, why do we bother with this non-sense, non-standard, butt-slow appserver, and use Java or PHP instead" and "nice, zope based solutions are really nice, not only feature wise, but also speedy. And they are clusterable too, neat!" /dario -- -- ------------------------------------------------------------------- Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech. Lyrics applied to programming & application design: "emancipate yourself from mental slavery" - redemption song, b. marley

Jeff Donsbach said the following on 2006-03-21 17:26:

Dario, Do you have any kind of comparison numbers of using CPU affinity vs not for your particular case? Also, are you using ZEO or not? It's not that I don't believe you when you say it matters a lot for you. I do believe you. Like Tino, I'm just generally interested in how much it matters in measurable terms. I can imagine there are a number of factors determining how much it matters, like Zope app/workload as well as the underlying hardware architecture (how big of a penalty is it to synchronize cache pages between CPUs) and the OS CPU scheduler as Tino mentioned.

To be honest, we have never had the inclination to do much of research in this area. Our situation has mostly been such that we experience horrible delays in zope-responsiveness in testing that vanish as soon as we use taskset. This is on both Solaris (with the equivalent of taskset there) and Linux . Since then (and for other reasons), we have migrated all of the central servers we manage from Solaris to Linux, so I cannot give any more input about Solaris. Yes, we use ZEO almost exclusively, it is realtively easy to setup even fpr development and we don't deploy without ZEO anymore. It makes a fair bit of improvement as well. We have an app (the one that tooks us on the road to zope) that for various reasons has not updated properly since 2002 when we first had our multicpu experiences. This particular app receives quite a bit of load, and since it is built entirely thru the web with by now age-old DTML and ye-olde-zope-techniques, it is not the speediest in the world. Add to that the fact that we use DCOracle2(*) to do Oracle queries, and we sometimes expericence hangups on the ZEO clients. For this app, we have "successfully" delayed it's demise and avoided total chaos by adding more ZEO clients (at the moment, these clients runt on two machines, four processes on one and two processes on the other). Well, we also have scripts that restart the nodes when the leak memory too much, and so on. BUT: the speed of the app has increased with each node we add. I am sorry that I cannot give more numbers - I hear from the traffic on the list that there are other factors involved nowadays that may in some way obsolete the need to bind to a particular CPU, however I do not understand how this can have an impact on the GIL. Let me be the first to admit my total lack of knowledge of kernel task schedulers, but generally speaking, unless the scheduler makes sure that a threaded python process never ever gets distributed over two processors simultaniously, then the GIL *will* be an issue. In any event, I'd love to be proved wrong about the need for taskset (especially if someone comes up with measurable data to that effect) because it means that my sysadmins can simplify their setup for managing Zope (making Zope even more acceptable :-). Cheers, /dario (*) We have not been able to use the versions of DCOracle2 that ChrisW has worked on, because we expericend nother set of problems with them and we never had the opportinuty to really spend time chasing those bugs. I believe ChrisW's DCO2 does solve some of the issues that the original DCO2 has Please note that the problems we had with it, may very well becasue of our particular setup (Oracle 8, bad code in our app, even worse sql, etc). I have heard that for other people Chris's DCOracle2 versions work better than the non-modified DCO2, so YMMV. -- -- ------------------------------------------------------------------- Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech. Lyrics applied to programming & application design: "emancipate yourself from mental slavery" - redemption song, b. marley

On Wed, 2006-03-22 at 08:24 -0500, Tres Seaver wrote:

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1

WHile at ZC, one of the other engineers and I did some testing on SMB boxes, and found that "one appserver per CPU" gave us near linear scaling of the application, without any explicit affinity set. I don't have the numbers (we were using stock Dell 1U dual CPU boxes, I think), but the win seemed clear enough that we quit invesigating taskset.

The other engineer was me, and it was close enough to double the load per instance, per cpu. If he needs to test it, the numbers are easy to get. Run a single instance on a SMP Server, and hit it with load. You will get Y. Run another instance and spread that load across both instances and you will get nearly 2xY. Take a single cpu server and run 1 instance and load test it, and you'll get Y. Take that same single cpu server and run 2 instance, and load test it across both instances, and you'll get Y. These tests were done on both Linux and FreeBSD and both were approximately the same. The conclusion was, no further complexity is required to get a substantial performance benfit by running SMP Servers with multiple Zeo Clients instances, where the # of CPUS == the # of Zeo Client instances. This has been posted to the lists serveral times BTW. The only snafu is to make sure you have enough RAM to run several zeo client instances. If you've got a 4way server, I'd recommend 16GBs of RAM if it's a big site. I personally like to put as much ram into a server as it can physically handle. Andrew Sawyers

Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org

iD8DBQFEIVAK+gerLs4ltQ4RAnb9AKC7qw38+BqNdAbY79bqPR4/G7USCwCbBmit UX+GcftjNQ5fUKajALsEbSk= =+E5c -----END PGP SIGNATURE-----

_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Chris Withers wrote:

Andrew Sawyers wrote:

...

The other engineer was me, and it was close enough to double the load per instance, per cpu.

tangential question: I'm assuming you were using LVS in front of these boxes? If so, how'd you get the both clients on the box accessible by LVS? 2 ip addresses?

Nope, different port numbere. LVS is configured per-service, rather than per-IP. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEIo3E+gerLs4ltQ4RAmBaAKCwiU6E/cnJNRQBTSNDk32j+hKRkACgjhgn awM/z7gLOzscAeCzKY5Aexk= =0HJ9 -----END PGP SIGNATURE-----

Chris Withers said the following on 2006-03-22 01:36:

Dario Lopez-Kästen wrote:

Dario, I actually think your comment here is a bit out of order if you're referring to this post of mine:

http://mail.zope.org/pipermail/zope/2006-March/165574.html

yes, it was and I apologise for it, you did point to relevant information and I was unfair towards you.

In all fairness, the quality of newbies was better back then. Too many people come to this list nowadays asking about stuff without bothering to do any research and often asking about Plohn, whereas their own lists would be much better...

They then get arsey because people won't bend over backwards to help them answer the same question they asked before, even though they're not even paying the people they're expecting to help them.

Unfortunaltey, that is the ways things work, and I think we all have to prepare to be nice to those newbies too. Back then, when we are a select "few" that used zope and zope was not so on-topic as it is today, the ones that were on the list were interested in Zope-the-technology and thus could ask sensible questions. Generally speaking, with the growing poularity of Zope-solutions, where Plone and CPS being popular solutions more or less "hide" the technology behind, we get a bunch of people that do not necessarily care about the technology behind plone/cps, they just wnat things to work. Also, especially having the "marketing zope" discussions happening on other lists, we need to come to terms with the fact that zope does not exist in isolation from it's environment. Questions on the general Zope list about issues and successes in deploying Zope, Zope performance *ARE* legitimate questions on the list. Even if the real answer is "it's a python thing". If folks don't care for politeness and a will to help users, then they should see it as a technical issue. Zope is probably the largest Python Threaded app there is at the moment (perhaps the largest Python app, period?) - Zope has lead to bugs being discovered in Python that get fixed in newer Python releases. So, questions and issues about threaded Pytohn apps are very likely to be related to Zope, and shoudl not be dismissed ad-hoc-ly, in spite of the answer being "google is your friend: zope+multi+cpu"

...

enough sleep, yada, yada, yada... I fail to see how that is a newbie's fault, so no need to take it out on them.

It's a two way street...

Indeed, and in hindsight, I guess I reverted to this behaviour as well :P. Sorry about that.

...

http://www.zope.org/Members/glpb/solaris/report_ps

Having chatted with both the author and the researcher of that paper, I don't remember the results being as clear-cut as you imply ;-)

having experienced this first hand, before the paper arrived (and it did help me find the solution), I think I disagree :-)

Still, had Hugo bothered to do even some cursory googling, would he not have found all that information?

http://www.google.co.uk/search?q=zope+multi+cpu

I guess he was under the wrong impression that the zope-list was a friendly and safe place to ask questions and get community support :) Joke aside, yes, he would have found the answer probably, but we cannot expect all netizens to be civilised and know proper behaviour. I think Checkov (or someone) said something along the lines of "good table manners is not to not spill sauce on the table, but to not notice when someone else does". I think we could do with something similar about netiquette.

...

Regarding all other advises you have gotten so far (get more memory, look at your disks, get a life, etc), I am sure that they have merits, but as far as I can see, they don't actually solve anything at all.

I think you're stepping outside the bounds of reasonable argument here...

The other advice, in fact, more often than not, has more impact on more zope installations...

I'd love to see more data and less opinion about this - and no, I am not being sarcastic here. If we can avoid taskset then the setup is *way* simpler for our sysadmins, so I have a real interest here.

Actually, most people run multiple zeo clients on multi-processor boxes and let the native task scheduler "do the right thing". For "most people" this seems to work just fine...

See above. /dario -- -- ------------------------------------------------------------------- Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech. Lyrics applied to programming & application design: "emancipate yourself from mental slavery" - redemption song, b. marley

Many things were said here... Lots of Kb's spent going around the world... The main question remained unexplained! Maybe Chris didn't know the answer and just directed this thread to another direction where he could be much more exposed as the "zope@zope.org father"??? I don't know. What I do know is that I'm not a newbie AND I did my homework (google+other sources) before coming here! Maybe I'm a newbie regarding the use of Zope on top of 4 CPU's but then again... Everybody is a bit newbie to something on this Earth! I got to know Chris while I was living in London (working with Zope) and he seemed a very nice guy having a pint of beer in front of him... But I also know him from this mailing list... So... When it comes to Chris answers I just ignore 80% of what he says and try to learn the other 20% (the knowledge itself). Having said this... Cheers! Hugo On 3/22/06, Chris Withers <chris@simplistix.co.uk> wrote:

Dario Lopez-Kästen wrote:

...

Unfortunaltey, that is the ways things work, and I think we all have to prepare to be nice to those newbies too.

I think we have a right to expect those newbies to be nice back ;-)

...

Generally speaking, with the growing poularity of Zope-solutions, where Plone and CPS being popular solutions more or less "hide" the technology behind, we get a bunch of people that do not necessarily care about the technology behind plone/cps, they just wnat things to work.

If that's the case, they should either be prepared to pay someone.

...

Also, especially having the "marketing zope" discussions happening on other lists, we need to come to terms with the fact that zope does not exist in isolation from it's environment. Questions on the general Zope list about issues and successes in deploying Zope, Zope performance *ARE* legitimate questions on the list.

Yes, but when you point out more helpful forums to people and they just whine back, they deserve all the abuse they get, I'm afraid...

...

So, questions and issues about threaded Pytohn apps are very likely to be related to Zope, and shoudl not be dismissed ad-hoc-ly, in spite of the answer being "google is your friend: zope+multi+cpu"

Not sure I agree. If people are expecting free help, they should at least have the courtesy to do the ground work.

...

...

...

http://www.zope.org/Members/glpb/solaris/report_ps

Having chatted with both the author and the researcher of that paper, I don't remember the results being as clear-cut as you imply ;-)

having experienced this first hand, before the paper arrived (and it did help me find the solution), I think I disagree :-)

I think we can agree that this isn't consistent for everyone, which kindof proves the point that other options might be best explored first ;-)

...

I guess he was under the wrong impression that the zope-list was a friendly and safe place to ask questions and get community support :)

For the vast majority of people, it is. If you're rude and lazy, then maybe it shouldn't be ;-)

...

Checkov (or someone) said something along the lines of "good table manners is not to not spill sauce on the table, but to not notice when someone else does".

Visiting someone's house without an invitation and throwing sauce in their face is not something I think covered by that phrase...

cheers,

Chris

-- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )

-- Hugo Ramos - ramosh@gmail.com http://www.orkut.com/Profile.aspx?uid=10082105466310142690

Maybe I just misspelled the word "affinity" and wrote it with just 1 f "afinity"... Hey Chris.. Cheer up dude! Try not be this hard to people that just want to learn a bit more... Specially because back in 2000 I had you as a Zope icon in Europe. I'm working in Brazil now... The entire Brazilian Federal Government decided to move into Open Source.. More specifically Linux+Zope+Plone. Here I'm learning something called respect for others. The rest of the world has a lot to learn from Brazilian people about social relations. Something I didn't even taste while living in England... Anyway... There's a lot of jobs for good professionals like you! Salary is much more than the average Brazilian makes... Maybe this country could be good for you! :-) Cheers Hugo On 3/23/06, Chris Withers <chris@simplistix.co.uk> wrote:

Hugo Ramos wrote:

...

I don't know. What I do know is that I'm not a newbie AND I did my homework (google+other sources) before coming here!

Then how come you managed to miss Paul Browning and Matt Hamilton's extensive documentation of the subject?

Chris

-- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

Hugo Ramos wrote:

I think these questions are very ZOPE RELATED since I asked them in a Zope mailing list don't you think?

No, we don't. This has nothing specific to do with Zope. Process affinity is an OS-specific thing, and if you care about it you should as on an OS-specific forum. In your case, you're probably going to get help much more quickly and usefully on #debian on irc.freenode.net. The guys there are really clued up. However, as an aside, how have you established that processor affinity is the problem here? I'd suggest looking at your disk and memory usage patterns. i/o wait or swap death could quite easily see your processors only hitting 30% even if the affinity is locked down exactly as you'd like... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

Hugo Ramos wrote:

...

However, as an aside, how have you established that processor affinity is the problem here? I'd suggest looking at your disk and memory usage patterns. i/o wait or swap death could quite easily see your processors only hitting 30% even if the affinity is locked down exactly as you'd like...

google for taskset and debian. that'll help tie a process to a cpu hth -- http://myzope.kedai.com.my - my-zope org

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hugo Ramos wrote:

Yes... But I'm looking for more specific answers about how to create the affinity between 1 of the zope processes and the CPU n.2, for example... How to make that a permanent choice?

That question is not Zope specific -- you would need to modify the start scripts for your appservers to make whatever kernel / libc-specific calls are needed (on Solaris, it would be invoking the 'pbind' command). In the worst case, you would end up writing a wrapper binary which set the affinity mask and then exec'ed Python. I don't know what facilities the kernel in your Linux provides for doing that (and haven't tried on any recent kernels of my own, either). Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFEGLOR+gerLs4ltQ4RAhMQAJ0bYJ+0Acnbpg011jVsBBivFF3zJACg2hIK Nw1h0QVMbFr8WqjY/v/v558= =a1fo -----END PGP SIGNATURE-----

Hugo Ramos wrote:

Thank you all !!!!!!!!!! (very insane multiple exclamation marks)

I've been trying different scenarios and Zope performance increased 10 to 11 times faster...

Any chance you could share what you actually did to accomplish this? /Aton

On Thu, Mar 16, 2006 at 02:38:04PM +0100, Stefan H. Holek wrote:

a) Your system may be I/O bound - not CPU bound - so you will never see the CPU max out because the limiting factors are memory and disk/ network access.

On that note, if running linux on a box with IDE disks, make sure that the disks have dma turned on. This can have a big impact on overall system responsiveness. man hdparm for details. -PW -- Paul Winkler http://www.slinkp.com