Hi, We made some basic load testing against different Zope2 versions and found some unexpected results. Everything is summarized here : http://talk.lastminute.com/wiki/index.php/Loadtest We would appreciate any comment about it, and of course will answer any question you might have about our test. Thanks a lot. Pascal ********************************************************************** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com **********************************************************************
Pascal Peregrina wrote:
We made some basic load testing against different Zope2 versions and found some unexpected results.
What's unexpected in particular? How many runs did you do for each setup and how did you average them? Also, was the hardware, memory usage and processor load, excluding that used by Zope identical over the course of all runs? (common problems are doing load tests on a box that is also doing other things) Also, what was the config? Plain Zope with FileStorage? ZEO? etc... What do you mean by "wsgi enabled"? How does wsgi+ differ from wsgi? The speed drop in ZPT I think is due to using Zope 3's ZPT engine in 2.10. Zope 3 has, unless someone tells me otherwise, the policy of being "slow but right" at the moment. Optimisation may be something the Zope 3 guys want to focus on once they've finished refactoring everything to death for the 7th or 8th time ;-) I'd be interested to see how Twiddler would compare against ZPT in your 2.10 test, mail me off list if you're interested... cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Hi Chris, On 10/5/06, Chris Withers <chris@simplistix.co.uk> wrote:
Pascal Peregrina wrote:
We made some basic load testing against different Zope2 versions and found some unexpected results.
What's unexpected in particular?
The average response time for the different types with different versions.
How many runs did you do for each setup and how did you average them?
There were a few runs for each case but I can't remember the exact number. Every run shown similar results. The last one was picked. The average shown is for the latest one.
Also, was the hardware, memory usage and processor load, excluding that used by Zope identical over the course of all runs? (common problems are doing load tests on a box that is also doing other things)
Hardware is detailed in the page. The box was doing nothing else except running one zope instance at the time.
Also, what was the config? Plain Zope with FileStorage? ZEO? etc...
FileStorage with stock zope.conf except for enabling wsgi were noted.
What do you mean by "wsgi enabled"? How does wsgi+ differ from wsgi?
The wsgi directive was enabled in 2.10.0b2. Re wsgi+, do you mean in the graphic? I need to correct that as the text was truncated. It should be wsgi+SpeedPack and wsgi+profile.
The speed drop in ZPT I think is due to using Zope 3's ZPT engine in 2.10. Zope 3 has, unless someone tells me otherwise, the policy of being "slow but right" at the moment. Optimisation may be something the Zope 3 guys want to focus on once they've finished refactoring everything to death for the 7th or 8th time ;-)
yes, i think that's the cause of the tal speedup in 2.10, but then static content is painfully slower. cheers, f.-
Federico Schwindt wrote:
What do you mean by "wsgi enabled"? How does wsgi+ differ from wsgi?
The wsgi directive was enabled in 2.10.0b2. Re wsgi+, do you mean in the graphic? I need to correct that as the text was truncated. It should be wsgi+SpeedPack and wsgi+profile.
The speed drop in ZPT I think is due to using Zope 3's ZPT engine in 2.10. Zope 3 has, unless someone tells me otherwise, the policy of being "slow but right" at the moment. Optimisation may be something the Zope 3 guys want to focus on once they've finished refactoring everything to death for the 7th or 8th time ;-)
Chris, the Zope 3 people actually spent a lot of time speeding up component architecture bits in Zope 3.3. People complain that its API got better, um, refactored, but it also got sped up quite a bit.
yes, i think that's the cause of the tal speedup in 2.10, but then static content is painfully slower. cheers,
It appears TAL is slower in 2.10.0b2 vanilla, but a lot faster in 2.10.b2 WSGI. That's pretty bizarre indeed. It might be WSGI somehow lifts performance issues elsewhere in the publisher stack, which then in this case more than compensate for what looks like a slight TAL slowdown. Why then the static content is so much slower is again weird. It almost looks like there's a fixed speed for WSGI. Someone who knows more about how the WSGI support may be able to say more... Regards, Martijn
Surprising indeed.
Why then the static content is so much slower is again weird. It almost looks like there's a fixed speed for WSGI. Someone who knows more about how the WSGI support may be able to say more...
WSGI in itself should not make a difference there. However, if the WSGI server used is Twisted, then Twisted has a higher overhead per request than ZServer. However, these results seem to indicate that this overhead suddenly becomes the major factor, which clearly is impossible.
the number of concurrent requests were 100
It's usually a good idea to test this with smaller numbers of concurrent requests as well, to see how speed reacts to increasing loads. Those WSGI tests make no sense, I think they are somehow faulty.
Ok, so far, list feedback is: + WSGI test looks wrong + 100 concurrent threads looks a little heavy + results are not the same as Tres' ones So, as soon as possible, we will try to make some more tests and will let you know when the wiki page is updated. Thanks a lot for the feedback so far. Pascal
De : Lennart Regebro <regebro@gmail.com> Date : Fri, 6 Oct 2006 11:21:04 +0200 À : Martijn Faassen <faassen@infrae.com> Cc : Federico Schwindt <federico.schwindt@gmail.com>, Chris Withers <chris@simplistix.co.uk>, Pascal Peregrina <Pperegrina@lastminute.com>, "zope@zope.org" <zope@zope.org> Objet : Re: [Zope] Surprising load test results?
Surprising indeed.
Why then the static content is so much slower is again weird. It almost looks like there's a fixed speed for WSGI. Someone who knows more about how the WSGI support may be able to say more...
WSGI in itself should not make a difference there. However, if the WSGI server used is Twisted, then Twisted has a higher overhead per request than ZServer. However, these results seem to indicate that this overhead suddenly becomes the major factor, which clearly is impossible.
the number of concurrent requests were 100
It's usually a good idea to test this with smaller numbers of concurrent requests as well, to see how speed reacts to increasing loads.
Those WSGI tests make no sense, I think they are somehow faulty.
********************************************************************** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com **********************************************************************
On 10/6/06, Pascal Peregrina <Pperegrina@lastminute.com> wrote:
+ 100 concurrent threads looks a little heavy
Or more to the point, you need to test with 1, 5, 10, 20 and 50 as well and look at performance effects of that. Just hitting it with a 100 and assuming that's useful and valid data doesn't work.
Hi all, I have added "ab" tests at the end of the page. (http://talk.lastminute.com/wiki/index.php/Loadtest) These extra results confirm everything we had found so far, except for 2.10 with use-wsgi "on"... We could see what others reported: in 2.10, WSGI adds a similar overhead to all type of requests (html, gif, swf, ...). So there must have been something wrong in our initial test. Again, Zope 2.8.8 remains the fastest version... Something new that we could not see in the previous test is the evolution of the results with the number of concurrent threads, as we have done the same test with 1,2,5,10,20,50 and 100 threads. Please have a look, and let me know what you think about these additional results. Thanks. Pascal ********************************************************************** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com **********************************************************************
On 10/9/06, Pascal Peregrina <Pperegrina@lastminute.com> wrote:
Hi all,
I have added "ab" tests at the end of the page. (http://talk.lastminute.com/wiki/index.php/Loadtest)
These extra results confirm everything we had found so far, except for 2.10 with use-wsgi "on"... We could see what others reported: in 2.10, WSGI adds a similar overhead to all type of requests (html, gif, swf, ...). So there must have been something wrong in our initial test.
Again, Zope 2.8.8 remains the fastest version...
Something new that we could not see in the previous test is the evolution of the results with the number of concurrent threads, as we have done the same test with 1,2,5,10,20,50 and 100 threads.
Well, as seen here the number of requests peak at around five threads, which is expected with Zopes default setting of four threads.
Please have a look, and let me know what you think about these additional results.
They seem to largely contractictyour earlier results, where the avergare response time for WSGI was similar for all document types. Now they are not. Also, this time, 2.10 without WSGI ends up somewhere between 2.9 and 2.10 with WSGI, while before 2.10 without WSGI was slightly slower (but not that much) than 2.9. So, basically, these tests are perhaps slightly less non-sensical than your earlier tests, but still surprising. I don't see why Zope 2.10 would be so much slower than 2.9 in serving static content. There has been no change there. Your first test was much more reasonable in that area. This test is on the otehr hand much more reasonable for the WSGI results. -- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/
Well, I should have removed the SpeedPack results for clarity... I might add these graphs today. But basically, the worrying thing is the performance degradation with each zope version... 2.8.8 seems to be the fastest, then 2.9, then 2.10... :( (even if you forget about WSGI) Pascal
De : Lennart Regebro <regebro@gmail.com> Date : Mon, 9 Oct 2006 21:46:39 +0200 À : Pascal Peregrina <Pperegrina@lastminute.com> Cc : "zope@zope.org" <zope@zope.org> Objet : Re: [Zope] Surprising load test results? (more results)
On 10/9/06, Pascal Peregrina <Pperegrina@lastminute.com> wrote:
Hi all,
I have added "ab" tests at the end of the page. (http://talk.lastminute.com/wiki/index.php/Loadtest)
These extra results confirm everything we had found so far, except for 2.10 with use-wsgi "on"... We could see what others reported: in 2.10, WSGI adds a similar overhead to all type of requests (html, gif, swf, ...). So there must have been something wrong in our initial test.
Again, Zope 2.8.8 remains the fastest version...
Something new that we could not see in the previous test is the evolution of the results with the number of concurrent threads, as we have done the same test with 1,2,5,10,20,50 and 100 threads.
Well, as seen here the number of requests peak at around five threads, which is expected with Zopes default setting of four threads.
Please have a look, and let me know what you think about these additional results.
They seem to largely contractictyour earlier results, where the avergare response time for WSGI was similar for all document types. Now they are not. Also, this time, 2.10 without WSGI ends up somewhere between 2.9 and 2.10 with WSGI, while before 2.10 without WSGI was slightly slower (but not that much) than 2.9.
So, basically, these tests are perhaps slightly less non-sensical than your earlier tests, but still surprising. I don't see why Zope 2.10 would be so much slower than 2.9 in serving static content. There has been no change there. Your first test was much more reasonable in that area. This test is on the otehr hand much more reasonable for the WSGI results.
-- Lennart Regebro, Nuxeo http://www.nuxeo.com/ CPS Content Management http://www.nuxeo.org/
********************************************************************** This email and any files transmitted with it are confidential and intended solely for the use of the individual or entity to whom they are addressed. If you have received this email in error please notify the system manager. This footnote also confirms that this email message has been swept by MIMEsweeper for the presence of computer viruses. www.mimesweeper.com **********************************************************************
Pascal Peregrina wrote:
But basically, the worrying thing is the performance degradation with each zope version... 2.8.8 seems to be the fastest, then 2.9, then 2.10... :( (even if you forget about WSGI)
Yep, maybe it's time to have a "push for performance" in both the Zope 2 and Zope 3 worlds? (I suspect the bulk of the work is to be done in the Zope 3 stuff...) Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Pascal Peregrina wrote:
Well, I should have removed the SpeedPack results for clarity... I might add these graphs today.
But basically, the worrying thing is the performance degradation with each zope version... 2.8.8 seems to be the fastest, then 2.9, then 2.10... :( (even if you forget about WSGI)
There deltas appear to be as follows (ignoring the Speedpack / WSGI results): Best Case (-c 5) Worst Case (-c 100) 2.8.8-2.9.4 2.9.4-2.10.0b2 2.8.8-2.9.4 2.9.4-2.10.0b2 ----- ----------- -------------- ----------- -------------- SWF 0.6 ms 0.5 ms 1.4 ms -0.3 ms GIF 0.2 ms 0.5 ms 0.3 ms 0.6 ms ZPT 0.3 ms 1.1 ms 0.3 ms 1.2 ms Note that the worst-case for the SWF file actually improved from 2.9.4 to 2.10.0b2 (the speedpack colors are a little confusing). There does appear to be a real bump in both ZPublisher owverhead (OFS.Image.File.index_html has not changed appreciably across the versions) and in ZPT rendering time. We probably need to audit the changes made to the Zope2 version of ZPT after the Zope3 version was forked, and find any optimizations which were made only there. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFK735+gerLs4ltQ4RAlGUAKDZYpwnf9Ro+I/VgzrqX1bwS6CkyQCdG16y OOjazPLN+zclxU86rTRrwVY= =sTLA -----END PGP SIGNATURE-----
Martijn Faassen wrote:
The speed drop in ZPT I think is due to using Zope 3's ZPT engine in 2.10. Zope 3 has, unless someone tells me otherwise, the policy of being "slow but right" at the moment. Optimisation may be something the Zope 3 guys want to focus on once they've finished refactoring everything to death for the 7th or 8th time ;-)
Chris, the Zope 3 people actually spent a lot of time speeding up component architecture bits in Zope 3.3. People complain that its API got better, um, refactored, but it also got sped up quite a bit.
Sorry, yes, that wasn't a fair comment and I apologise.
It appears TAL is slower in 2.10.0b2 vanilla, but a lot faster in 2.10.b2 WSGI. That's pretty bizarre indeed.
If I read Tres comment correctly, he's saying "I don't trust your test results" ;-) cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Federico Schwindt wrote:
What do you mean by "wsgi enabled"? How does wsgi+ differ from wsgi?
The wsgi directive was enabled in 2.10.0b2.
Okay, but what does this actually do?
Re wsgi+, do you mean in the graphic? I need to correct that as the text was truncated. It should be wsgi+SpeedPack and wsgi+profile.
Right.
The speed drop in ZPT I think is due to using Zope 3's ZPT engine in 2.10. Zope 3 has, unless someone tells me otherwise, the policy of being "slow but right" at the moment. Optimisation may be something the Zope 3 guys want to focus on once they've finished refactoring everything to death for the 7th or 8th time ;-)
yes, i think that's the cause of the tal speedup in 2.10,
Urm, I was commenting that tal had slowed down in 2.10...
but then static content is painfully slower.
Really? Is that a comment based on the graphs or on actual use? cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On 10/6/06, Chris Withers <chris@simplistix.co.uk> wrote:
Federico Schwindt wrote:
What do you mean by "wsgi enabled"? How does wsgi+ differ from wsgi?
The wsgi directive was enabled in 2.10.0b2.
Okay, but what does this actually do?
use a different web server.
[..]
The speed drop in ZPT I think is due to using Zope 3's ZPT engine in 2.10. Zope 3 has, unless someone tells me otherwise, the policy of being "slow but right" at the moment. Optimisation may be something the Zope 3 guys want to focus on once they've finished refactoring everything to death for the 7th or 8th time ;-)
yes, i think that's the cause of the tal speedup in 2.10,
Urm, I was commenting that tal had slowed down in 2.10...
yes, i got messed up here, sorry. i meant that with wsgi it seemed to be a bit faster but then all the results posted by tres indicates otherwise so we'll repeat the tests and see what the results are.
but then static content is painfully slower.
Really? Is that a comment based on the graphs or on actual use?
tests results. .f-
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Pascal Peregrina wrote:
We made some basic load testing against different Zope2 versions and found some unexpected results. Everything is summarized here : http://talk.lastminute.com/wiki/index.php/Loadtest
We would appreciate any comment about it, and of course will answer any question you might have about our test.
Can you tell us what the scale is on the graph? Also, for the benefit of those "playing along at home", I'm attaching a .zexp file of the test tree, made according to your spec. My results, using 'ab -c 4 -n 100' against that site, are as follows (all times are in ms, averaged across all requests): Zope Version test.html flag.gif anim.swf - ------------------ --------- -------- -------- Zope 2.7 trunk 6.4 4.4 5.1 Zope 2.8 trunk 6.6 5.2 5.5 Zope 2.9 trunk 6.6 5.1 6.0 Zope 2.10 trunk 7.9 5.5 6.3 Zope 2.10 trunk 8.3 6.0 7.5 with WSGI I'm *not* using psyco anywhere in these tests, and running them on my 1.6 Ghz laptop, running Ubuntu Dapper. Observations: 1. The big jump in ZPT rendering is between 2.9 and 2.10 (~20%), which corresponds to the switch to use Zope3's ZPT. Maybe we lost som optimizations in that switch? 2. WSGI never wins, which isn't surprising: it adds overhead to the publishing process (0.5 ms or so, likely not a big problem). Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFJTYW+gerLs4ltQ4RAhWwAJ4oIxNZmNVTvjRZ1fKnlc7Fmwb6AwCg1elL h5G3ruAjas5Y2/0Ga2l0Y3A= =Nf/Z -----END PGP SIGNATURE-----
On 10/5/06, Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Pascal Peregrina wrote:
We made some basic load testing against different Zope2 versions and found some unexpected results. Everything is summarized here : http://talk.lastminute.com/wiki/index.php/Loadtest
We would appreciate any comment about it, and of course will answer any question you might have about our test.
Can you tell us what the scale is on the graph? Also, for the benefit of those "playing along at home", I'm attaching a .zexp file of the test tree, made according to your spec.
The tests you've made are substantially different. If you look at the zip files, the number of concurrent requests were 100. I'll try to explain the parameters used in the tests tomorrow to clarify the situation. f.-
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Federico Schwindt wrote:
On 10/5/06, Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Pascal Peregrina wrote:
We made some basic load testing against different Zope2 versions and found some unexpected results. Everything is summarized here : http://talk.lastminute.com/wiki/index.php/Loadtest
We would appreciate any comment about it, and of course will answer any question you might have about our test.
Can you tell us what the scale is on the graph? Also, for the benefit of those "playing along at home", I'm attaching a .zexp file of the test tree, made according to your spec.
The tests you've made are substantially different. If you look at the zip files, the number of concurrent requests were 100. I'll try to explain the parameters used in the tests tomorrow to clarify the situation.
f.- _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
OK, here is with 100 concurrent: $ ab -c 100 -n 1000 http://localhost:8080/test/test2/test3/test.html This is ApacheBench, Version 2.0.41-dev <$Revision: 1.141 $> apache-2.0 Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright (c) 1998-2002 The Apache Software Foundation, http://www.apache.org/ Benchmarking localhost (be patient) Completed 100 requests Completed 200 requests Completed 300 requests Completed 400 requests Completed 500 requests Completed 600 requests Completed 700 requests Completed 800 requests Completed 900 requests Finished 1000 requests Server Software: Zope/(Zope Server Hostname: localhost Server Port: 8080 Document Path: /test/test2/test3/test.html Document Length: 326 bytes Concurrency Level: 100 Time taken for tests: 6.116692 seconds Complete requests: 1000 Failed requests: 0 Write errors: 0 Total transferred: 540214 bytes HTML transferred: 342626 bytes Requests per second: 163.49 [#/sec] (mean) Time per request: 611.669 [ms] (mean) Time per request: 6.117 [ms] (mean, across all concurrent requests) Transfer rate: 86.16 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 0 1.6 0 6 Processing: 174 586 209.2 584 6096 Waiting: 170 570 206.8 577 6086 Total: 180 587 209.3 584 6102 Percentage of the requests served within a certain time (ms) 50% 584 66% 594 75% 619 80% 656 90% 695 95% 826 98% 833 99% 864 100% 6102 (longest request) Note that the per-request average is still around 6 ms (Zope 2.7 here). Increasing the concurrency above 4 doesn't stress Zope CPU-wise, because (by default) there are only 4 appserver "worker" threads available to respond to requests. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.2.2 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFFJX7M+gerLs4ltQ4RAtxTAJ4waJhWazqiaAvJvkMoKvEXS7VCAwCglIiK wxClKwFBUwIC9RmS/AIbujo= =ORqb -----END PGP SIGNATURE-----
participants (6)
-
Chris Withers -
Federico Schwindt -
Lennart Regebro -
Martijn Faassen -
Pascal Peregrina -
Tres Seaver