Hello Matt, Thanks for writing in. --- Matt Hamilton <matth@netsight.co.uk> wrote:
Trevor Warren wrote: Look at the RAMCacheManager included with Zope. This will make a big difference. [snip]
Yep matt, someone just suggested the same. Am looking at using the capability of the same towards boosting up the response times and TPS.
Dynamic Thread Management Features in Zope You can adjust the number of threads in the zope.conf file (for Zope 2.7).
[snip] Have tried that out too. But for a 100 user with ear user performing 1 iteration with 110 threads enabled in the first round. And 70 in the second. In both the instance zope response times goes for a toss post 10 mintues of bombardment. It recovers at the end of the 15 minute test. Was hoping for atleast a sustained performance throughput. The number of threads (70-110) were kept proportional to the number of users(100). Performance degradation post 10mins of the test tleading to absymal response times.
Be careful with this. Unlike most other systems, Zope has its own built in object store (ZODB) and hence does not 'look' like many other systems that have an app server and then persist the data to a separate SQL server. [snip]
Okie captain.
hints. If you are doing something like an SQL select from a RDBMS and then iterating over a large result set, there are some well known techniques to speed things up. [snip]
Here at the performance group we do have access to some excellent repositoried on tuning all kinds of apps/db servers, but i will surely get back with results of sql tests when i start them. These tests i am mentioning are short circuit tests Matt. Tests w/o any sql queries as of now.
code. One thing to mention, is that python does not utilize multiple processes very well. [snip]
ZEO i am checking out rite now. Could you please spend some time on your current statement about python not scaling across processes. If this is the case how can we expect Zope based on python threads to scale over SMP based systems service large number of concurrent users.
Hence if you have a multi-processor server, you are better off running multiple ZEO clients, one for each processor. [snip]
Great..Will try out the same matt.
I hope this helps. If you can send us any more information about your test setup and benchmarks we may be able to help more. :)
[snip] You betcha. Am working on the same and will keep all of you posted on the same. Thanks a million for responding. Lets hope i am able to push zope to the wall and extract acceptable TPS values as compared to commercial offerings. Trevor
-Matt
-- Matt Hamilton matth@netsight.co.uk Netsight Internet Solutions, Ltd. Business Vision on the Internet http://www.netsight.co.uk +44 (0)117 9090901 Web Design | Zope/Plone Development & Consulting | Co-location | Hosting
===== ( >- -< ) /~\ __ Scaling FLOSS in the __ /~\ | \) / Enterprise : trevor.w@pvision.biz \ (/ | |_|_ \ Call Now: 9820349221 / _|_| \____________________________________/ __________________________________ Do you Yahoo!? SBC Yahoo! - Internet access at a great low price. http://promo.yahoo.com/sbc/
On Tue, May 18, 2004 at 10:06:58PM -0700, Trevor Warren wrote:
The number of threads (70-110) were kept proportional to the number of users(100). Performance degradation post 10mins of the test tleading to absymal response times.
FYI, that's a really unusually high thread count for zope. You may actually be hurting yourself there by adding overhead for all those threads. Also, keep the number of ZODB connections at least as high as the number of worker threads. -- Paul Winkler http://www.slinkp.com
Hello Paul, --- Paul Winkler <pw_lists@slinkp.com> wrote:
FYI, that's a really unusually high thread count for zope. You may actually be hurting yourself there by adding overhead for all those threads. [snip]
Okie. For a 50 user test what would be the number of child threads required towards servicing the 50+ concurrent iterations.
Also, keep the number of ZODB connections at least as high as the number of worker threads.
[snip] Where does one tweak this????. Would appreciate a pointer on the same please. Trevor
--
Paul Winkler
===== ( >- -< ) /~\ __ Scaling FLOSS in the __ /~\ | \) / Enterprise : trevor.w@pvision.biz \ (/ | |_|_ \ Call Now: 9820349221 / _|_| \____________________________________/ __________________________________ Do you Yahoo!? SBC Yahoo! - Internet access at a great low price. http://promo.yahoo.com/sbc/
On Wed, 2004-05-19 at 01:59, Trevor Warren wrote:
Hello Paul,
--- Paul Winkler <pw_lists@slinkp.com> wrote:
FYI, that's a really unusually high thread count for zope. You may actually be hurting yourself there by adding overhead for all those threads. [snip]
Okie. For a 50 user test what would be the number of child threads required towards servicing the 50+ concurrent iterations.
A test emulating 50 users doesn't necessarily need 50 threads. Highly concurrent Zope applications usually get away with using 4-7 threads for this purpose.
Also, keep the number of ZODB connections at least as high as the number of worker threads. [snip]
Where does one tweak this????. Would appreciate a pointer on the same please.
This is explained on http://www.plope.com/misc/szweb/img19.html zserver-threads in zope.conf is the number of Zope threads the "pool-size" parameter in the <zodb_db main> section of zope.conf is the number of ZODB connections. So if I wanted to futz around with it by bumping up the number of Zope threads to 20, I might have something like this in my zope.conf file: zserver-threads 20 <zodb_db main> # Main FileStorage database <filestorage> path $INSTANCE/var/Data.fs </filestorage> mount-point / pool-size 25 </zodb_db> You can retest after changing these numbers to various things. Just keep pool-size at some number just above the number of threads and you should be ok. Another important setting is explained on http://www.plope.com/misc/szweb/img22.html. Make *sure* to turn off debug mode in zope.conf. By default it is on (to help developers). Put this in your zope.conf file to turn it off: debug-mode off A final important setting, the ZODB cache size, is explained on http://www.plope.com/misc/szweb/img20.html . It can effect performance dramatically. It's controlled by the cache-size parameter of the zodb_db section like so: <zodb_db main> # Main FileStorage database <filestorage> path $INSTANCE/var/Data.fs </filestorage> mount-point / pool-size 25 cache-size 20000 </zodb_db> I'll be interested to hear your findings after you play around with this stuff. - C
Hello Chris, 50 users, short circuit(no sql), 1 iteration per user, test duration 15mins. Threads are 10 and DB connections are 25. Will add DB Cache in next iteration. Have made only one change, cant change all params at once such as to gauge performance change. Test progressing well until 7th minute. hits/sec fallen to <50 from 250/sec and resp times gone up dramatically. TPS is at 25+ currently for the set of 50 users. Overall throughput has increased drastically. About a 100% increase than the previous set of tests. Will get back.....!!! Trevor ===== ( >- -< ) /~\ __ Scaling FLOSS in the __ /~\ | \) / Enterprise : trevor.w@pvision.biz \ (/ | |_|_ \ Call Now: 9820349221 / _|_| \____________________________________/ __________________________________ Do you Yahoo!? SBC Yahoo! - Internet access at a great low price. http://promo.yahoo.com/sbc/
On Wed, 2004-05-19 at 03:20, Trevor Warren wrote:
Hello Chris,
50 users, short circuit(no sql), 1 iteration per user, test duration 15mins. Threads are 10 and DB connections are 25.
FWIW, not really a big deal, but you have approximately 14 DB connections sitting around not doing much except potentially consuming some meory, you can probably bump the number of DB connections down to 11 or so for 10 threads and get the same results.
Will add DB Cache in next iteration.
Right.
Test progressing well until 7th minute. hits/sec fallen to <50 from 250/sec and resp times gone up dramatically. TPS is at 25+ currently for the set of 50 users.
No idea what this could be. I still don't know what a transaction is to you, so TPS doesn't mean much to me. hits/sec does, though, and having the hitrate drop by 200 hits a second sounds very suspicious. I'd make sure your test rig is doing what you think it's doing. Benchmarking is tricky. Usually I'll try to "prime" the server before trying any benchmarking. I'll start up the server and do a test run first before timing anything. That populates all of the caches and whatnot so I'm not measuring any initialization overhead. I've also found that sometimes it's the test *client* that is at fault for strange results (like, not generating enough requests to saturate the server). I suspect that may be the problem in your setup. - C
Hey Chris, Yep i do accept that the client performing the test could be at fault but that may not be the case here. A 20 user test load is as fishy as the rest...:D. The test is for a duration of 15mins. Hit rate remains constant at 150 hits/sec. Suddenly post 7th minute Zope goes for a toss. This is really surprising. Initially i was under the impression that the test for 50 imultaenous users was taking the server down. But now even this test for 20 uses post the 7th minute send zope for a toss. Hit rate falls from 150 hits/sec to <40hits per sec and resp times more than quadruple. ZopeDB caching 20000 + Threads for Zope at 10 + ZODB thread pool at 10. Should i look at apache/zope mix???. Havent tried out RaMCacheManager yet. Or whatever that was. Trevor ===== ( >- -< ) /~\ __ Scaling FLOSS in the __ /~\ | \) / Enterprise : trevor.w@pvision.biz \ (/ | |_|_ \ Call Now: 9820349221 / _|_| \____________________________________/ __________________________________ Do you Yahoo!? SBC Yahoo! - Internet access at a great low price. http://promo.yahoo.com/sbc/
Trevor Warren wrote:
A 20 user test load is as fishy as the rest...:D. The test is for a duration of 15mins. Hit rate remains constant at 150 hits/sec. Suddenly post 7th minute Zope goes for a toss. This is really surprising.
Initially i was under the impression that the test for 50 imultaenous users was taking the server down. But now even this test for 20 uses post the 7th minute send zope for a toss.
I'm wondering if during your test the server is starting a garbage collection of the cache or something like that at the 7th minute. Again using ZEO will help alliviate this as the different ZEO clients will GC at different times. -Matt -- Matt Hamilton matth@netsight.co.uk Netsight Internet Solutions, Ltd. Business Vision on the Internet http://www.netsight.co.uk +44 (0)117 9090901 Web Design | Zope/Plone Development & Consulting | Co-location | Hosting
--- Matt Hamilton <matth@netsight.co.uk> wrote:
I'm wondering if during your test the server is starting a garbage collection of the cache or something like that at the 7th minute. Again using ZEO will help alliviate this as the different ZEO clients will GC at different times. [snip]
Matt...having GC throwing the response times thro the roof and sending the server response for a toss isn't acceptable. Anyone specifically working on the performance aspects of Zope code Matt????. Trevor
-Matt
===== ( >- -< ) /~\ __ Scaling FLOSS in the __ /~\ | \) / Enterprise : trevor.w@pvision.biz \ (/ | |_|_ \ Call Now: 9820349221 / _|_| \____________________________________/ __________________________________ Do you Yahoo!? Yahoo! Domains � Claim yours for only $14.70/year http://smallbusiness.promotions.yahoo.com/offer
You can help by confirming that your slowdown is or isn't due to garbage collection by applying the attached diff to the Zope 2.7.0 source code (it turns Python's cyclic garbage collector off). On Wed, 2004-05-19 at 22:59, Trevor Warren wrote:
--- Matt Hamilton <matth@netsight.co.uk> wrote:
I'm wondering if during your test the server is starting a garbage collection of the cache or something like that at the 7th minute. Again using ZEO will help alliviate this as the different ZEO clients will GC at different times. [snip]
Matt...having GC throwing the response times thro the roof and sending the server response for a toss isn't acceptable.
Anyone specifically working on the performance aspects of Zope code Matt????.
Trevor
-Matt
===== ( >- -< ) /~\ __ Scaling FLOSS in the __ /~\ | \) / Enterprise : trevor.w@pvision.biz \ (/ | |_|_ \ Call Now: 9820349221 / _|_| \____________________________________/
__________________________________ Do you Yahoo!? Yahoo! Domains Claim yours for only $14.70/year http://smallbusiness.promotions.yahoo.com/offer
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Trevor Warren wrote:
Anyone specifically working on the performance aspects of Zope code Matt????.
No, because no-one apart from you seems ot perceive there's a problem here... Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
--- Chris Withers <chris@simplistix.co.uk> wrote:
No, because no-one apart from you seems ot perceive there's a problem here... [snip]
A 2cpu, 1GB Ram machine, 20 users and Zope just doesn't scale. I don't think i want to start a flame war Chris. Being a FLOSS supporter i would more than luv to propogate the same here. But to accept that it Zope can't scale and work on it rather than tell me that one will have to buy more boxes to c Zope scale isn't acceptable. Bottomline....it just doesn't scale. A simple 3 page test give me throughput so bad i can't speak of. Zope didn't scale...hence i tried Zeo/Zope....that didn't scale too(On the same machine). Am trying Zope/Zeo across machines now and will let you know about the results. Apache+Zope is something i havent looked at yet. Probably python is the issue here...but accepting the facts will let us move ahead and improve chris. We can't sit on our laurels when there literally isn't any. Trevor
Chris
===== ( >- -< ) /~\ __ Scaling FLOSS in the __ /~\ | \) / Enterprise : trevor.w@pvision.biz \ (/ | |_|_ \ Call Now: 9820349221 / _|_| \____________________________________/ __________________________________ Do you Yahoo!? Yahoo! Domains � Claim yours for only $14.70/year http://smallbusiness.promotions.yahoo.com/offer
Trevor Warren wrote:
Bottomline....it just doesn't scale. A simple 3 page test give me throughput so bad i can't speak of. Zope didn't scale...hence i tried Zeo/Zope....that didn't scale too(On the same machine). Am trying Zope/Zeo across machines now and will let you know about the results. Apache+Zope is something i havent looked at yet.
You must be doing *something* wrong. Not sure what yet. There are already quite a few large sites running Zope, and they seem to be doing just fine. Our main Zope server is a single box (Dual PIII 1.4GhZ) and during the day runs at 10 req/sec averaged over a 5 minute period. Not exactly huge, but this is driving over 60 *actual* sites, most with quite a bit of complexity (or rather not much time spent on optimization) and the 5 minute load average never goes above 0.4. And that is a server running a number of zope instances, databases, and other web related stuff. www.lastminute.com is running Zope. I am not sure of the specifics of hardware, but it is a ZEO cluster with squid caching. I would imagine they load test to quite high loads. We are working with a company to put a divisional 15,000 user intranet in place on a cluster of four ZEO (dual proc Xeon) clients and a single ZEO server. This hardware is overkill for that load, but we want expansion capacity. There will potentiallly be quite a bit of per-user personalisation, which is expensive. I'm sure others have plenty of other case studies. Chris McDonough has done quite a bit of study into this. Trust him. :)
Probably python is the issue here...but accepting the facts will let us move ahead and improve chris. We can't sit on our laurels when there literally isn't
Yes, python could be an issue. As I mentioned before, we saw issues with python threading on Solaris, and due to thread deadlocks the performance suffered very badly. What OS are you running this on? What database adapter are you using? I rememeber there were some issues with some of the database adapters causing trouble with threading (again on Solaris) are you still actually calling the SQL method in your 'short circuited' tests? Just checking our setup, and: ab -n 50 -c 5 http://www.netsight.co.uk/ gives 22 req/sec ab -n 100 -c 20 http://www.netsight.co.uk/ gives 30 req/sec This is on a page that does a resonable amount of computation, it gathers news articles, truncates the summary, displays them, builds the menu dynamically etc. I have not done any tuning on it, it was written in half an hour, and those results aboce are on a server that is already serving about 6 req/sec to real users. The Zope server is behind Apache, but apache is not caching this page. -Matt -- Matt Hamilton matth@netsight.co.uk Netsight Internet Solutions, Ltd. Business Vision on the Internet http://www.netsight.co.uk +44 (0)117 9090901 Web Design | Zope/Plone Development & Consulting | Co-location | Hosting
--- Matt Hamilton <matth@netsight.co.uk> wrote:
server is behind Apache, but apache is not caching this page. [snip]
With 3 Zope Servers eache on 2 machines and 1 Zeo Client i get hits/sec of ~100. My total transaction throughput tho is still in the pits...:D. Its about ~10. Trevor
-Matt
===== ( >- -< ) /~\ __ Scaling FLOSS in the __ /~\ | \) / Enterprise : trevor.w@pvision.biz \ (/ | |_|_ \ Call Now: 9820349221 / _|_| \____________________________________/ __________________________________ Do you Yahoo!? Yahoo! Domains � Claim yours for only $14.70/year http://smallbusiness.promotions.yahoo.com/offer
Trevor Warren wrote:
--- Matt Hamilton <matth@netsight.co.uk> wrote:
server is behind Apache, but apache is not caching this page.
[snip]
With 3 Zope Servers eache on 2 machines and 1 Zeo Client i get hits/sec of ~100. My total transaction throughput tho is still in the pits...:D.
Without any more information about what the transactions are, or what the code is you have written, I can't really help anymore. A very simple ZPT (e.g. one with the default code in it when you add a new ZPT) gets around about 200 req/sec on a single ZEO client here. Again, I don't know what your transactions are. Can you post the code? -Matt -- Matt Hamilton matth@netsight.co.uk Netsight Internet Solutions, Ltd. Business Vision on the Internet http://www.netsight.co.uk +44 (0)117 9090901 Web Design | Zope/Plone Development & Consulting | Co-location | Hosting
Hey Matt, --- Matt Hamilton <matth@netsight.co.uk> wrote:
Trevor Warren wrote: Again, I don't know what your transactions are. Can you post the code? [snip]
Will send the code across to you directly instead of posting the whole file to the list. What would be a better way of handling the images below??. Is the current code doable??. <P ALIGN="center"> <dtml-with shopcart.gif> <A href="tpcw" ><IMG height=30 alt="Shopping Cart" src="<dtml-var absolute_url>" width=120 border=0></A> </dtml-with> <dtml-with search.gif> <A href="searchrequest" ><IMG height=30 alt=Search src="<dtml-var absolute_url>" width=120 border=0></A> </dtml-with> <dtml-with orderstatus.gif> <A href="tpcw" ><IMG height=30 alt="Order Status" src="<dtml-var absolute_url>" width=120 border=0></A> </dtml-with> Trevor
-Matt
===== ( >- -< ) /~\ __ Scaling FLOSS in the __ /~\ | \) / Enterprise : trevor.w@pvision.biz \ (/ | |_|_ \ Call Now: 9820349221 / _|_| \____________________________________/ __________________________________ Do you Yahoo!? Yahoo! Domains � Claim yours for only $14.70/year http://smallbusiness.promotions.yahoo.com/offer
On Fri, May 21, 2004 at 06:00:10AM -0700, Trevor Warren wrote:
Hey Matt,
--- Matt Hamilton <matth@netsight.co.uk> wrote:
Trevor Warren wrote: Again, I don't know what your transactions are. Can you post the code? [snip]
Will send the code across to you directly instead of posting the whole file to the list.
What would be a better way of handling the images below??. Is the current code doable??.
(snip)
<dtml-with shopcart.gif> <A href="tpcw" ><IMG height=30 alt="Shopping Cart" src="<dtml-var absolute_url>" width=120 border=0></A> </dtml-with>
(snip) Not an optimization, but a simplification... If you give shopcart.gif a Title property with the value "Shopping Cart", and assuming the height and width you specified above are the actual height/width of the image, you can replace the above 4 lines with: <a href="tpcw"><dtml-var shopcart.gif></a> It will render the height, width, and alt attributes for you. It will also do border=0 by default. -- Paul Winkler http://www.slinkp.com
Just to throw into the debate: AMD 2.13 Ghz, 1.5 GB Ram, 2 x 36 GB SCSI (Raid 1) - Red Hat Enterprise Linux ES 2.1 - Python 2.3.3, Zope 2.7, Apache 2 Proxy Pass to Zope VHM - Also runs mail, webmail, other apache sites, daily stats, etc All the Zope sites combined run at about 100,000 hits a day (give or take on an average day). Since I have a lot of RAM (Zope/Python love RAM) I have it up to 40,000 objects per thread and about 10 threads active. It runs at about 45% of RAM with 230,000 objects loaded out of 110,000+ objects in ZODB with Zope being up (no restart) for 33 days. So all, of that combined, the server on a daily average (using sa command) at about 3 to 5% of capacity. Now, I have some RAM caches, etc but there are also some Zope apps that both read/write to the ZODB a lot (AdManager) and some that take some horsepower (ZUBB). Jake -- http://www.ZopeZone.com Matt Hamilton said:
Trevor Warren wrote:
Bottomline....it just doesn't scale. A simple 3 page test give me throughput so bad i can't speak of. Zope didn't scale...hence i tried Zeo/Zope....that didn't scale too(On the same machine). Am trying Zope/Zeo across machines now and will let you know about the results. Apache+Zope is something i havent looked at yet.
You must be doing *something* wrong. Not sure what yet. There are already quite a few large sites running Zope, and they seem to be doing just fine.
Our main Zope server is a single box (Dual PIII 1.4GhZ) and during the day runs at 10 req/sec averaged over a 5 minute period. Not exactly huge, but this is driving over 60 *actual* sites, most with quite a bit of complexity (or rather not much time spent on optimization) and the 5 minute load average never goes above 0.4. And that is a server running a number of zope instances, databases, and other web related stuff.
www.lastminute.com is running Zope. I am not sure of the specifics of hardware, but it is a ZEO cluster with squid caching. I would imagine they load test to quite high loads.
We are working with a company to put a divisional 15,000 user intranet in place on a cluster of four ZEO (dual proc Xeon) clients and a single ZEO server. This hardware is overkill for that load, but we want expansion capacity. There will potentiallly be quite a bit of per-user personalisation, which is expensive.
I'm sure others have plenty of other case studies. Chris McDonough has done quite a bit of study into this. Trust him. :)
Probably python is the issue here...but accepting the facts will let us move ahead and improve chris. We can't sit on our laurels when there literally isn't
Yes, python could be an issue. As I mentioned before, we saw issues with python threading on Solaris, and due to thread deadlocks the performance suffered very badly. What OS are you running this on? What database adapter are you using? I rememeber there were some issues with some of the database adapters causing trouble with threading (again on Solaris) are you still actually calling the SQL method in your 'short circuited' tests?
Just checking our setup, and:
ab -n 50 -c 5 http://www.netsight.co.uk/ gives 22 req/sec ab -n 100 -c 20 http://www.netsight.co.uk/ gives 30 req/sec
This is on a page that does a resonable amount of computation, it gathers news articles, truncates the summary, displays them, builds the menu dynamically etc. I have not done any tuning on it, it was written in half an hour, and those results aboce are on a server that is already serving about 6 req/sec to real users. The Zope server is behind Apache, but apache is not caching this page.
-Matt
-- Matt Hamilton matth@netsight.co.uk Netsight Internet Solutions, Ltd. Business Vision on the Internet http://www.netsight.co.uk +44 (0)117 9090901 Web Design | Zope/Plone Development & Consulting | Co-location | Hosting
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
On Fri, May 21, 2004 at 03:24:17AM -0700, Trevor Warren wrote:
--- Chris Withers <chris@simplistix.co.uk> wrote:
No, because no-one apart from you seems ot perceive there's a problem here... [snip]
A 2cpu, 1GB Ram machine, 20 users and Zope just doesn't scale. I don't think i want to start a flame war Chris.
<grumble> Well then, please refrain from making vague assertions that contradict our experience and only underscore the lack of attention you have paid to all the good free advice you are getting. </grumble>
Bottomline....it just doesn't scale. A simple 3 page test give me throughput so bad i can't speak of. Zope didn't scale...hence i tried Zeo/Zope....that didn't scale too(On the same machine).
Did you put a load balancer in front of them? Did your test program hit the load balancer or one of the zope servers directly? Did you bind each zope server to a different CPU?
Am trying Zope/Zeo across machines now and will let you know about the results. Apache+Zope is something i havent looked at yet.
Running zope behind a caching proxy, whether apache or squid, is the best practice way to get performance. <grumble> We've been telling you that since this thread started... </grumble> -- Paul Winkler http://www.slinkp.com
Trevor Warren wrote:
A 2cpu, 1GB Ram machine, 20 users and Zope just doesn't scale. I don't think i want to start a flame war Chris.
Then why are you trolling so much? ;-)
propogate the same here. But to accept that it Zope can't scale and work on it rather than tell me that one will have to buy more boxes to c Zope scale isn't acceptable.
Zope scales fine for everyone except you ;-)
Bottomline....it just doesn't scale.
Bullshit.
A simple 3 page test give me throughput so bad i can't speak of.
Then you're doing something very wrong. But 3 static pages is a pointless use of Zope anyway, so why bother benchamrking it?
didn't scale...hence i tried Zeo/Zope....that didn't scale too(On the same machine).
Zope/ZEO on the same machine will make things slower, but will give you the felxibility to bring in new machine and do some more interesting data import things without any changes...
results. Apache+Zope is something i havent looked at yet.
?!
Probably python is the issue here...
No, I think the user is the issue here... Chris - who's having a "grumpy" day -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
On Tue, May 25, 2004 at 02:55:11PM +0100, Chris Withers wrote: (snip)
Chris - who's having a "grumpy" day
... and should have read more of the list before posting ;-) Trevor sorted out his issues, apparently his test client was bogus. -- Paul Winkler http://www.slinkp.com
Paul Winkler wrote:
On Tue, May 25, 2004 at 02:55:11PM +0100, Chris Withers wrote: (snip)
Chris - who's having a "grumpy" day
... and should have read more of the list before posting ;-) Trevor sorted out his issues, apparently his test client was bogus.
What?! You mean *shock* *horror* Zope DOES actually scale? Yaknow, I NEVER would have though it ;-) Chris - who read all the thread and is STILL having a grumpy day ;-) -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
--- Chris Withers <chris@simplistix.co.uk> wrote:
No, I think the user is the issue here...
Chris - who's having a "grumpy" day [snip]
Chris, take a hike. BTW, Python still is the biggest bottle neck Zope has towards scaling on one cpu. Zope does give acceptable results but far lesser than other commercial app servers. I am thankful to the community for the help and support. And am happy with the extent to which zope scales for me. PN: You need to clean up your vocab before you post such abusive languae on a mailing list. Admin are you listening??? Trevor
===== ( >- -< ) /~\ ______________________________________ /~\ | \) / Scaling FLOSS in the Enterprise \ (/ | |_|_ \ trevorwarren@yahoo.com / _|_| \____________________________________/ __________________________________ Do you Yahoo!? Friends. Fun. Try the all-new Yahoo! Messenger. http://messenger.yahoo.com/
Trevor Warren wrote:
BTW, Python still is the biggest bottle neck Zope has towards scaling on one cpu. Zope does give acceptable results but far lesser than other commercial app servers.
Give me an example where the speed differences you're whining about make a perceptable difference from a user point of view, especially given how cheap hardware is nowadays. If you want raw speed, put 100GB of RAM in the box and run the whole damn thing through Psyco. Raw machine code fast enough for you? (anyone seen Armin Rigo's presentation, it rocks :-)
PN: You need to clean up your vocab before you post such abusive languae on a mailing list. Admin are you listening???
The temptation to say "bite me, muppet boy" in response is almost overwhelming... *grinz* Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk
Trevor Warren wrote at 2004-5-19 00:20 -0700:
... Test progressing well until 7th minute. hits/sec fallen to <50 from 250/sec and resp times gone up dramatically. TPS is at 25+ currently for the set of 50 users.
You need to understand what happens near the 7th minute. Does the memory consumption increase? I observed that Zope becomes slower and slower as it takes more and more memory. I attribute this to effects of the cyclic garbage collector. -- Dieter
Morning Dieter, --- Dieter Maurer <dieter@handshake.de> wrote:
You need to understand what happens near the 7th minute.
Does the memory consumption increase? [snip]
Nope....the machine is just as responsive as it was before. It is a very high end configuration... a HP bl20p blade server, 1 cpu - 1gb ram with oodles of other management paraphnelia. Hence as far as the config is concerned i am sure that it can sustain much higher workloads that sending apps for a toss so early in the test.
I observed that Zope becomes slower and slower as it takes more and more memory. I attribute this to effects of the cyclic garbage collector.
[snip] Just yest before i left for home i noticed sudden drop in hits/sec. The max i would achieve for 50 users was about 250 hits/s. But i guess i made some config changes which dropped this to <100hits/sec and about 10TPS overall. But this still isn't an excuse for the zope process to go for a toss at the 7-8th mintue of the test. We should atleast get Zope to be able to withstand to the current load if not scale lineraly to the increase in load over time. It 250+ hits/sec is not poss fine...atleast the smarto shouldn't go down. How can we force a GC on this process periodically. The GC for WEblogic works in a very simple manner cleaning up the "Stack Space" for the process once the upper limit is hit. Comments please????. Trevor
-- Dieter
===== ( >- -< ) /~\ __ Scaling FLOSS in the __ /~\ | \) / Enterprise : trevor.w@pvision.biz \ (/ | |_|_ \ Call Now: 9820349221 / _|_| \____________________________________/ __________________________________ Do you Yahoo!? Yahoo! Domains � Claim yours for only $14.70/year http://smallbusiness.promotions.yahoo.com/offer
Trevor, That's not how it works. If you think in terms of Apache processes == Zope threads, you are wrong :). In fact, the recommended number of threads depends highly on your application. If you have a (well-dimensioned and busy) SQL backend, using more that 4 (the default) worker threads *may* increase performance. If your app is CPU bound, *reducing* the number of worker threads may give you better results. At the end of the day it is multi-processing (not multi-threading) that makes Zope fly (*coughs*). Stefan On Mittwoch, Mai 19, 2004, at 07:06 Europe/Vienna, Trevor Warren wrote:
The number of threads (70-110) were kept proportional to the number of users(100). Performance degradation post 10mins of the test tleading to absymal response times. -- The time has come to start talking about whether the emperor is as well dressed as we are supposed to think he is. /Pete McBreen/
Hiyee Stefan, --- "Stefan H. Holek" <stefan@epy.co.at> wrote:
At the end of the day it is multi-processing (not multi-threading) that makes Zope fly (*coughs*). [snip]
Thanks stefan, i reduced the number of threads down to 10 for 20/50 user tests. The results are still grim...:(. Will keep you updated. Trevor
Stefan
===== ( >- -< ) /~\ __ Scaling FLOSS in the __ /~\ | \) / Enterprise : trevor.w@pvision.biz \ (/ | |_|_ \ Call Now: 9820349221 / _|_| \____________________________________/ __________________________________ Do you Yahoo!? SBC Yahoo! - Internet access at a great low price. http://promo.yahoo.com/sbc/
I was thinking of 2 worker threads. What do you need 10 threads for? Unless you have a well-saturated Oracle cluster driving your site, there is NO need for upping the thread count. Really. Stefan On Mittwoch, Mai 19, 2004, at 14:08 Europe/Vienna, Trevor Warren wrote:
Thanks stefan, i reduced the number of threads down to 10 for 20/50 user tests. The results are still grim...:(. Will keep you updated.
-- The time has come to start talking about whether the emperor is as well dressed as we are supposed to think he is. /Pete McBreen/
participants (8)
-
Chris McDonough -
Chris Withers -
Dieter Maurer -
Jake -
Matt Hamilton -
Paul Winkler -
Stefan H. Holek -
Trevor Warren