Zope fine tuning HOW-TO / Zope Performance
Hi all: We are relatively new to Zope. I have found problems related to performance issues. Even though it is known that Apache alone is much faster than Zope, and when migrating (even static pages) to Zope a performance degradation may be expected, the differentials we are getting are very significant. Forget it when we add the dynamic parts. My question is: Has anyone seen a Zope fine tuning guideline. ? It would be interesting to have something like a cookbook on Tuning Zope and the overall web serving environment for specific cases. I am using RH Linux 6.1, Zope, PostgreSQL...all latest versions. As of hardware, at the minimum we have Pentium III, 128 MB RAM. As of now, very few hits but slow response though. Also, any names (URLs) of high traffic sites that are currently running Zope and serving dynamic content would be appreciated. Thanks Felipe
"Felipe E. Barousse" wrote:
Hi all:
We are relatively new to Zope. I have found problems related to performance issues. Even though it is known that Apache alone is much faster than Zope, and when migrating (even static pages) to Zope a performance degradation may be expected, the differentials we are getting are very significant. Forget it when we add the dynamic parts.
Are you using PCGI? It's very slow. You could try FastCGI, but the adaptor you need is a snapshot version. I recommend proxying or naked zope for best performance: http://www.zope.org/Members/anser/apache_zserver good luck, -- ethan mindlace fremen mindlace@imeme.net zope -&- imap email -&- mailing list weave your web with the web at http://imeme.net
In fact, I am NOT using PCGI nor FastCGI, yet. I am hitting direct against ZServer. Eventually, (for SSL reasons), I will install FastCGI with Apache mindlace escribió:
"Felipe E. Barousse" wrote:
Hi all:
We are relatively new to Zope. I have found problems related to performance issues. Even though it is known that Apache alone is much faster than Zope, and when migrating (even static pages) to Zope a performance degradation may be expected, the differentials we are getting are very significant. Forget it when we add the dynamic parts.
Are you using PCGI? It's very slow. You could try FastCGI, but the adaptor you need is a snapshot version. I recommend proxying or naked zope for best performance:
http://www.zope.org/Members/anser/apache_zserver
good luck, -- ethan mindlace fremen mindlace@imeme.net zope -&- imap email -&- mailing list weave your web with the web at http://imeme.net
"Felipe E. Barousse" wrote:
In fact, I am NOT using PCGI nor FastCGI, yet. I am hitting direct against ZServer. Eventually, (for SSL reasons), I will install FastCGI with Apache
Ok, so what is "slow"? What performance problems do you note? On a direct connection, most of my partially dynamic pages load "now". Is there some benchmark that's not giving the results you like? It's my undertstanding that ~30 requests per sec is what you get out of solid hardware on an essentially static (header & footer) static page. I use SSL with proxy, and that works pretty well for me. -- ethan mindlace fremen mindlace@imeme.net zope -&- imap email -&- mailing list weave your web with the web at http://imeme.net
This is definitely something I am interested in. I have used AB and done a lot of testing on my Zope installation (Pentium 300 MHz, 256 MB Ram, RH 6.1, PostgreSQL) and have found some interesting facts and figures. I have them all in one big word document that I am working on making into a Zope.org document. What would be nice is some other servers, configurations to see if some of my test results hold up. I ran about 20 tests but was mainly looking at the Database side of things. J
Hi all:
We are relatively new to Zope. I have found problems related to performance issues. Even though it is known that Apache alone is much faster than Zope, and when migrating (even static pages) to Zope a performance degradation may be expected, the differentials we are getting are very significant. Forget it when we add the dynamic parts.
My question is: Has anyone seen a Zope fine tuning guideline. ? It would be interesting to have something like a cookbook on Tuning Zope and the overall web serving environment for specific cases.
I am using RH Linux 6.1, Zope, PostgreSQL...all latest versions. As of hardware, at the minimum we have Pentium III, 128 MB RAM. As of now, very few hits but slow response though.
Also, any names (URLs) of high traffic sites that are currently running Zope and serving dynamic content would be appreciated.
Thanks
Felipe
"Felipe E. Barousse" wrote:
My question is: Has anyone seen a Zope fine tuning guideline. ? It would be interesting to have something like a cookbook on Tuning Zope and the overall web serving environment for specific cases.
Switching to FastCGI/whatever and tuning the webserver should be the last thing you do. To speed things up: 1. Do the kind of optimization you'd do in any programming environment. E.g. move repeated calculations out of loops, etc.. 2. Rewrite key sections of your DTML in Python should help. 3. Add caching of results. Sending prerendered data is 5-10 times faster than sending the DTML that generates the data. This is an especially good idea if you're dealing with structured text. 4. Add HTTP caching headers so your pages get called less. -- Itamar S.T. itamars@ibm.net
Itamar Shtull-Trauring wrote:
Switching to FastCGI/whatever and tuning the webserver should be the last thing you do.
I do not agree here. If you are looking to optomise your serving environment tuning Zope extends beyond just tuning zope. If you want to have a fast serving enviroment splitting the work between Zope and your basic webserver (we use Apache) is the way to go. Let Apache (or what ever webserver) do the repetative serving of images, and pages that don't change often. This way you free up Zope for doing what it does best, managing a dynamic serve environment, and rendering dynamic content. We actually use Apache to cache semi-dynamic sections of our site that may only change once a day as well as images. This way we off load as much work as possible from Zope; freeing it up to do the work of dynamic content and site management duties. Apache w/ mod_rewrite can also afford you the ablity to load balance between multiple zope backends, and give you a safety margin when one of the zope backends is down, or restarting. We actually monitor all our zope backends and reconfigure which backend servers are available every 15s so as not to overload a zope server that is starting to become sluggish - this has proved very helpful. -- ------------------------------- tonyr@ep.newtimes.com Director of Web Technology New Times, Inc. -------------------------------
If you want to have a fast serving enviroment splitting the work between Zope and your basic webserver (we use Apache) is the way to go. Let Apache (or what ever webserver) do the repetative serving of images, and pages that don't change often. This way you free up Zope for doing what it does best, managing a dynamic serve environment, and rendering dynamic content.
Are you sure that serving images off of Apache has any effect on performance? I did a lot of testing on that (below) and did see any major difference whether Zope served up the image or Apache did (out of a Zope document). There was a slight increase but not enough (I felt) to deal with the extra hassles of not having the images in Zope. A test document (test.html) just pointing to one image.. Server Software: Zope/Zope Server Port: 80 Document Path: /test.html Document Length: 137 bytes Concurrency Level: 25 Time taken for tests: 21.599 seconds Complete requests: 1000 Failed requests: 0 Total transferred: 340000 bytes HTML transferred: 137000 bytes Requests per second: 46.30 Transfer rate: 15.74 kb/s received Connnection Times (ms) min avg max Connect: 0 4 205 Processing: 183 528 465 Total: 183 532 670 Concurrency Level: 1000 Time taken for tests: 31.391 seconds Complete requests: 1000 Failed requests: 0 Total transferred: 340000 bytes HTML transferred: 137000 bytes Requests per second: 31.86 Transfer rate: 10.83 kb/s received Connnection Times (ms) min avg max Connect: 2 203 21006 Processing: 436 4141 4548 Total: 438 4344 25554 If I referenced the image on the Apache Server at port 8080 Server Software: Zope/Zope Server Port: 80 Document Path: /test.html Document Length: 85 bytes Concurrency Level: 25 Time taken for tests: 19.514 seconds Complete requests: 1000 Failed requests: 0 Total transferred: 287000 bytes HTML transferred: 85000 bytes Requests per second: 51.25 Transfer rate: 14.71 kb/s received Concurrency Level: 1000 Time taken for tests: 29.301 seconds Complete requests: 1000 Failed requests: 0 Total transferred: 287000 bytes HTML transferred: 85000 bytes Requests per second: 34.13 Transfer rate: 9.79 kb/s received BZ
"J. Atwood" wrote:
Are you sure that serving images off of Apache has any effect on performance? I did a lot of testing on that (below) and did see any major difference whether Zope served up the image or Apache did (out of a Zope document). There was a slight increase but not enough (I felt) to deal with the extra hassles of not having the images in Zope.
YES. I am sure that serving images off Apache has a big effect on performance. Not only does Apache serving blow the pants off zope serving it alows some of the workload to be offloaded to a system that scales better when hit w/ high concurrent requests. I would say a performance increase of 3x to 9x is worth the extra configuration work. Here are some numbers from my tests w/ ab: Test using ab requesting the same 25K image via three access methods: 1 - Apache Direct: using one file check in httpd.conf to check for existence of file and serving it from FS if it exists. 2 - Zope via FCGI: using mod_FastCGI from Apache to retrieve image from a Zope server on a separate server from Apache. 3 - Zope Direct: using port 8080 accessing Zope directly and requesting the same image. Image Size: 25,646 bytes rps = Request per second min = Connection Times (ms) Total: line min column min = Connection Times (ms) Total: line avg column min = Connection Times (ms) Total: line max column rps min avg max -n 10 -c 1 ============================================= Apache Direct 84.03 11 11 12 (3x over Zope Direct) Zope via FCGI 11.83 44 84 245 Zope Direct 22.32 28 44 139 And if these numbers aren't enough, the chasim just grows from there. FCGI does add some overhead but if you're caching to FS that overhead quickly becomes nominal when you figure the increased serving speed of going staight from file. -n 100 -c 10 (run 1) ============================================= Apache Direct 137.36 15 67 188 (9x over Zope Direct) Zope via FCGI 14.26 215 667 1953 Zope Direct 15.55 289 615 805 -n 100 -c 10 (run 2) ============================================= Apache Direct 142.65 15 65 320 Zope via FCGI 18.55 314 523 1558 Zope Direct 15.19 352 624 819 -n 100 -c 10 (run 3) ============================================= Apache Direct 117.10 17 77 282 Zope via FCGI 16.45 495 582 914 Zope Direct 17.51 178 556 909 -n 100 -c 25 (run 1) ============================================= Apache Direct 136.99 19 162 336 Zope via FCGI 11.89 1359 1937 3050 Zope Direct 15.40 238 1432 1759 -n 100 -c 25 (run 2) ============================================= Apache Direct 139.47 18 153 318 Zope via FCGI 15.67 632 1423 2103 Zope Direct 15.64 270 1402 1731 -n 100 -c 25 (run 3) ============================================= Apache Direct 124.69 19 173 414 Zope via FCGI 14.23 700 1550 2119 Zope Direct 15.53 379 1427 1755 RE: your test suite. What where you actually testing in the ab printouts? I'm confused how you could test image serving rates using ab against test.html. Wouldn't that just test the load times for the HTML and not the images? Don't get me wrong I'm not saying zope is not the greatest dynamic serving enviroment I've every used. It is. I'm just pointing out when it comes to serving a site that's expected to get some serious load, look to the tools at your disposal. Zope alone may not make the cut, but Zope/Apache is truely a great combination. The configuration pains are more than worth it. -- ------------------------------- tonyr@ep.newtimes.com Director of Web Technology New Times, Inc. -------------------------------
What about using the cache with ProxyPass and still serving static stuff from Zope? The problem I have with ProxyPass is mod_proxy's documentation which is terribly little. I couldn't find out whether it's possible to use caching with ProxyPass. I didn't get any speedup with cache enabled and expires set to 1h later. Anyone tried something similar? What about Squid in between (in http accelerator mode)? --Ragnar
"J. Atwood" wrote:
Are you sure that serving images off of Apache has any effect on performance? I did a lot of testing on that (below) and did see any major difference whether Zope served up the image or Apache did (out of a Zope document). There was a slight increase but not enough (I felt) to deal with the extra hassles of not having the images in Zope.
YES. I am sure that serving images off Apache has a big effect on performance. Not only does Apache serving blow the pants off zope serving it alows some of the workload to be offloaded to a system that scales better when hit w/ high concurrent requests. I would say a performance increase of 3x to 9x is worth the extra configuration work.
Here are some numbers from my tests w/ ab:
Test using ab requesting the same 25K image via three access methods:
1 - Apache Direct: using one file check in httpd.conf to check for existence of file and serving it from FS if it exists.
2 - Zope via FCGI: using mod_FastCGI from Apache to retrieve image from a Zope server on a separate server from Apache.
3 - Zope Direct: using port 8080 accessing Zope directly and requesting the same image.
Image Size: 25,646 bytes
rps = Request per second min = Connection Times (ms) Total: line min column min = Connection Times (ms) Total: line avg column min = Connection Times (ms) Total: line max column
rps min avg max
-n 10 -c 1 ============================================= Apache Direct 84.03 11 11 12 (3x over Zope Direct) Zope via FCGI 11.83 44 84 245 Zope Direct 22.32 28 44 139
And if these numbers aren't enough, the chasim just grows from there. FCGI does add some overhead but if you're caching to FS that overhead quickly becomes nominal when you figure the increased serving speed of going staight from file.
-n 100 -c 10 (run 1) ============================================= Apache Direct 137.36 15 67 188 (9x over Zope Direct) Zope via FCGI 14.26 215 667 1953 Zope Direct 15.55 289 615 805
-n 100 -c 10 (run 2) ============================================= Apache Direct 142.65 15 65 320 Zope via FCGI 18.55 314 523 1558 Zope Direct 15.19 352 624 819
-n 100 -c 10 (run 3) ============================================= Apache Direct 117.10 17 77 282 Zope via FCGI 16.45 495 582 914 Zope Direct 17.51 178 556 909
-n 100 -c 25 (run 1) ============================================= Apache Direct 136.99 19 162 336 Zope via FCGI 11.89 1359 1937 3050 Zope Direct 15.40 238 1432 1759
-n 100 -c 25 (run 2) ============================================= Apache Direct 139.47 18 153 318 Zope via FCGI 15.67 632 1423 2103 Zope Direct 15.64 270 1402 1731
-n 100 -c 25 (run 3) ============================================= Apache Direct 124.69 19 173 414 Zope via FCGI 14.23 700 1550 2119 Zope Direct 15.53 379 1427 1755
RE: your test suite.
What where you actually testing in the ab printouts? I'm confused how you could test image serving rates using ab against test.html. Wouldn't that just test the load times for the HTML and not the images?
Don't get me wrong I'm not saying zope is not the greatest dynamic serving enviroment I've every used. It is. I'm just pointing out when it comes to serving a site that's expected to get some serious load, look to the tools at your disposal. Zope alone may not make the cut, but Zope/Apache is truely a great combination. The configuration pains are more than worth it.
-- ------------------------------- tonyr@ep.newtimes.com Director of Web Technology New Times, Inc. -------------------------------
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
On Tue, 18 Apr 2000, Ragnar Beer wrote: :What about Squid in between (in http accelerator mode)? :--Ragnar i use squid in http acc. mode, works fine. i had to set "Expires:" headers on the objects i wanted to cache (theres been a suggestion on how to use "Last-Modified" headers on the list lately). when serving from cache, i got 400 requests/sec. at a concurrency level of 300 (tested with apache benchmark on a pII 400 / 256mb ram). this should do for most cases, i guess. peter. : :>"J. Atwood" wrote: :> > :> > Are you sure that serving images off of Apache has any effect on :> > performance? I did a lot of testing on that (below) and did see any major :> > difference whether Zope served up the image or Apache did (out of a Zope :> > document). There was a slight increase but not enough (I felt) to deal with :> > the extra hassles of not having the images in Zope. :> :>YES. I am sure that serving images off Apache has a big effect on :>performance. Not only does Apache serving blow the pants off zope :>serving it alows some of the workload to be offloaded to a system that :>scales better when hit w/ high concurrent requests. I would say a :>performance increase of 3x to 9x is worth the extra configuration work. :> :>Here are some numbers from my tests w/ ab: :> :>Test using ab requesting the same 25K image via three access methods: :> :>1 - Apache Direct: using one file check in httpd.conf to check for :>existence of file and serving it from FS if it exists. :> :>2 - Zope via FCGI: using mod_FastCGI from Apache to retrieve image from :>a Zope server on a separate server from Apache. :> :>3 - Zope Direct: using port 8080 accessing Zope directly and requesting :>the same image. :> :> :>Image Size: 25,646 bytes :> :>rps = Request per second :>min = Connection Times (ms) Total: line min column :>min = Connection Times (ms) Total: line avg column :>min = Connection Times (ms) Total: line max column :> :> :> rps min avg max :> :>-n 10 -c 1 :>============================================= :>Apache Direct 84.03 11 11 12 (3x over Zope Direct) :>Zope via FCGI 11.83 44 84 245 :>Zope Direct 22.32 28 44 139 :> :>And if these numbers aren't enough, the chasim just grows from there. :>FCGI does add some overhead but if you're caching to FS that overhead :>quickly becomes nominal when you figure the increased serving speed of :>going staight from file. :> :> :>-n 100 -c 10 (run 1) :>============================================= :>Apache Direct 137.36 15 67 188 (9x over Zope Direct) :>Zope via FCGI 14.26 215 667 1953 :>Zope Direct 15.55 289 615 805 :> :>-n 100 -c 10 (run 2) :>============================================= :>Apache Direct 142.65 15 65 320 :>Zope via FCGI 18.55 314 523 1558 :>Zope Direct 15.19 352 624 819 :> :>-n 100 -c 10 (run 3) :>============================================= :>Apache Direct 117.10 17 77 282 :>Zope via FCGI 16.45 495 582 914 :>Zope Direct 17.51 178 556 909 :> :> :> :>-n 100 -c 25 (run 1) :>============================================= :>Apache Direct 136.99 19 162 336 :>Zope via FCGI 11.89 1359 1937 3050 :>Zope Direct 15.40 238 1432 1759 :> :>-n 100 -c 25 (run 2) :>============================================= :>Apache Direct 139.47 18 153 318 :>Zope via FCGI 15.67 632 1423 2103 :>Zope Direct 15.64 270 1402 1731 :> :>-n 100 -c 25 (run 3) :>============================================= :>Apache Direct 124.69 19 173 414 :>Zope via FCGI 14.23 700 1550 2119 :>Zope Direct 15.53 379 1427 1755 :> :> :> :>RE: your test suite. :> :>What where you actually testing in the ab printouts? I'm confused how :>you could test image serving rates using ab against test.html. Wouldn't :>that just test the load times for the HTML and not the images? :> :> :>Don't get me wrong I'm not saying zope is not the greatest dynamic :>serving enviroment I've every used. It is. I'm just pointing out when :>it comes to serving a site that's expected to get some serious load, :>look to the tools at your disposal. Zope alone may not make the cut, :>but Zope/Apache is truely a great combination. The configuration pains :>are more than worth it. :> :>-- :>------------------------------- :>tonyr@ep.newtimes.com :>Director of Web Technology :>New Times, Inc. :>------------------------------- :> :>_______________________________________________ :>Zope maillist - Zope@zope.org :>http://lists.zope.org/mailman/listinfo/zope :>** No cross posts or HTML encoding! ** :>(Related lists - :> http://lists.zope.org/mailman/listinfo/zope-announce :> http://lists.zope.org/mailman/listinfo/zope-dev ) : : :_______________________________________________ :Zope maillist - Zope@zope.org :http://lists.zope.org/mailman/listinfo/zope :** No cross posts or HTML encoding! ** :(Related lists - : http://lists.zope.org/mailman/listinfo/zope-announce : http://lists.zope.org/mailman/listinfo/zope-dev ) : -- _________________________________________________ peter sabaini, mailto: sabaini@niil.at -------------------------------------------------
:What about Squid in between (in http accelerator mode)? :--Ragnar
i use squid in http acc. mode, works fine. i had to set "Expires:" headers on the objects i wanted to cache (theres been a suggestion on how to use "Last-Modified" headers on the list lately). when serving from cache, i got 400 requests/sec. at a concurrency level of 300 (tested with apache benchmark on a pII 400 / 256mb ram). this should do for most cases, i guess.
peter.
Cool! Do you use Squid as "frontend" to ZServer directly or do you have Apache in between? What about SSL? --Ragnar
On Tue, 18 Apr 2000 10:29:19 +0200, Ragnar Beer <rbeer@uni-goettingen.de> wrote:
What about Squid in between (in http accelerator mode)?
I'm using squid as the front end to Zope in our intranet. I'm more concerned with caching some very expensive external methods, rather than handling large numbers of hits. So far I'm more than happy with it. Toby Dickenson tdickenson@geminidataloggers.com
On Tue, 18 Apr 2000 10:29:19 +0200, Ragnar Beer <rbeer@uni-goettingen.de> wrote:
What about Squid in between (in http accelerator mode)?
I'm using squid as the front end to Zope in our intranet. I'm more concerned with caching some very expensive external methods, rather than handling large numbers of hits.
So far I'm more than happy with it.
Toby Dickenson tdickenson@geminidataloggers.com
I have some external methods that make a lot of SQL requests and even caching them for a couple of minutes only would be great progress. Do you know anything about caching and SSL or do you have any experience with that? --Ragnar
On Tue, 18 Apr 2000 17:18:39 +0200, Ragnar Beer <rbeer@uni-goettingen.de> wrote:
On Tue, 18 Apr 2000 10:29:19 +0200, Ragnar Beer <rbeer@uni-goettingen.de> wrote:
What about Squid in between (in http accelerator mode)?
So far I'm more than happy with it.
I have some external methods that make a lot of SQL requests and even caching them for a couple of minutes only would be great progress. Do you know anything about caching and SSL or do you have any experience with that?
Sorry, no. Toby Dickenson tdickenson@geminidataloggers.com
I've read about the new features of the Linux 2.4 kernel. One is a kernel http demon. For just serving static content (images etc.) this should be the fastest option possible. Of course that wouldn't replace Apache for SSL and other more demanding server tasks. Has anybody tried the kernel httpd already?
Ragnar Beer wrote:
What about using the cache with ProxyPass and still serving static stuff from Zope? The problem I have with ProxyPass is mod_proxy's documentation which is terribly little. I couldn't find out whether it's possible to use caching with ProxyPass. I didn't get any speedup with cache enabled and expires set to 1h later. Anyone tried something similar?
I have used the ProxyPass stuff and yes it is short on documentation, but this should help http://www.apache.org/docs/mod/mod_proxy.html. It did give us some better performance. In fact I originally tried it w/ the /manage pages for the images, but did not see much imporvement. It worked fine for general caching of images etc. My problem w/ ProxyPass caching is it messes w/ Apache cookie tracking that we are implementing, so I had to either patch ProxyPass or roll my own caching. I opted to roll my own caching so that we could control cache at a much finer grain. Now we can explicitly clear cache, explicitly force a page into cache and have specific sections of the site automatically fall under cache control. Some factors for ProxyPass are the parameters that indicate size of the cache and the location, make sure those are set within reason. I also seem to remember something w/ Last-Modified header having to be in place in conjunction w/ CacheLastModifiedFactor to make the object cachable by default. CacheForceCompletion also comes into play when caching images, the default of 90% will allow broken images to be placed in cache.
What about Squid in between (in http accelerator mode)?
Never really have had a chance to investigate this option. From what I gather it might work. We have log file issues, and pages served reports that might make this option not work for us. -- ------------------------------- tonyr@ep.newtimes.com Director of Web Technology New Times, Inc. -------------------------------
HI all Today, after a regular checkup of the web server, I found out that Zope was dead. I did a "ps aux" to find out that the Z2 processes were "running", actually several instances of it. Nevertheless, Zope was dead. No response at all, not even the manage pages. I do have a page served by Apache which, in turn, redirects the client to a page on the 8080 port (Zope, that is). The apache part worked fine, Zope was dead though. I checked on the Zope og (Z2.log) and all I found was that the last page served was last night, many hours ago with no clue whatsoever of what caused Zope to die. I stopped and started Zope and everything went as smooth as always. Has any of you experienced something similart ? By the way, this is a RH Linux 6.1 system, with latest version of Zope. Any comments are welcome. Thanks in advance. Felipe Barousse }
participants (9)
-
Felipe E. Barousse -
Itamar Shtull-Trauring -
J. Atwood -
joachim.werner@iuveno.de -
mindlace -
Peter Sabaini -
Ragnar Beer -
Toby Dickenson -
Tony Rossignol