Hi, We are new to Zope and are trying to create a form- and db-intensive web application using Zope 2.4.0 (though I'm testing in 2.5 also). I've noticed that our complex forms (which use nested dtml-in loops) are running extremely slow, taking more than 30 seconds on the server side (450MHz Pentium II w/ linux). So I created a snippet benchmark and noticed something peculiar, in that running a simple loop for the first time is often very slow, however in subsequent refreshes of the document the loop runs almost 8-10 times faster. I turned off caching as much as I know how (in the browser and zope). Here is the benchmark I'm using: <dtml-var standard_html_header> <dtml-comment>Note: My standard header turns off browser caching.</dtml-comment> <dtml-call "REQUEST.set('starttime',_.DateTime().millis())"> Start Time (ms): <dtml-var starttime><br> <dtml-call "REQUEST.set('testlist',['1','2','3','4','5','6','7','8','9','10'])"> <dtml-in "testlist"> <dtml-var sequence-item> </dtml-in> <br> <dtml-call "REQUEST.set('endtime',_.DateTime().millis())"> End Time (ms): <dtml-var endtime><br> <hr size=1> Delay (ms) = <dtml-var "endtime - starttime"><br> <dtml-var standard_html_footer> Any help would be extremely welcome...as is, the site is too slow to demo to potential clients. Thanks, Bill Ort billort@billort.com
What about using state-of-the-art hardware for demonstrations? -aj ----- Original Message ----- From: "Bill Ort" <billort2002@yahoo.com> To: <zope@zope.org> Sent: Friday, April 05, 2002 10:38 Subject: [Zope] performance concern
Hi,
We are new to Zope and are trying to create a form- and db-intensive web application using Zope 2.4.0 (though I'm testing in 2.5 also). I've noticed that our complex forms (which use nested dtml-in loops) are running extremely slow, taking more than 30 seconds on the server side (450MHz Pentium II w/ linux). So I created a snippet benchmark and noticed something peculiar, in that running a simple loop for the first time is often very slow, however in subsequent refreshes of the document the loop runs almost 8-10 times faster. I turned off caching as much as I know how (in the browser and zope). Here is the benchmark I'm using:
<dtml-var standard_html_header> <dtml-comment>Note: My standard header turns off browser caching.</dtml-comment> <dtml-call "REQUEST.set('starttime',_.DateTime().millis())"> Start Time (ms): <dtml-var starttime><br> <dtml-call "REQUEST.set('testlist',['1','2','3','4','5','6','7','8','9','10'])"> <dtml-in "testlist"> <dtml-var sequence-item> </dtml-in> <br> <dtml-call "REQUEST.set('endtime',_.DateTime().millis())"> End Time (ms): <dtml-var endtime><br> <hr size=1> Delay (ms) = <dtml-var "endtime - starttime"><br> <dtml-var standard_html_footer>
Any help would be extremely welcome...as is, the site is too slow to demo to potential clients. Thanks, Bill Ort billort@billort.com
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
hi, On Fri, 05 Apr 2002 10:38:50 -0500 "Bill Ort" <billort2002@yahoo.com> wrote:
We are new to Zope and are trying to create a form- and db-intensive web application using Zope 2.4.0 (though I'm testing in 2.5 also). I've noticed that our complex forms (which use nested dtml-in loops) are running extremely slow, taking more than 30 seconds on the server side (450MHz Pentium II w/ linux). ... Any help would be extremely welcome...as is, the site is too slow to demo to potential clients.
my advice: put a squid as accelerator in front of your zope and cache your queries... then your hardware will be fast enough to impress every potential client. greetings, maik. -- maik jablonski http://www.zfl.uni-bielefeld.de/ universitaet bielefeld http://www.sachunterricht-online.de/ zentrum fuer lehrerbildung 33501 bielefeld / germany tel: + 49 (0) 521 106 4234
in general, *nested* dtml-ins are always a bad idea. try to use python for complicated processing instead of using DTML. jens On Friday, April 5, 2002, at 12:49 , Maik Jablonski wrote:
hi,
On Fri, 05 Apr 2002 10:38:50 -0500 "Bill Ort" <billort2002@yahoo.com> wrote:
We are new to Zope and are trying to create a form- and db-intensive web application using Zope 2.4.0 (though I'm testing in 2.5 also). I've noticed that our complex forms (which use nested dtml-in loops) are running extremely slow, taking more than 30 seconds on the server side (450MHz Pentium II w/ linux). ... Any help would be extremely welcome...as is, the site is too slow to demo to potential clients.
my advice: put a squid as accelerator in front of your zope and cache your queries... then your hardware will be fast enough to impress every potential client.
greetings, maik.
-- maik jablonski http://www.zfl.uni-bielefeld.de/ universitaet bielefeld http://www.sachunterricht-online.de/ zentrum fuer lehrerbildung 33501 bielefeld / germany tel: + 49 (0) 521 106 4234
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Bill Ort wrote:
Hi,
We are new to Zope and are trying to create a form- and db-intensive web application using Zope 2.4.0 (though I'm testing in 2.5 also). I've noticed that our complex forms (which use nested dtml-in loops) are running extremely slow, taking more than 30 seconds on the server side (450MHz Pentium II w/ linux). So I created a snippet benchmark and noticed something peculiar, in that running a simple loop for the first time is often very slow, however in subsequent refreshes of the document the loop runs almost 8-10 times faster. I turned off caching as much as I know how (in the browser and zope). Here is the benchmark I'm using:
<dtml-var standard_html_header> <dtml-comment>Note: My standard header turns off browser caching.</dtml-comment> <dtml-call "REQUEST.set('starttime',_.DateTime().millis())"> Start Time (ms): <dtml-var starttime><br> <dtml-call "REQUEST.set('testlist',['1','2','3','4','5','6','7','8','9','10'])"> <dtml-in "testlist"> <dtml-var sequence-item> </dtml-in> <br> <dtml-call "REQUEST.set('endtime',_.DateTime().millis())"> End Time (ms): <dtml-var endtime><br> <hr size=1> Delay (ms) = <dtml-var "endtime - starttime"><br> <dtml-var standard_html_footer>
Any help would be extremely welcome...as is, the site is too slow to demo to potential clients.
Whaddayamean, slow? bleutgen@baal:~ > cat /proc/cpuinfo processor : 0 vendor_id : GenuineIntel cpu family : 6 model : 5 model name : Pentium II (Deschutes) stepping : 2 cpu MHz : 350.799293 cache size : 512 KB Start Time (ms): 1018026618031L 1 2 3 4 5 6 7 8 9 10 End Time (ms): 1018026618038L -------------------------------------------------------------------------------- Delay (ms) = 7L bleutgen@baal:~/work/bin > ab -n 100 http://localhost:30080/test/testrequest This is ApacheBench, Version 1.3c <$Revision: 1.38 $> apache-1.3 Copyright (c) 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Copyright (c) 1998-1999 The Apache Group, http://www.apache.org/ Server Software: Zope/Zope Server Hostname: localhost Server Port: 30080 Document Path: /test/testrequest Document Length: 270 bytes Concurrency Level: 1 Time taken for tests: 3.247 seconds Complete requests: 100 Failed requests: 0 Total transferred: 46900 bytes HTML transferred: 27000 bytes Requests per second: 30.80 Transfer rate: 14.44 kb/s received Connnection Times (ms) min avg max Connect: 0 0 1 Processing: 31 31 33 Total: 31 31 34 30 req. per second from localhost, 7ms over 100Mbit ethernet is too slow? Guess there's something else going on. cheers, oliver
On the topic of performace concerns in general I decided to code a python version of PHP's PEAR BenchMarkTimer class... does nothing complex at all and is just a wrapper for what you are already using in your code but puts it in an external method and gives it a more easy to type interface ... perhaps might be userfull... I simply found it very familiar and quick for marking my queries during development Usage: declare an external method to newBenchmarkTimer() with id BenchmarkTimer <dtml-let timer="BenchmarkTimer()"> <dtml-call "timer.start()"> <dtml-call expensiveCode> <dtml-call "timer.stop()"> <dtml-var "timer.timeElapsed()"> <dtml-call "timer.setMarker('my_start')"> <dtml-call expensiveCode> <dtml-call "timer.setMarker('my_stop')"> <dtml-var "timer.timeElapsed('my_start','my_stop')"> </dtml-let> can be used in python just the same ==== BEGIN benchmarktimer.py ==== from AccessControl import ClassSecurityInfo from Acquisition import Implicit import Globals import time def newBenchmarkTimer(): return BenchmarkTimer() class BenchmarkTimer(Implicit): def __init__(self): "constructor" self.markers = { 'Start' : 0, 'Stop' : 0 } security = ClassSecurityInfo() security.declareObjectPublic() security.declarePublic('start') def start(self): "set the marker 'Start'" self.setMarker('Start') security.declarePublic('stop') def stop(self): "set the marker 'Stop'" self.setMarker('Stop') security.declarePublic('setMarker') def setMarker(self,name): "set the specific marker" self.markers[name] = time.clock() security.declarePublic('timeElapsed') def timeElapsed(self, start='Start', end='Stop' ): "difference between two markers, order is important" return self.markers[end]-self.markers[start] security.declarePublic('getProfiling') def getProfiling(self): """ name -> name of marker time -> absolute time set in marker diff -> difference between this marker and last marker total -> difference between this marker and 'Start' marker """ i = 0 total = 0 profile = [] for ( marker, time ) in self.markers: if marker == 'Start': diff = 0 else: diff = time - temp; total = total + diff; profile[i]['name'] = marker profile[i]['time'] = time profile[i]['diff'] = diff profile[i]['total'] = total temp = time i = i+1 return result Globals.InitializeClass(BenchmarkTimer) ==== END benchamrktimer.py ====
<dtml-call "REQUEST.set('starttime',_.DateTime().millis())"> Start Time (ms): <dtml-var starttime><br> <dtml-call "REQUEST.set('testlist',['1','2','3','4','5','6','7','8','9','10'])"> <dtml-in "testlist"> <dtml-var sequence-item> </dtml-in> <br> <dtml-call "REQUEST.set('endtime',_.DateTime().millis())"> End Time (ms): <dtml-var endtime><br> <hr size=1> Delay (ms) = <dtml-var "endtime - starttime"><br> <dtml-var standard_html_footer>
Any help would be extremely welcome...as is, the site is too slow to demo to potential clients.
On Friday, April 05, 2002 5:38 PM, "Bill Ort" <billort2002@yahoo.com> wrote:
Hi,
We are new to Zope and are trying to create a form- and db-intensive web application using Zope 2.4.0 (though I'm testing in 2.5 also). I've noticed that our complex forms (which use nested dtml-in loops) are running extremely slow, taking more than 30 seconds on the server side (450MHz Pentium II w/ linux). Any help would be extremely welcome...as is, the site is too slow to demo to potential clients.
To which "Andreas Jung" <andreas@zope.com> answered:
What about using state-of-the-art hardware for demonstrations?
-aj
and "Maik Jablonski" <maik.jablonski@uni-bielefeld.de> answered:
my advice: put a squid as accelerator in front of your zope and cache your queries... then your hardware will be fast enough to impress every potential client.
greetings, maik.
Hi! I think you are missing the point here, gentlemen - the problem was not to impress potential clients or what kind of hardware to use, but to find a way to optimise the queries and the speed of DB-accesses. The advice that Andreas and Maik are giving is good - however, they do not offer a solution to the problem, but merely a workaround. Now, we don't know what kind of DB server Bill is using, it might be a crappy old Netware PC running and old version of Oracle thru the serial port for all we know. It could also be that his 450 MHz machine has 32 MB's of RAM and is running an heavily loaded news-server for the world, nor do we know if there is some misconfiguration of the server. Also he does not say whether the simple test he provided is the one that takes 30 seconds to complete (if it does, well, then we know the answer - it is the machine that is faulty :) This does not change the fact that there are some problems or issues with DB intensive Zope apps. Basically what Andreas and Maik are saying is that Zope sucks^H^H^H is sub-optimal when building DB intesive apps unless it is on the top-of-the line hardwara - which is kind of sad, because only last summer a 450 Mhz machine with Linux was pretty decent, and something that competing products would have no problem performing pretty well on. I run my app on a 1.8 GHz P4 Linux box, and it is not as fast as could be expected. We have *all* our content in Oracle running off a Sun 450, 4 CPU, 4 GB Ram. All Zope operations happen in memory on the Linux machine (which means no swapping) and still i have at times 98% CPU usage for something like 4-8 concurrent queries and the users are complaining - a bit. I am caching in Zope and using Apache with chachin on. I dunno how much faster a PC you can buy, but not that much faster that I am using now. I have a lot of optimisations left to do, so I can probably tweak it some more and get greater performance. Now, it just happens to be so that there is a thread about this very issue Bill is asking about: "[Zope] Z-SQL Method performance" which discusses what to do to improve speed in html-rendering (in short don't render html from dtml if you have very large result-sets), and here: "[Zope] Call SQLMethods/SQL Statements directly with python" which might be of interest too - wether it applies to Bills problem's I don't know, but they apply to mine, and I will be testing them on my app. Apologies for the rant - I will now step back into lurk-o-mode again. Cheers, /dario - -------------------------------------------------------------------- Dario Lopez-Kästen, dario@ita.chalmers.se IT Systems & Services System Developer/System Administrator Chalmers University of Tech.
On Fri, 05 Apr 2002 21:13:10 +0200 "Dario Lopez-Kästen" <dario@ita.chalmers.se> wrote:
I think you are missing the point here, gentlemen - the problem was not to impress potential clients or what kind of hardware to use, but to find a way to optimise the queries and the speed of DB-accesses.
This does not change the fact that there are some problems or issues with DB intensive Zope apps. Basically what Andreas and Maik are saying is that Zope sucks^H^H^H is sub-optimal when building DB intesive apps unless it is on the top-of-the line hardwara - which is kind of sad, because only last summer a 450 Mhz machine with Linux was pretty decent, and something that competing products would have no problem performing pretty well on.
hi, what I tried to say was: if you use a clever system of caching you can ran heavy load with moderate hardware. In my opinion clever caching is the key to build highly dynamic internet-applications which will give you although an impression of speed. Using the right combination of Accelerated Cache Managers, RAM-Caches and reverse Proxies will give you performance. Optimizing your DTML, your DatabaseQueries and so is necessary, but thats peanuts in comparison. Most people want ZOPE serving many users with many DB-queries without caching at all and complain about ZOPE's performance... that's unfair, I would guess...;-) greetings, maik.
On Fri, 2002-04-05 at 12:13, Dario Lopez-Kästen wrote:
This does not change the fact that there are some problems or issues with DB intensive Zope apps. Basically what Andreas and Maik are saying is that Zope sucks^H^H^H is sub-optimal when building DB intesive apps unless it is on the top-of-the line hardwara - which is kind of sad, because only last summer a 450 Mhz machine with Linux was pretty decent, and something that competing products would have no problem performing pretty well on.
I think last summers good machines were near-GHz ... but that's another issue. ;^)
I run my app on a 1.8 GHz P4 Linux box, and it is not as fast as could be expected. We have *all* our content in Oracle running off a Sun 450, 4 CPU, 4 GB Ram. All Zope operations happen in memory on the Linux machine (which means no swapping) and still i have at times 98% CPU usage for something like 4-8 concurrent queries and the users are complaining - a bit.
I am caching in Zope and using Apache with chachin on. I dunno how much faster a PC you can buy, but not that much faster that I am using now. I have a lot of optimisations left to do, so I can probably tweak it some more and get greater performance.
Lemme tell you about a little experiment I've been having fun with. I have two machines running ZEO Zope Instances, one is a dual 300 w/~364MB RAM, the other a dual Athlon XP1600 running 512MB. I have a third machine, a P133 w64MB RAM running Apache in Proxypass mode to serve up the sites. Now, that combo is pushing mid-80's/sec on an average request (and up to 109 on easier requests). These are to CMF served pages, btw. Each ZEO machine has 3 instances/domain on it. So 6 total ZEO-Zope Servers. I've added for fun and curiosity, mind you, a SQUID proxy to the dual athlon. I run ApacheBench (ab) through the SQUID proxy (which talks to the Apache box) and see over an order of magnitude improvement. Yes, I have watched it pull 1800 RPS, on those same queries. Now granted, this is all without SQL, so it may not apply to the original poster's concern, but I think they are impressive numbers, so I figured I'd post them. :^) Still, ZEO may be an option to speed up ZSQL queries, but I cannot say for sure, since I rarely use ZSQl anymore anyway. BTW, the request count was 10000, with a concurrency of 500. :^) -- Bill Anderson Linux in Boise Club http://www.libc.org Amateurs built the Ark, professionals built the Titanic. Amateurs build Linux, professionals build Windows(tm).
participants (8)
-
Andreas Jung -
Bill Anderson -
Bill Ort -
Dario Lopez-Kästen -
Jens Vagelpohl -
Maik Jablonski -
Oliver Bleutgen -
zopedan