Re: [Zope] How many hits can zope handle?
Maybe.
That's a lot of traffic. Ten billion hits per day equates to 115,740 hits per second of continuous steady-state traffic (10,000,000,000 / 86400). A typical rule of thumb for system scaling is that 5 * nonpeak hits/sec needs to be accounted for in the design, so you'd probably need to design for about 580,000 hits/sec to meet peak load.
If you can offload 90% of the traffic to a caching tier (very possible if you don't need to do many writes), that means every 1 out of 10 requests or so will make it to a Zope appserver. You can also assume that a single cache server on Intel hardware (you'll need to convert this to mainframe terms) can handle about 600 requests per second. Thus, your application will require about 966 cache server instances (580,000 / 600).
If 10% of the traffic makes it to Zope, this means that the Zope tier will need to service about 58,000 requests per second. If you design your application properly, you can expect a hit rate from a single Zope instance on Intel hardware of about 50 - 90 requests/sec. This means you're going to need at least 644 appservers (58,000 *.1 / 90). I don't know how many ZEO servers this setup would need, we've never needed to project ZEO server scale like that. Probably about
Thanks for the quick response Chris. My first comment is-My gawd: our estimates produced similar number though in a more primative fashion. Our estimate is a 6 month projection based upon conservitive estimates, if it flops it flops but we have divided our company up in such a fasion that it wont impact our other products and our income(hopefully: but not likely straight away with the money thing :-(. Akamai have provided a quote per gig of traffic that will severly reduce our overheads to a reasonable level and of course the server farm(9600 CDN). We are just trying to see if we can 'plug' zope into it because we have had such good luck/fun with it in the past. Unfortunately our server hit rate will be 100% with zero cache as each hit auotatically submits info to the database. No web pages are called. We will probly only recieve about 10 ~ 15 million hits a month for web page download(max). All sounds like doom and gloom but with slow uptake (6months) we should be able to include zope in the loop somewhere. It would be a shame not to... Wayne --- In zope@yahoogroups.com, Chris McDonough <chrism@z...> wrote: 50.
FWIW, just to get an idea of the scale of something
to handle this:
- Your estimated bandwidth requirements for such a
setup will be
about 1160 Mbits/s during peak load (if you assume each hit consumes about 250 bytes to account for overhead, which is a very low estimate).
- If a T1 costs US $700 a month, you'll be spending ~$525,000.00 a month (1160 / 1.544 * 700) retail just for the connectivity charges to meet your peak demand.
- The initial hardware investment (if you need it) will cost about US$3.5 million+ (1660 servers * 2100/apiece).
- If you figure that one competent sysadmin can administer about 60 machines, you'll likely need to hire around 26 full time admins to keep the system going.
Given all that, are you sure about your hit projection?
- C
On Mon, 2003-07-07 at 21:40, Wayne Connolly wrote:
Can Zope handle 10 billion plus hits a day? while at the same time cramming info from these hits into a database(berkley db)?
Ive got zope running on a mainframe and we are slowly approaching a point where one of our sites will be achieving such hits(less that 0.1kb per hit) ...
Im just curioso....
Wayne
__________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com
_______________________________________________ Zope maillist - Zope@z... http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
_______________________________________________ Zope maillist - Zope@z... http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
__________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com
On Mon, Jul 07, 2003 at 08:58:15PM -0700, Wayne Connolly wrote:
Unfortunately our server hit rate will be 100% with zero cache as each hit auotatically submits info to the database. No web pages are called. We will probly
Your database infrastructure wil be ... IMPRESSIVE.
All sounds like doom and gloom but with slow uptake
Are you kidding? 1B hits a day sounds like one of the ingredients for amazing success (assuming there's some sort of monetary side to it :) ) -roy
On Mon, 2003-07-07 at 23:58, Wayne Connolly wrote:
Thanks for the quick response Chris. My first comment is-My gawd: our estimates produced similar number though in a more primative fashion.
Good to have a doublecheck, then. I thought you might have slipped a decimal point or two. ;-)
Our estimate is a 6 month projection based upon conservitive estimates, if it flops it flops but we have divided our company up in such a fasion that it wont impact our other products and our income(hopefully: but not likely straight away with the money thing :-(. Akamai have provided a quote per gig of traffic that will severly reduce our overheads to a reasonable level and of course the server farm(9600 CDN). We are just trying to see if we can 'plug' zope into it because we have had such good luck/fun with it in the past.
I'm glad to hear that.
Unfortunately our server hit rate will be 100% with zero cache as each hit auotatically submits info to the database. No web pages are called. We will probly only recieve about 10 ~ 15 million hits a month for web page download(max).
That will make things much more expensive. Unfortunately, I have no way estimate how much more expensive, because all the system architecture designs that I've been a part of have been for typical web traffic where there are many reads and few writes: not your target usage pattern. There are likely other folks here that have more experience in OLAP/OLTP than I do, so maybe they can give you their insights.
All sounds like doom and gloom but with slow uptake (6months) we should be able to include zope in the loop somewhere. It would be a shame not to...
Not sure where the doom and gloom comes in, sounds like fun to me, as long as it pays for itself! ;-)
Wayne
Maybe.
That's a lot of traffic. Ten billion hits per day equates to 115,740 hits per second of continuous steady-state traffic (10,000,000,000 / 86400). A typical rule of thumb for system scaling is that 5 * nonpeak hits/sec needs to be accounted for in the design, so you'd probably need to design for about 580,000 hits/sec to meet peak load.
If you can offload 90% of the traffic to a caching tier (very possible if you don't need to do many writes), that means every 1 out of 10 requests or so will make it to a Zope appserver. You can also assume that a single cache server on Intel hardware (you'll need to convert this to mainframe terms) can handle about 600 requests per second. Thus, your application will require about 966 cache server instances (580,000 / 600).
If 10% of the traffic makes it to Zope, this means that the Zope tier will need to service about 58,000 requests per second. If you design your application properly, you can expect a hit rate from a single Zope instance on Intel hardware of about 50 - 90 requests/sec. This means you're going to need at least 644 appservers (58,000 *.1 / 90). I don't know how many ZEO servers this setup would need, we've never needed to project ZEO server scale like that. Probably about
--- In zope@yahoogroups.com, Chris McDonough <chrism@z...> wrote: 50.
FWIW, just to get an idea of the scale of something
to handle this:
- Your estimated bandwidth requirements for such a
setup will be
about 1160 Mbits/s during peak load (if you assume each hit consumes about 250 bytes to account for overhead, which is a very low estimate).
- If a T1 costs US $700 a month, you'll be spending ~$525,000.00 a month (1160 / 1.544 * 700) retail just for the connectivity charges to meet your peak demand.
- The initial hardware investment (if you need it) will cost about US$3.5 million+ (1660 servers * 2100/apiece).
- If you figure that one competent sysadmin can administer about 60 machines, you'll likely need to hire around 26 full time admins to keep the system going.
Given all that, are you sure about your hit projection?
- C
On Mon, 2003-07-07 at 21:40, Wayne Connolly wrote:
Can Zope handle 10 billion plus hits a day? while at the same time cramming info from these hits into a database(berkley db)?
Ive got zope running on a mainframe and we are slowly approaching a point where one of our sites will be achieving such hits(less that 0.1kb per hit) ...
Im just curioso....
Wayne
__________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com
_______________________________________________ Zope maillist - Zope@z... http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists -
http://mail.zope.org/mailman/listinfo/zope-announce
_______________________________________________ Zope maillist - Zope@z... http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
__________________________________ Do you Yahoo!? SBC Yahoo! DSL - Now only $29.95 per month! http://sbc.yahoo.com
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Hi Wayne, Wayne Connolly wrote:
Thanks for the quick response Chris. My first comment is-My gawd: our estimates produced similar number though in a more primative fashion.
Our estimate is a 6 month projection based upon conservitive estimates, if it flops it flops but we have divided our company up in such a fasion that it wont impact our other products and our income(hopefully: but not likely straight away with the money thing :-(. Akamai have provided a quote per gig of traffic that will severly reduce our overheads to a reasonable level and of course the server farm(9600 CDN). We are just trying to see if we can 'plug' zope into it because we have had such good luck/fun with it in the past. Unfortunately our server hit rate will be 100% with zero cache as each hit auotatically submits info to the database. No web pages are called. We will probly only recieve about 10 ~ 15 million hits a month for web page download(max).
All sounds like doom and gloom but with slow uptake (6months) we should be able to include zope in the loop somewhere. It would be a shame not to...
looks you're doing kind of loggin? This could be more performant with some scripts running in the say frontend apache - eating the log output from apache and stuff it into a pipe directed to the database. But keep in mind: logging - so far you dont have a decent tracing of users due to cookies (which also _have_ to be acceppted all the time) is usually pointless due to cache/proxy issues. Regards Tino
participants (4)
-
Chris McDonough -
Roy S. Rapoport -
Tino Wildenhain -
Wayne Connolly