[Zope] How many hits can zope handle?

[Chris McDonough]

On Mon, 2003-07-07 at 23:58, Wayne Connolly wrote:
> Thanks for the quick response Chris. My first comment
> is-My gawd: our estimates produced similar number
> though in a more primative fashion.

Good to have a doublecheck, then.  I thought you might have slipped a
decimal point or two.  ;-)

> Our estimate is a 6 month projection based upon
> conservitive estimates, if it flops it flops but we
> have divided our company up in such a fasion that it
> wont impact our other products and our
> income(hopefully: but not likely straight away with
> the money thing :-(. Akamai have provided a quote per
> gig of traffic that will severly reduce our overheads
> to a reasonable level and of course the server
> farm(9600 CDN). We are just trying to see if we can
> 'plug' zope into it because we have had such good
> luck/fun with it in the past.

I'm glad to hear that.

> Unfortunately our server hit rate will be 100% with
> zero cache as each hit auotatically submits info to
> the database. No web pages are called. We will probly
> only recieve about 10 ~ 15 million hits a month for
> web page download(max).

That will make things much more expensive.  Unfortunately, I have no
way  estimate how much more expensive, because all the system
architecture designs that I've been a part of have been for typical web
traffic where there are many reads and few writes: not your target usage
pattern.  There are likely other folks here that have more experience in
OLAP/OLTP than I do, so maybe they can give you their insights.

> All sounds like doom and gloom but with slow uptake
> (6months) we should be able to include zope in the
> loop somewhere. It would be a shame not to...

Not sure where the doom and gloom comes in, sounds like fun to me, as
long as it pays for itself! ;-)

> 
> Wayne
> 
> 
> --- In zope@yahoogroups.com, Chris McDonough
> <chrism@z...> wrote:
> > Maybe. 
> > 
> > That's a lot of traffic.  Ten billion hits per day
> equates to 115,740
> > hits per second of continuous steady-state traffic
> (10,000,000,000 /
> > 86400).  A typical rule of thumb for system scaling
> is that 5 * nonpeak
> > hits/sec needs to be accounted for in the design, so
> you'd probably need
> > to design for about 580,000 hits/sec to meet peak
> load. 
> > 
> > If you can offload 90% of the traffic to a caching
> tier (very possible
> > if you don't need to do many writes), that means
> every 1 out of 10
> > requests or so will make it to a Zope appserver. 
> You can also assume
> > that a single cache server on Intel hardware (you'll
> need to convert
> > this to mainframe terms) can handle about 600
> requests per second. 
> > Thus, your application will require about 966 cache
> server instances
> > (580,000 / 600). 
> > 
> > If 10% of the traffic makes it to Zope, this means
> that the Zope tier
> > will need to service about 58,000 requests per
> second.  If you design
> > your application properly, you can expect a hit rate
> from a single Zope
> > instance on Intel hardware of about 50 - 90
> requests/sec.  This means
> > you're going to need at least 644 appservers (58,000
> *.1 / 90).  I don't
> > know how many ZEO servers this setup would need,
> we've never needed to
> > project ZEO server scale like that.  Probably about
> 50. 
> > 
> > FWIW, just to get an idea of the scale of something
> to handle this:  
> > 
> > - Your estimated bandwidth requirements for such a
> setup will be 
> >   about 1160 Mbits/s during peak load (if you assume
> each hit 
> >   consumes about 250 bytes to account for overhead,
> which is a very 
> >   low estimate). 
> > 
> > - If a T1 costs US $700 a month, you'll be spending
> ~$525,000.00 a 
> >   month (1160 / 1.544 * 700) retail just for the
> connectivity charges 
> >   to meet your peak demand. 
> > 
> > - The initial hardware investment (if you need it)
> will cost about 
> >    US$3.5 million+ (1660 servers * 2100/apiece). 
> > 
> > - If you figure that one competent sysadmin can
> administer about 60 
> >   machines, you'll likely need to hire around 26
> full time admins to 
> >   keep the system going. 
> > 
> > Given all that, are you sure about your hit
> projection? 
> > 
> > - C 
> > 
> > 
> > On Mon, 2003-07-07 at 21:40, Wayne Connolly wrote: 
> > > Can Zope handle 10 billion plus hits a day? while
> at
> > > the same time cramming info from these hits into a
> > > database(berkley db)?
> > > 
> > > Ive got zope running on a mainframe and we are
> slowly
> > > approaching a point where one of our sites will be
> > > achieving such hits(less that 0.1kb per hit) ...
> > > 
> > > Im just curioso....
> > > 
> > > Wayne
> > > 
> > > __________________________________
> > > Do you Yahoo!?
> > > SBC Yahoo! DSL - Now only $29.95 per month!
> > > http://sbc.yahoo.com
> > > 
> > > _______________________________________________
> > > Zope maillist  -  Zope@z...
> > > http://mail.zope.org/mailman/listinfo/zope
> > > **   No cross posts or HTML encoding!  **
> > > (Related lists - 
> > > 
> http://mail.zope.org/mailman/listinfo/zope-announce
> > >  http://mail.zope.org/mailman/listinfo/zope-dev )
> > 
> > 
> > _______________________________________________
> > Zope maillist  -  Zope@z...
> > http://mail.zope.org/mailman/listinfo/zope
> > **   No cross posts or HTML encoding!  **
> > (Related lists - 
> >  http://mail.zope.org/mailman/listinfo/zope-announce
> >  http://mail.zope.org/mailman/listinfo/zope-dev )
> 
> __________________________________
> Do you Yahoo!?
> SBC Yahoo! DSL - Now only $29.95 per month!
> http://sbc.yahoo.com
> 
> _______________________________________________
> Zope maillist  -  Zope@zope.org
> http://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists - 
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope-dev )