In a load-distributed system (i.e. round robin, or anything that is load-balancing without detection of an out-of-service web node), you need to manage availability yourself, and this sucks for clusters of web nodes larger than 2, because you have to set them up in a ring, using IP takeover. The problem with this, say in a 3 node setup, is that if n1 fails, and n2 and n3 are still alive, one or the other of n2 or n3 will need to take on n1's load - they can't split it. In a true load-balanced arrangement, the forward LB/Proxy will detect that a box is out, and remove it from the pool. This is ideal for a lot of nodes. It sounds to me like Squid+Zope+ICP sounds like the best way to do this, of course, with the use of monitoring software like mon as well. In the case of a 2-node cluster, IP takeover is easier and cheaper; but this doesn't scale well for HTTP traffic, given the alternative of load-balancing is better. The new setup I plan on using for future projects is a pair of load-balanced proxy servers balanced with a L4 switch with direct packet return, a feature of my L4 switch which has a speed advantage over a proxied approach; the downside to this is that I have to manage availability of my proxy server boxes themselves; I do this with IP-takeover based software like heartbeat, which fills this niche very well. I also use IP takeover for my relational database, file serving, and ZEO storage server setups. The only tier of this setup that will not be managed by IP takeover clustering will be the web servers themselves, which will be managed by the 2 proxy servers acting in a capacity of a load-balancer (Squid+Zope+ICP patches). | ^ v | >[Router]< / | \ / v \ / [L4Switch ] \ Hardware Load-Balancer / | | \ DMZ | v v | ====[Cache1]:::::[Cache2] -->Squid Proxies Private | | For Caching, LoadBalancing, Netwk | | security, and availability | | of web servers _|________|_ -Clustered w/Heartbeat / | | \ [n1] [n2] [n3] [n*] -->Web Servers - Availability | | | | managed via Squid/ICP +----+----+----+ | -->Storage Svrs.(ZSS/RDB/File) | Primary and Hot Backup | Clustering w/Heartbeat [Storage1]:::::::::[Storage2] I hope this makes sense and/or is helpful in some way... I think it is a good generic blueprint for a relatively scalable setup for a large site. Sean -----Original Message----- From: Derek Simkowiak [mailto:dereks@realloc.net] Sent: Thursday, January 24, 2002 2:00 PM To: sean.upton@uniontrib.com Cc: zope@zope.org Subject: RE: [Zope] Static fail-over -> I guess what I'm saying is there are places where IP-takeover based -> clustering is appropriate, sometimes even in conjunction with forward -> traffic direction. Sean: Thanks for the I.P. Takeover info! While we're on the topic, I have a quick 'opinion' question about clusters. This applies directly to a Zope cluster I'll be building soon. A fully redundant, yet not load-balanced, H.A. system requires *almost* all the same hardware and software as a 2-node load-balanced system. That is, you need to detect a service failure and, if found, make sure traffic goes to the backup system, not the primary system. (You'd also want to send an alert, etc., and if you only have dual redundancy, you'll also want to monitor the backup to make sure it'll be there when the failover is needed.) In a load balanced system (with homogenous nodes), you need to watch all nodes for failure and, if found, fail out that particular node. But instead of the backup hardware going "wasted", just waiting for a failover, you've halved the hardware workload by distributing the work to both machines. This may result in a faster response to endusers. My question: Does it ever make sense to set up a redundant system without load balancing? After all, plopping in new nodes on an as-needed basis is a very handy feature. The only thing I can think of is this: Imagine a site that must serve 1 zillion requests per day (a zillion being a Very Big Number). If you use a simple failover system, then you buy two boxes, both capable of handling 1 zillion requests. If the site grows more popular so you must handle 2 zillion requests/day, you just upgrade both servers to handle 2 zillion requests/day. (This is a thought experiment, ignore the fact that you should have planned for the growth in the first place :) Now imagine those same two (1 zillion/day capable) boxes have been configured for load balancing. Immediately, each server is only serving .5 zillion requests/day. As the site grows to it's new 2 zillion/day load, both servers being in use means no hardware upgrade is needed. BUT --and this is a big but-- you no longer have an H.A. system. You've lost your redundancy. If one of the servers go down, about half of your customers will get an HTTP 502 "Overloaded" error message. So to keep full redundancy, you actually need THREE nodes. In fact, for however many nodes you want your cluster to be, you need to add one extra "redundant" node that would handle the traffic for any failed node (just until that failed node is repaired). So I guess I answered my own question: in a two-node load balanced system, the second node would really be nothing more than a backup node (even though it's handling traffic), and thus you'd need to upgrade your hardware (or rather, just add more nodes) as soon as your traffic exceeded the limit of (nodecount - 1) * traffic_per_node And the "traffic_per_node" you'd have to assume would be peak usage traffic, i.e. ( total_peak_traffic / (nodecount - 1) ) In the real world of public websites, however, I think a load-balanced system may actually offer extra redundancy. Because no site will get its peak load on a 24/7 basis, the load balanced system can use any extra resources (which are free because the cluster is not at 100% capacity) to fill in for the failed node -- and this would be *in addition to* your extra failover node. In a simple failover-system, you don't get this. Any additional comments would be greatly appreciated. --Derek