[Zope] Zope High Availability

[Gareth Allen]

Hi guys,

Hope this is the right list, if not please correct me.
Currently I'm running a Zope cluster which consists of 3 servers, one 
with squid and the other 2, which have quad core processors and 4GB 
RAM,have 4 zeo instances running over DRBD and each Zope instance has 4 
clients (2 on each box). The problem  started when I ran Squid on the 
same box as the Zeo's (the Zeo's live on one box, but migrate when one 
of the servers fail) and for some reason Squid ended up sending requests 
on the box it was running on and the other box remained idle. My first 
move was to set weights to try and get Squid to see the servers as 
equal, but that didn't help either. Next I noticed the site would hang 
as Squid was sending requests to a client that was either unreachable 
due to high load and wasn't handling HTTP requests, but it seemed like 
ICP was still answering which didn't help the situation. I removed ICP 
and the problem still persisted, weights set to 1 on the box running 
Squid and the Zeo's and 100 on the other box. The next move was to put 
Squid on its own box, but I'm still experiencing problems where Squid 
sends requests to only one box and eventually one of the clients becomes 
unresponsive, but ICP still answers. My next step was to put Pound 
behind squid and let Squid forward all requests to Pound which in turn 
did the load balancing, but this morning I noticed 500 errors coming 
from Pound. 2 of the instances, one on each box, kept timing out. I 
restarted all 4 clients in question, reloaded Squid and restarted Pound, 
but Pound kept timing out when sending requests to 2 of the instances. 
The strange thing is that Pound was still able to forward requests to 4 
of the other clients serving another instance, but this one was under 
less load.

Any advice?

Thanks,
Gareth