[Zope-dev] How to make Zope fail nicely under high load?

[Peter Sabaini]

This very much depends on your application and requirements (and your 
definition of "nicely" :-)), but I'd argue that it rarely make sense to 
handle this at the TCP Connection level (just to think about browsers 
opening multiple connections, HTTP/1.0 or /1.1 compliant browsers, 
proxies etc.)

As an example, one client of ours had the following requirements wrt. to 
this problem which I think should be fairly common:

* Allow X logged-in users till a certain responsiveness threshold is reached

* If said threshold is reached:
    - inform all users trying to log that the site is too loaded
    - while allowing already-logged-in users to still use the site with 
acceptable speed

This means we had to a) measure responsiveness (which we did with some 
cobbled together heuristic involving the rate of user logins (something 
like 5 users per minute) and an external script effectively "HTTP 
Pinging" the site every X minutes) and b) redirecting users to a 
statically hosted page (which can be served cheaply) if said conditions 
were met.

This kind of thing clearly cannot be done at the TCP level because TCP 
Connection != User session

cheers,
peter.


Bjorn Stabell wrote:
> Hi,
> 
> We've run into an interesting problem when load-testing a Zope site
> behind Apache, a problem that we've also encountered in real life.
> 
> Basically, when the load gets high, Zope has a huge backload of work
> (several minutes of requests), making the average latency for each
> request many minutes.  What are effective ways to do this kind of
> overload management so that the backlog of work doesn't get that big?
> 
> The ideal would be for requests to fail immediately if the backlog of
> work is more than a certain number of requests (or even better,
> estimated time to process).
> 
> 
> Here's what we've tried:
> 
> Naively, we thought we could just set the socket.listen() backlog in
> Apache and Zope to a lower value, but in TCP connect()'s apparently
> don't fail if the server's listen backlog is full; instead requests are
> retried, resulting in a client side managed "listen backlog", also
> giving the same long latency.  (If someone knows this stuff, please
> confirm/deny these allegations against TCP :)
> 
> It appears the way to control it would for Apache or Zope to return "503
> Service Unavailable" when the load is too high, but we haven't found a
> good way to do this; Zope doesn't appear to have any mechanism for it,
> and Apache's ProxyPass doesn't either.  I guess load balancers would,
> but that's a bit overkill since we run the server on one machine.
> 
> 
> Regards,
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3461 bytes
Desc: S/MIME Cryptographic Signature
Url : http://mail.zope.org/pipermail/zope-dev/attachments/20040211/28371773/smime.bin