RE: [Zope-dev] How to make Zope fail nicely under high load?
Toby wrote: [...]
Zope's ZServer manages a queue of requests that have been recieved over http, but not dispatched to the publisher. This is handled in PubCore/ZRendezvous.py. I suspect this queue will be holding your backlog.
You might get some benefit from capping the length of that queue.
Thanks Toby. It doesn't seem like that queue (the requests list) has a cap at all. Have been stresstesting Zope for a few minutes and the queue is up to 300+ requests in the backlog queue. Not sure how to force a 503 return from within ZRendezvous.handle(), though; that's ZServer medusa magic. 300 requests at 1 request per 2 seconds yields a backlog of work of 10 minutes. This is before we've started optimizing. I wonder if Zope also processes requests that have been closed from the client end? I know Zope will continue processing a request even if it times out at least (so you can do long-running requests, like packing, through the web). Many (all?) of these 300 requests would have been closed by the time Zope got to process them, and so Zope shouldn't bother with them. Bye, -- Bjorn
On Wed, 11 Feb 2004 20:25:14 +0800 "Bjorn Stabell" <bjorn@exoweb.net> wrote:
Toby wrote: [...]
Zope's ZServer manages a queue of requests that have been recieved over http, but not dispatched to the publisher. This is handled in PubCore/ZRendezvous.py. I suspect this queue will be holding your backlog.
You might get some benefit from capping the length of that queue.
Thanks Toby.
It doesn't seem like that queue (the requests list) has a cap at all. Have been stresstesting Zope for a few minutes and the queue is up to 300+ requests in the backlog queue. Not sure how to force a 503 return from within ZRendezvous.handle(), though; that's ZServer medusa magic.
300 requests at 1 request per 2 seconds yields a backlog of work of 10 minutes. This is before we've started optimizing.
What kinds of requests are these? Do they all require a dynamic output? If not, then you should put better caching in front of Zope, or at a minimum tweak you caching headers so that some could be served as 304s for instance. If they are all dynamic, how dynamic? How different is the page for one session than another? If much of the page is the same then you might benefit from an ESI approach where you cache parts of pages and assemble them in the cache, serving only small pieces from Zope. I'm a bit surprised that you have not (from what I can see), used the most common Zope speedup (and often the cheapest overall): Buy more hardware and use ZEO.
I wonder if Zope also processes requests that have been closed from the client end? I know Zope will continue processing a request even if it times out at least (so you can do long-running requests, like packing, through the web). Many (all?) of these 300 requests would have been closed by the time Zope got to process them, and so Zope shouldn't bother with them.
Why would they be closed by the client? -Casey
On Thursday 12 February 2004 01:23, Casey Duncan wrote:
What kinds of requests are these? Do they all require a dynamic output? If not, then you should put better caching in front of Zope, or at a minimum tweak you caching headers so that some could be served as 304s for instance.
If they are all dynamic, how dynamic? How different is the page for one session than another? If much of the page is the same then you might benefit from an ESI approach where you cache parts of pages and assemble them in the cache, serving only small pieces from Zope.
I'm a bit surprised that you have not (from what I can see), used the most common Zope speedup (and often the cheapest overall): Buy more hardware and use ZEO.
I was going to suggest these things too. And a bunch of other stuff. A while ago, I wrote a page about making Zope go faster - it might help. http://zope.org/Members/richard/docs/zope_optimisation.html Richard
participants (3)
-
Bjorn Stabell -
Casey Duncan -
Richard Jones