- Zope-Dev - Zope lists

RE: [Zope-dev] How to make Zope fail nicely under high load?
by Bjorn Stabell 12 Feb '04

12 Feb '04

> Bjorn wrote: > > I wonder if Zope also processes requests that have been closed from > > the client end? I know Zope will continue processing a request even > > if it times out at least (so you can do long-running requests, like > > packing, through the web). Many (all?) of these 300 requests would > > have been closed by the time Zope got to process them, and so Zope > > shouldn't bother with them. Casey wrote: > Why would they be closed by the client? Two scenarios: 1) The user gets tired of waiting for the page to come up after a few seconds and stopes or reloads it. This is very common if the response time is above a few seconds. 2) After clicking on a link the user discovers another, better link, and quickly clicks on that one as well. This is quite common for me, at least :) When Zope gets very loaded, response times suffer and these things happen more often; in our case with 300 requests in the pending queue, nobody would bother waiting 10 minutes for their page to come up (they'd stop the page loading and go somewhere else), so all the requests (most likely) would be needless; Zope is in fact just processing dead requests. Bye, -- Bjorn

2 1

Zope, forking and cpu-affinity
by Dario Lopez-Kästen 12 Feb '04

12 Feb '04

Hello, I hope this is not the wrong list... I have some trouble doing cpu-afiinity wiht zope. I can affinitise the parent process just fine, however, zope forks and the children get spread out all over the CPUs. I do not yet wish to resort to shell-hacking the ouput of ps -ef, so I was wodering if there are some documents that explain the forking nature of zope. Any help is appreciated. Sincerely, /dario -- -- ------------------------------------------------------------------- Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech.

3 4

RE: [Zope-dev] How to make Zope fail nicely under high load?
by Bjorn Stabell 12 Feb '04

12 Feb '04

Richard Jones wrote: > On Thursday 12 February 2004 01:23, Casey Duncan wrote: > > What kinds of requests are these? Do they all require a dynamic > > output? If not, then you should put better caching in front of Zope, > > or at a minimum tweak you caching headers so that some > > could be served as 304s for instance. > > > > If they are all dynamic, how dynamic? How different is the page for > > one session than another? If much of the page is the same then you > > might benefit from an ESI approach where you cache parts of > > pages and assemble them in the cache, serving only small pieces from Zope. > > > > I'm a bit surprised that you have not (from what I can > > see), used the most common Zope speedup (and often the cheapest overall): Buy more > > hardware and use ZEO. > > I was going to suggest these things too. And a bunch of other > stuff. A while ago, I wrote a page about making Zope go faster - it might help. > > http://zope.org/Members/richard/docs/zope_optimisation.html Okay, I'd better explain our situation in more detail, then. Richard, Casey, thanks. Except for using ZEO, we're already doing everything Richard and you mentions. Zope only serves pages (no images etc) that are dynamic, the object cache is set at 20,000 objects with only 2 threads (otherwise we got too much deactivation of objects), and we also use the RAM Cache Manager for static bits of dynamic pages. We could probably have used it more, but since it can't cache persistent objects, it's a bit of a pain sometimes. The point is, our template is very dynamic, and our application (a BBS) is very dynamic as well. We have done one round of optimization (basically making things as cacheable as possible) and after testing the scalability now we're going to do another round of optimization, tweaking the algorithm and removing as many ZCatalog searches as possible. One of the optimization we're thinking of is storing results of ZCatalog searches (e.g., number of replies to postings) in volatile variables so we don't have to run the catalog search at all. We'd like to use memory space shared between threads for this. Using ZEO would require us to store this in the ZODB. I'm sure we can make the application run faster, but we'd like to support 500-1000 simultaneous users whereas now we can only support 20-25! I don't know if we can scale it enough just by tweaking the application. Another option is to scrap Zope for the BBS part and use one of the fully featured, and optimized, BBS applications for PHP. That would definitely be a safer choice, but would require extra application integration time (user database, searching). Regards, -- Bjorn

3 2

RE: [Zope-dev] How to make Zope fail nicely under high load?
by Bjorn Stabell 11 Feb '04

11 Feb '04

Toby wrote: [...] > Zope's ZServer manages a queue of requests that have been > recieved over http, > but not dispatched to the publisher. This is handled in > PubCore/ZRendezvous.py. I suspect this queue will be holding > your backlog. > > You might get some benefit from capping the length of that queue. Thanks Toby. It doesn't seem like that queue (the requests list) has a cap at all. Have been stresstesting Zope for a few minutes and the queue is up to 300+ requests in the backlog queue. Not sure how to force a 503 return from within ZRendezvous.handle(), though; that's ZServer medusa magic. 300 requests at 1 request per 2 seconds yields a backlog of work of 10 minutes. This is before we've started optimizing. I wonder if Zope also processes requests that have been closed from the client end? I know Zope will continue processing a request even if it times out at least (so you can do long-running requests, like packing, through the web). Many (all?) of these 300 requests would have been closed by the time Zope got to process them, and so Zope shouldn't bother with them. Bye, -- Bjorn

3 2

RE: [Zope-dev] How to make Zope fail nicely under high load?
by Bjorn Stabell 11 Feb '04

11 Feb '04

> Bjorn Stabell wrote: > > Basically, when the load gets high, Zope has a huge > backload of work > > (several minutes of requests), making the average latency for each > > request many minutes. What are effective ways to do this kind of > > overload management so that the backlog of work doesn't get that big? > > Make the app faster so that doesn't happen. You'll always reach the bottleneck sooner or later, and in 99% of the cases it'll be Zope. It's not exactly a racing horse. [...] > ...return 503 to every new request probably isn't a good > idea, 503 to every new session is probably OK. Maybe. It > depends on the workload. If you have a user who sends 10 > requests and 5 of them fail, they're going to keep hitting > reload to try to get them all to work, which just compounds > your problems. The idea is that you want tweak it so some of > your users get everything, and some get nothing, instead of > everybody getting partials. This means you have to track > sessions though... thats probably easier to do in ZServer > than it is in Apache 1.3 just because of the process model, > but that doesn't mean its easy. > > By default, Apache doesn't track sessions and do that kind of > planned failure. There are probably modules that can enable > that behavior. > > ZServer's multiplexed IO model just hides the accepting > socket from the poller if its concurrency level is reached, > and yeah, then the incoming connections end up in the > backlog. You'd have to change ZServer so it handled those > new requests instead, identified if they were part of a > session, queued them up if they were, or spat out a 503 if > they weren't. With KeepAlive on, wouldn't everything in one "session" sent over one TCP connection? If that is the case, each accept() will basically service a whole session, and so you should be able to do the easy thing of refusing a connection just after an accept(). > Frankly, I bet you'd have more fun just making your app faster. We'll do that as well, but this problem is pretty serious; we can't overload our server for even just a little while before latencies start building up to ridiculous levels and users start hitting reload, further compounding the problem. In the end we have to restart the server, same as Dario reported. Bye, -- Bjorn

4 4

How to make Zope fail nicely under high load?
by Bjorn Stabell 11 Feb '04

11 Feb '04

Hi, We've run into an interesting problem when load-testing a Zope site behind Apache, a problem that we've also encountered in real life. Basically, when the load gets high, Zope has a huge backload of work (several minutes of requests), making the average latency for each request many minutes. What are effective ways to do this kind of overload management so that the backlog of work doesn't get that big? The ideal would be for requests to fail immediately if the backlog of work is more than a certain number of requests (or even better, estimated time to process). Here's what we've tried: Naively, we thought we could just set the socket.listen() backlog in Apache and Zope to a lower value, but in TCP connect()'s apparently don't fail if the server's listen backlog is full; instead requests are retried, resulting in a client side managed "listen backlog", also giving the same long latency. (If someone knows this stuff, please confirm/deny these allegations against TCP :) It appears the way to control it would for Apache or Zope to return "503 Service Unavailable" when the load is too high, but we haven't found a good way to do this; Zope doesn't appear to have any mechanism for it, and Apache's ProxyPass doesn't either. I guess load balancers would, but that's a bit overkill since we run the server on one machine. Regards, -- Bjorn Stabell bjorn(a)exoweb.net Tel +86 (10) 65918490

6 6

RE: [Zope-dev] How to make Zope fail nicely under high load?
by Bjorn Stabell 11 Feb '04

11 Feb '04

Toby wrote: > Hmm, yes. ZRendezvous doesnt have any choice but to queue up > the request. > Handling the request at that point could cause problems > because the support > for pipelined http requests might cause another recursive call to > ZRendezvous.handle, and deadlock. You need to handle it earlier. > > Maybe tweak the readable() method of zhttp_server and > zhttp_channel in > HTTPServer.py, to stop bytes coming in over the socket if the > queue is too large. > > ? Hm. And making this limit (and the listen backlog parameter) available as configuration options. :) Not letting bytes through is not enough, though. I think a 503 error should be returned as quickly as possible back to the client. > > I wonder if Zope also processes requests that have been closed from > > the client end? > > It will > > > I know Zope will continue processing a request even if it > > times out > > at least (so you can do long-running requests, like > > packing, through > > the web). Many (all?) of these 300 requests would have > > been closed by > > the time Zope got to process them, and so Zope shouldn't > > bother with them. > > That doesnt need fixing. It wouldnt be a problem if the queue > was kept small. Well, I think it would help in many cases. Many people click on links before the server has finished (or before it has even visibly started) serving up pages. Searching the web today I saw a commercial product (I think it was a load balancer) promoting this as an important feature. Any page for which the client has closed the connection before the server has started serving up the page could be skipped. If the server has already started serving up the page, it could complete the transaction. Bye, -- Bjorn

1 0

Re: [Zope-dev] Adapters in Zope 2
by santi＠zetadb.com 10 Feb '04

10 Feb '04

>> I've been trying to add some extra functionallity to a Folder object >> without inheriting, but I'm not able to make it work. > > First question is "Why?". Unless you want to change the fucntionality of > *all* folder objects inheriting is the way to go. > For instance, If I've a repository of Folders and Files and I write two applications using it. In the first application I need an extra method to obtain contained objects in some especific order, and in the second application I need a different order. I can write a new class inheriting from Folder with two diferent new methods, but I think that this is not a good aproach. If tomorrow I write a new application with some other needings I should modify my InheritedFolderClass. I prefer to add this logic in each application, using and adapter. If all we create inherited versions of Folders, collaborations between applications is more difficult. I you have an application using a repository of inheritedFolderA's and I have another one using inheritedFolderB's, our applications can't use the same repository, what would be possible using a repository of Folders and adapters in each application. >> Anybody knows if there is a way to use adapters in zope2 ? Or, if >> not, there is some other way to add functionallity on-the-fly, without >> inheriting ? > > MonkeyPatching. Python is highly dynamic. You rewrite the methods to how > you would like them, and then you replace them with a quick. > > TheClass.OldMethod = NewMethod. > > And now you call the new method with > object = TheClass() > object.OldMethod() > Very cool. Yes, this is a possible way, but not so nice as Adapters (in my modest opinion). Thanks for your answer Santi Camps http://zetadb.sourceforge.net

1 0

data migration from zope 2.6.1 to zope 2.7.0-b2
by altkey 07 Feb '04

07 Feb '04

Hi, I am fairly new to zope & python but I am working to migrate some data/objects from zope 2.6.1 to zope 2.7.0-b2 (and eventually 2.7.0 final. What I am doing is this. (1) get Dublin core metadata + some Dublin core extensions for the objects (2) get pure text representation of object content - I can't use a simple import/export in zexp format because the classes in the receiving zope instance differ significantly from the classes in the sending zope instance. I am able to collect all the data mentioned above and pass it across to the new 2.7.0-b2 zope instance but i am having trouble getting my objects to live where they need to live; specifically their path is not being accepted properly when i recreate the objects. here's a code snippet for an external method i am working on <preliminary stuff removed ...> def addObjects(self, objectList): from Products.CMFCore.PortalFolder import PortalFolder from OFS.Moniker import Moniker import urllib nameList = [] for myObject in objectList: metadata = myObject[0] content = myObject[1] dcMetadata = metadata['DC'] edcMetadata = metadata['EXTENDED_DC'] name = metadata['OBJECT_ID'] oType = metadata['TYPE'] portalType = metadata['PORTAL_TYPE'] metaType = metadata['META_TYPE'] DCType = dcMetadata['Type'] path = metadata['PATH'] root = metadata['ROOT'] selfPath = self.getPhysicalPath() nameList.append((selfPath,name,portalType,metaType,path)) self.invokeFactory(portalType,'__'+name) q = content['CONTENTS'] uq = urllib.unquote(q) <other code to manipulate the object's content removed ...> <clean up stuff removed ...> the line "self.invokeFactory(portalType,'__'+name)" should be creating the new objects should be created in the context (self) that the external method is executing in - i checked what path 'self' has and it is correct (in this case /DF/testStuff). but instead of being created where I'd expect the new objects are being created in /DF I am sure this all come down to my own inexperience but I would really appreciate any advice that a more experiences python/zope person can give. Cheers, altkey (aka Phil)

3 3

Warning: E-mail viruses detected
by MailScanner 06 Feb '04

06 Feb '04

Our virus detector has just been triggered by a message you sent:- To: phpxmlrpc(a)usefulinc.com Subject: Hello Date: Fri Feb 6 16:17:42 2004 One or more of the attachments are on the list of unacceptable attachments for this site and will not have been delivered. Consider renaming the files or putting them into a "zip" file to avoid this constraint. The virus detector said this about the message: Report: Windows Screensavers often hide viruses in email (data.scr) -- MailScanner Email Virus Scanner www.mailscanner.info

1 0