February 2004 - Zope-Dev

RE: [Zope-dev] How to make Zope fail nicely under high load?
by Bjorn Stabell 11 Feb '04

11 Feb '04

> Bjorn Stabell wrote: > > Basically, when the load gets high, Zope has a huge > backload of work > > (several minutes of requests), making the average latency for each > > request many minutes. What are effective ways to do this kind of > > overload management so that the backlog of work doesn't get that big? > > Make the app faster so that doesn't happen. You'll always reach the bottleneck sooner or later, and in 99% of the cases it'll be Zope. It's not exactly a racing horse. [...] > ...return 503 to every new request probably isn't a good > idea, 503 to every new session is probably OK. Maybe. It > depends on the workload. If you have a user who sends 10 > requests and 5 of them fail, they're going to keep hitting > reload to try to get them all to work, which just compounds > your problems. The idea is that you want tweak it so some of > your users get everything, and some get nothing, instead of > everybody getting partials. This means you have to track > sessions though... thats probably easier to do in ZServer > than it is in Apache 1.3 just because of the process model, > but that doesn't mean its easy. > > By default, Apache doesn't track sessions and do that kind of > planned failure. There are probably modules that can enable > that behavior. > > ZServer's multiplexed IO model just hides the accepting > socket from the poller if its concurrency level is reached, > and yeah, then the incoming connections end up in the > backlog. You'd have to change ZServer so it handled those > new requests instead, identified if they were part of a > session, queued them up if they were, or spat out a 503 if > they weren't. With KeepAlive on, wouldn't everything in one "session" sent over one TCP connection? If that is the case, each accept() will basically service a whole session, and so you should be able to do the easy thing of refusing a connection just after an accept(). > Frankly, I bet you'd have more fun just making your app faster. We'll do that as well, but this problem is pretty serious; we can't overload our server for even just a little while before latencies start building up to ridiculous levels and users start hitting reload, further compounding the problem. In the end we have to restart the server, same as Dario reported. Bye, -- Bjorn

4 4

How to make Zope fail nicely under high load?
by Bjorn Stabell 11 Feb '04

11 Feb '04

Hi, We've run into an interesting problem when load-testing a Zope site behind Apache, a problem that we've also encountered in real life. Basically, when the load gets high, Zope has a huge backload of work (several minutes of requests), making the average latency for each request many minutes. What are effective ways to do this kind of overload management so that the backlog of work doesn't get that big? The ideal would be for requests to fail immediately if the backlog of work is more than a certain number of requests (or even better, estimated time to process). Here's what we've tried: Naively, we thought we could just set the socket.listen() backlog in Apache and Zope to a lower value, but in TCP connect()'s apparently don't fail if the server's listen backlog is full; instead requests are retried, resulting in a client side managed "listen backlog", also giving the same long latency. (If someone knows this stuff, please confirm/deny these allegations against TCP :) It appears the way to control it would for Apache or Zope to return "503 Service Unavailable" when the load is too high, but we haven't found a good way to do this; Zope doesn't appear to have any mechanism for it, and Apache's ProxyPass doesn't either. I guess load balancers would, but that's a bit overkill since we run the server on one machine. Regards, -- Bjorn Stabell bjorn(a)exoweb.net Tel +86 (10) 65918490

6 6

RE: [Zope-dev] How to make Zope fail nicely under high load?
by Bjorn Stabell 11 Feb '04

11 Feb '04

Toby wrote: > Hmm, yes. ZRendezvous doesnt have any choice but to queue up > the request. > Handling the request at that point could cause problems > because the support > for pipelined http requests might cause another recursive call to > ZRendezvous.handle, and deadlock. You need to handle it earlier. > > Maybe tweak the readable() method of zhttp_server and > zhttp_channel in > HTTPServer.py, to stop bytes coming in over the socket if the > queue is too large. > > ? Hm. And making this limit (and the listen backlog parameter) available as configuration options. :) Not letting bytes through is not enough, though. I think a 503 error should be returned as quickly as possible back to the client. > > I wonder if Zope also processes requests that have been closed from > > the client end? > > It will > > > I know Zope will continue processing a request even if it > > times out > > at least (so you can do long-running requests, like > > packing, through > > the web). Many (all?) of these 300 requests would have > > been closed by > > the time Zope got to process them, and so Zope shouldn't > > bother with them. > > That doesnt need fixing. It wouldnt be a problem if the queue > was kept small. Well, I think it would help in many cases. Many people click on links before the server has finished (or before it has even visibly started) serving up pages. Searching the web today I saw a commercial product (I think it was a load balancer) promoting this as an important feature. Any page for which the client has closed the connection before the server has started serving up the page could be skipped. If the server has already started serving up the page, it could complete the transaction. Bye, -- Bjorn

1 0

Re: [Zope-dev] Adapters in Zope 2
by santi＠zetadb.com 10 Feb '04

10 Feb '04

>> I've been trying to add some extra functionallity to a Folder object >> without inheriting, but I'm not able to make it work. > > First question is "Why?". Unless you want to change the fucntionality of > *all* folder objects inheriting is the way to go. > For instance, If I've a repository of Folders and Files and I write two applications using it. In the first application I need an extra method to obtain contained objects in some especific order, and in the second application I need a different order. I can write a new class inheriting from Folder with two diferent new methods, but I think that this is not a good aproach. If tomorrow I write a new application with some other needings I should modify my InheritedFolderClass. I prefer to add this logic in each application, using and adapter. If all we create inherited versions of Folders, collaborations between applications is more difficult. I you have an application using a repository of inheritedFolderA's and I have another one using inheritedFolderB's, our applications can't use the same repository, what would be possible using a repository of Folders and adapters in each application. >> Anybody knows if there is a way to use adapters in zope2 ? Or, if >> not, there is some other way to add functionallity on-the-fly, without >> inheriting ? > > MonkeyPatching. Python is highly dynamic. You rewrite the methods to how > you would like them, and then you replace them with a quick. > > TheClass.OldMethod = NewMethod. > > And now you call the new method with > object = TheClass() > object.OldMethod() > Very cool. Yes, this is a possible way, but not so nice as Adapters (in my modest opinion). Thanks for your answer Santi Camps http://zetadb.sourceforge.net

1 0

data migration from zope 2.6.1 to zope 2.7.0-b2
by altkey 07 Feb '04

07 Feb '04

Hi, I am fairly new to zope & python but I am working to migrate some data/objects from zope 2.6.1 to zope 2.7.0-b2 (and eventually 2.7.0 final. What I am doing is this. (1) get Dublin core metadata + some Dublin core extensions for the objects (2) get pure text representation of object content - I can't use a simple import/export in zexp format because the classes in the receiving zope instance differ significantly from the classes in the sending zope instance. I am able to collect all the data mentioned above and pass it across to the new 2.7.0-b2 zope instance but i am having trouble getting my objects to live where they need to live; specifically their path is not being accepted properly when i recreate the objects. here's a code snippet for an external method i am working on <preliminary stuff removed ...> def addObjects(self, objectList): from Products.CMFCore.PortalFolder import PortalFolder from OFS.Moniker import Moniker import urllib nameList = [] for myObject in objectList: metadata = myObject[0] content = myObject[1] dcMetadata = metadata['DC'] edcMetadata = metadata['EXTENDED_DC'] name = metadata['OBJECT_ID'] oType = metadata['TYPE'] portalType = metadata['PORTAL_TYPE'] metaType = metadata['META_TYPE'] DCType = dcMetadata['Type'] path = metadata['PATH'] root = metadata['ROOT'] selfPath = self.getPhysicalPath() nameList.append((selfPath,name,portalType,metaType,path)) self.invokeFactory(portalType,'__'+name) q = content['CONTENTS'] uq = urllib.unquote(q) <other code to manipulate the object's content removed ...> <clean up stuff removed ...> the line "self.invokeFactory(portalType,'__'+name)" should be creating the new objects should be created in the context (self) that the external method is executing in - i checked what path 'self' has and it is correct (in this case /DF/testStuff). but instead of being created where I'd expect the new objects are being created in /DF I am sure this all come down to my own inexperience but I would really appreciate any advice that a more experiences python/zope person can give. Cheers, altkey (aka Phil)

3 3

Warning: E-mail viruses detected
by MailScanner 06 Feb '04

06 Feb '04

Our virus detector has just been triggered by a message you sent:- To: phpxmlrpc(a)usefulinc.com Subject: Hello Date: Fri Feb 6 16:17:42 2004 One or more of the attachments are on the list of unacceptable attachments for this site and will not have been delivered. Consider renaming the files or putting them into a "zip" file to avoid this constraint. The virus detector said this about the message: Report: Windows Screensavers often hide viruses in email (data.scr) -- MailScanner Email Virus Scanner www.mailscanner.info

1 0

Minor windows intaller bug and a question
by Sergey Volobuev 05 Feb '04

05 Feb '04

Hi all, Installing 2.7.0rc2 windows zope installer, i noticed that it uses hard-coded value for instance directory. The proposed value is "C:\Zope-Instance". However, some users (me :) don't have C: drive. For some reasons windows choose G: letter for boot drive and (unlike not-boot drives) this letter can't be changed. Most installers autodetect this situation but Zope installer not. And now the question: I want to make customized Zope installer. I.e. i want to include some products, data in ZODB, etc. so i can give single exe file to my customer, he will install it and get ready-to-work site figurally in one click. Something like Plone installer did. So, are there some guidelines how it can be done (howtos, docs, scripts, etc)? Unfortunately, my expirience in making installers is limited to InstallShield usage in mid 90th :(

5 9

FW: [Zope-dev] strange behavior of VirtualHostMonster
by zope-dev＠netchan.cotse.net 05 Feb '04

05 Feb '04

Okay, I debugged a bit more: > [u'loginUser', u'usermgrFolder', '/', 'VHM'] <= this looks weird to me It seems it's pretty okay, however the last line in VirtualHostMonster.__bobo_traverse__ says: return parents.pop() # He'll get put back on Of course, the popped item is 'a', that's the one with the beforetraversal hook, so if it's really put back on later, than the hook will run again. I think this is the explanation, but the question remains: how can I avoid the second call to the beforetraverse hook? Regards, Sandor _______________________________________________ Zope-Dev maillist - Zope-Dev(a)zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )

2 1

strange behavior of VirtualHostMonster
by zope-dev＠netchan.cotse.net 05 Feb '04

05 Feb '04

I have a zope-behind-an-apache configuration (OpenBSD 3.4, Zope2.7rc1, python 2.3.3, apache 1.3x). Inside the zope root, there's a folder called 'Instances', and inside this there's my custmo object (eLMS) called 'a': /Instances/a. A beforetraverse hook is registered on 'a'. I configured two access paths to this object in apache: <VirtualHost 1.2.3.4:443> RewriteRule ^/(.*) http://127.0.0.1:8003/VirtualHostBase/https/test.server.org:443/Instance s/a/VirtualHostRoot/$1 [L,P] </VirtualHost> <VirtualHost 1.2.3.4:444> RewriteRule ^/(.*) http://127.0.0.1:8003/VirtualHostBase/https/test.server.org:444/VirtualH ostRoot/$1 [L,P] </VirtualHost> (I omitted the irrelevant parts) The first one is the 'main entrance' directly to the 'a' object via https://test.server.org, the other one is a 'service entrance' to the root folder via https://test.server.org:444. Problem: if I try to access anything via the 'main entrance', the beforetraverse hook is called two times. Also I have strange things inside BaseRequest.traverse, like '/' on the path. My question: is the first rewrite rule a valid one? If not, how can I achieve the same results with a valid one? A debug shot from the traverse call: > /zope/zopestable/lib/python/ZPublisher/BaseRequest.py(257)traverse() (Pdb) path [u'loginUser', u'usermgrFolder', '/', 'VHM'] <= this looks weird to me (Pdb) parents [<Application instance at 3cdb9fb0>, <Folder instance at 3cdcc8f0>, <eLMS instance at 3cdc5260>] (Pdb) request.get_header('PATH_INFO') '/VirtualHostBase/https/test.elms.vanth.org:443/Instances/a/VirtualHostR oot/' Regards, Sandor

1 0

change to zLOG.LOG() behavior
by Jeremy Hylton 04 Feb '04

04 Feb '04

I noticed that Zope 2.7 and Zope 2.8 changed the default zLOG implementation to use the logging package. There's one subtle consequence of that change that I wonder if people are aware of. The zLOG.LOG() call takes an optional error argument that contains a 3-tuple of exception information as returned by sys.exc_info(). In the old zLOG, this exception was formatted and logged. In the new zLOG, the error argument is treated as a flag; if it is not None, the *current* exception is formatted and logged. Should I change my code to pass True for error instead of sys.exc_info()? There is at least one place in ZODB that is broken as a result of that change. The old API allowed code to capture an exception with sys.exc_info() and log it later. The common pattern seemed to be to capture the error, try to recover, and only log the original exception of recovery failed. I don't think there's a good solution to the problem, except to change all the code that tries to be clever about capturing exceptions. This affects some ZODB code that was a culprit in recent bug reports that locked up Zope. As a result my fix -- which was supposed to log a helpful traceback rather than prevent Zope from locking up -- probably didn't work. Should we give up on fixing this bug for Zope 2.7? Jeremy

1 0