August 2004 - Zope-Dev

Possible Windows Service improvements.
by Mark Hammond 12 Oct '04

12 Oct '04

Hi all, I am starting to venture into the wonderful world of Zope! With the benefit of a complete lack of Zope experience, I have been able to look at the Windows service support from a fairly clean slate. However, I also realize this lack of experience means my ideas may be naive - hence I have attempted to split them into discrete issues for discrete rejection <wink>. 1) startup error redirection. I've noticed that the main Zope service driver for Windows seems to work fine when everything is setup correctly, but when things go wrong it offers no clues as to what. This is reflected in collector item 1020 ("poor error reporting on product initialisation failure under windows"). Issue 1408 ("Configuration file imports don't see INSTANCE_HOME when running Zope as a windows service"), via the referenced thread, has evidence of someone burning a day due to this. It cost me alot of time too :) I propose: Each time the child process terminates with a non-zero return code, the tail x-bytes of the child output be written to the Windows event log, where x~2k. 2) reporting of "successful start" and "backoff" strategy. A trivial startup error (eg, PYTHONPATH not set) will cause the Zope service to hopelessly retry for a number of minutes, and not respond to shutdown requests during a retry. At the moment, as soon as the service starts it reports "successful startup" to Windows. It then begins an attempt to start the child. If the child immediately fails, the code immediately begins the "backoff" strategy. This strategy appears to have 2 main purposes: * Startup may fail due to other 'services' not having yet started, so retry in the hope they become available. * The process may die due to some obscure error - restart it. On windows, assuming we install the service to depend on the "tcpip" service, I see no reason that the first reason is valid. If the process fails quickly the first time we attempt to launch it, it is almost certainly going to fail every time we try and launch it. The current strategy also means that 3rd party services could not themselves depend on the Zope service - the Zope service will report successful startup before it really has (and therefore the dependent service may itself fail). This isn't a known requirement today, but who knows! "net start" and other front ends also fail to detect fatal errors - they all say Zope started OK. I propose: We insist the child process can be created and continues to run for x seconds (where x~5). If that fails, we report an error (never reporting to Windows that we started successfully). If the child process stays alive for this period, we report success to Windows, and then use the existing backoff strategy should it die. If the machine is heavily loaded, this 5 seconds may expire before the fatal error is hit in the child - in that case, we are simply doing what we do now - using the backoff strategy to hopelessly attempt a restart - ie, a win in most cases, and no loss in the others. 3) environment setting The service process should set a number of environment variables before spawning the child - PYTHONPATH at a minimum, and according to issue #1408, INSTANCE_HOME. It already knows these values thanks to mkzopeinstance. I'm yet to determine where these values comes from for in binary build, but I see no reason not to fix this (and possibly remove whatever magic the binary does) I propose: A few trivial os.environ insertions based on the substitutions done by mkzopeinstance, before we create the child process(es). Alternatively, we create an explicit new environment we pass to CreateProcess, but I see no good reason for that.) 4) Currently, when the process is stopped, we immediately terminate the child process. This seems dangerous. We should find a way to gracefully terminate the child, and try that before we simply kill it. I propose: That someone help me work out how to do this <wink>. I've already worked out how if the service knows the username/password of a Zope administrator, but it doesn't! Sending a Ctrl_C 'signal' doesn't work without hacks to run.py (and I'm yet to confirm it will even with such hacks). I welcome any feedback on these issues. Obviously I am willing to back each of these proposals up (except 4!) with code that seems to work :) I would also welcome feedback on the best way to proceed (ie, create a new collector for each issue? thrash it out here? give up?<wink>, etc) Note that none of these issues would require a win32all/pywin32 update. If anyone was really upset by issue 1423 ("Zope 2.7.1 won't run as service under NT"), and also able to test, I'd be willing to fix it - but that *would* require a pywin32 upgrade. Tim has already kindly filled me in on that background, so it may not be trivial (ie, I would need help!) Thanks! Mark

8 19

Developing plugins for PluggableAuthService
by Lennart Regebro 01 Sep '04

01 Sep '04

I just made a very neat plugin for PAS. I added it and activated it for everything it can do. It is now the *only* plugin active for Extraction, Challenge and Reset Credentials. The problem is howvere, that nothing happens. None of the methods in the plugin gets called, ever! So, eh, has anybody else here written plugins for PAS so they can point me in the right direction? //Lennart

4 16

Pointless code in ZPublisher.HTTPRequest ?
by Paul Winkler 01 Sep '04

01 Sep '04

Can anyone enlighten me on what this is for? >From ZPublisher.HTTPRequest.HTTPRequest.keys: n=0 while 1: n=n+1 key = "URL%s" % n if not self.has_key(key): break n=0 while 1: n=n+1 key = "BASE%s" % n if not self.has_key(key): break Unless I am particularly stupid today (quite possible), the above has no affect on the return value, and no direct side effects. I tried commenting it out and ran all unit tests with no apparent difference. While digging deeper for indirect side effects, I see that self.has_key() calls self.__getitem__() which calls self.get(). The only side effect of self.get('URLn') that I see is that if self.other.has_key('PUBLISHED'), the calculated URL will get cached for the duration of the request in self.other['URLn']. (And also in self._urls which AFAICT is used only to keep track of which keys we've written to self.other, so we can quickly clean them out in _resetURLS().**) So maybe that's the sole purpose of the code in question; some extra logging suggests that it does in fact have that effect. But it's hard for me to be sure. Can anyone confirm or deny? Or is there something else that I'm missing? ** btw, this too smells a bit funny to me... why bother to keep a separate data structure in sync, when we could just do: for i in range(9): try: del(self.other['URL%d' % i]) i += 1 except KeyError: break -- Paul Winkler http://www.slinkp.com

3 5

Transcience ? Problems
by Dario Lopez-Kästen 31 Aug '04

31 Aug '04

Greetings I have the following error showing up in my event.log (Zope 2.7.1, python 2.34, Plone 2.03) ---- ... File "/usr/local/zope/instances/cdks/Products/PlacelessTranslationService/Negotiator.py", line 266, in _negotiate userchoices = getLangPrefs(request, kind) File "/usr/local/zope/instances/cdks/Products/PlacelessTranslationService/Negotiator.py", line 54, in getLangPrefs accepted = handler.getAccepted(env, kind) File "/usr/local/zope/instances/cdks/Products/PlacelessTranslationService/Negotiator.py", line 172, in getAccepted language = request.SESSION.get('pts_language', None) File "/usr/local/zope/software/zope-2.7.1/lib/python/ZPublisher/HTTPRequest.py", line 1218, in __getattr__ v = self.get(key, default, returnTaints=returnTaints) File "/usr/local/zope/software/zope-2.7.1/lib/python/ZPublisher/HTTPRequest.py", line 1178, in get if callable(v): v = v() File "/usr/local/zope/software/zope-2.7.1/lib/python/Products/Sessions/SessionDataManager.py", line 94, in getSessionData return self._getSessionDataObject(key) File "/usr/local/zope/software/zope-2.7.1/lib/python/Products/Sessions/SessionDataManager.py", line 181, in _getSessionDataObject ob = container.new_or_existing(key) File "/usr/local/zope/software/zope-2.7.1/lib/python/Products/Transience/Transience.py", line 768, in new_or_existing self[key] = item File "/usr/local/zope/software/zope-2.7.1/lib/python/Products/Transience/Transience.py", line 439, in __setitem__ current_bucket = self._data[current_ts] KeyError: 1093704740 ---- After this my site locks up ans is not responding. What I do get in the log is thousands of conflict errors: -- ****** File "/usr/local/zope/software/zope-2.7.1/lib/python/Products/Transience/Transience.py", line 439, in __setitem__ current_bucket = self._data[current_ts] KeyError: 1093704740 ------ 2004-08-28T16:52:53 INFO(0) ZODB conflict error at /VirtualHostBase/https/XXX:443/cdks/VirtualHostRoot/heartbeat.txt (430 conflicts since startup at 2 004-08-26T15:50:14) ------ 2004-08-28T16:52:59 INFO(0) ZODB conflict error at /VirtualHostBase/https/XXX:443/cdks/VirtualHostRoot/heartbeat.txt (431 conflicts since startup at 2 004-08-26T15:50:14) ... ------ 2004-08-30T08:46:22 INFO(0) ZODB conflict error at /VirtualHostBase/https/XXX:443/cdks/VirtualHostRoot/heartbeat.txt (60082 conflicts since startup at 2004-08-26T15:50:14) ------ 2004-08-30T08:46:23 INFO(0) ZODB conflict error at /VirtualHostBase/https/XXX:443/cdks/VirtualHostRoot/heartbeat.txt (60083 conflicts since startup at 2004-08-26T15:50:14) -- ****** The only thing that helps at his point is restarting. Any ideas on what might be happening? And why I keep getting these conflict errors? Thanks, /dario -- -- ------------------------------------------------------------------- Dario Lopez-Kästen, IT Systems & Services Chalmers University of Tech.

3 2

UI improvement for catalog
by Christian Theune 31 Aug '04

31 Aug '04

Hi. I'm landing a small UI improvement for the ZCatalog ZMI interface. It let's you set a path filter to show catalog entries from, if a PathIndex called "path" exists. (This is created by default in CMF and Plone sites). Feedback is welcome. Theuni -- Christian Theune <ct(a)gocept.com> gocept gmbh & co. kg

2 1

how to start workflow instance on another zope server with SOAP?
by Aruna Kathiriya 31 Aug '04

31 Aug '04

Hi, I am very new to open source. I am trying to start an instance of a workflow on a Zope server from another Zope server. I did successfully with XML-RPC call but I need to do it with SOAP. I am note sure weather I can do it or not. I am running two Zope servers on my local m/c on 2 different ports. One is 8080 and 2nd one is 8090. On 8080: Page_template : SoapTest2 <html> <body> Activate instance at other server: <span tal:define= " l_user python:request.get('AUTHENTICATED_USER');"> <form action="SubmitNewSoapTest2" method="post" enctype="multipart/form-data"> <input type="file" name="p_file"><br> <input type="hidden" name="user" tal:attributes= "value python:l_user;"> <input type="hidden" name="password" value="Aruna"> <input type="submit" value="Submit"> </form> </span> </body> </html> SubmitNewSoapTest2 is external method. the python script for that is testTrigger as follow. from SOAPpy import SOAPProxy import xml.sax def startTestInstance2(self,p_file,user,password): myFile =p_file.read() # parser = xml.sax.make_parser() # parser.parse(myFile) url = 'http://localhost:8090/' namespace = 'urn:xmethods-testInstance' server = SOAPProxy(url, namespace) return server.SoapTestInstance(p_file,user,password) On 8090: I have a python script SoapTestInstance having 3 parameters (p_file,user,password) and following code. wf = context.TestFlow i_id = wf.addInstance('TestProcess', '', '', '', 0) instance = wf.getInstance(i_id) instance.manage_addFile(id="theFile", file=p_file, title="") wf.startInstance(i_id) wf.activateWorkitem(i_id, '0', user ) wf.completeWorkitem(i_id, '0') return "created new instance with code %s" % i_id It gives me following error. Error Type: SAXParseException Error Value: http://www.w3.org/TR/REC-html40/loose.dtd:31:2: error in processing external entity reference Any help will be appriciated. I already stucked here.. Aruna Kathiriya Sr.Consultant CIGNEX Technologies, Inc U: www.cignex.com "Implement IT Right"

2 1

is an object in a folder
by Florent Guillaume 31 Aug '04

31 Aug '04

I'd like to add a method like def hasObject(self, id): """Test if an object is in the current object.""" for o in self._objects: if o['id'] == id: return 1 return 0 to ObjectManager. This would bring it in line with BTreeFolder2 (who already has an hasObject method) and we could then always use the most efficient method to test if a folder has a given subobject id. Opinions ? Florent -- Florent Guillaume, Nuxeo (Paris, France) +33 1 40 33 71 59 http://nuxeo.com mailto:fg@nuxeo.com

6 7

closed collector issues 1252 and 1308
by yuppie 31 Aug '04

31 Aug '04

Hi! Just want to let you know that I'm not happy with the way these two collector issues were "resolved": <http://zope.org/Collectors/Zope/1252> <http://zope.org/Collectors/Zope/1308> Hope at least one of them will be re-opened. Cheers, Yuppie

4 11

Questions and suggestions for PAS and BasicRequest.
by Lennart Regebro 31 Aug '04

31 Aug '04

Background for zope-dev: Me, Tino and Zach has been discussing and working a bit on PAS, to fix the problems I had last week. Most have now been fixed. Some new ones arrived instead. :) The two first have to do with both Zope and PAS: 1. The challenge() currently requires you to either raise a Redirect or and Authenticated exception, to go to a login-page or get a login-box. This is because if you just return from challenge without any exception BasicRequest will later raise unauthorized. This means you can not do response.setBody() or even response.redirect(). One possible solution for this is to NOT provoce the challenge from the user folders verify() method, but challenge from Response.unauthorized(). 2. I would assume that a redirecting challenge will redirect a webdav client too. That is a bad idea, and needs to be taken care of (by PAS I assume, but not 100% sure how). These further two are more PAS specific: 3. If you have a PAS in the Zope root, and you have a broken challenge plugin, you may not be able to log in at all. This also needs to be taken care of in one way or another. One way could be to always allow username:password@url authentication for emergency user. Other ideas could be to have some setting to disable challenges and restart the server, or something like that. 4. The only plugin to actually do any proper challenges as of today, is my newly created CAS-plugin. Question: Do you want that as a part of PAS? I can simply check it in there. Otherwise it will become a separate product. //Lennart

1 0

New function for error log
by Christian Theune 29 Aug '04

29 Aug '04

Hi. I'm continuing to improve my small nagios plugin for Zope. I started to include error messages in the sensor. Therefore I was missing a way to get error messages out of the error log to signal nagios that everything is fine. I have a working patch attached (it references an image that is ready to be added as well). Due to the fact that this is a feature and the code provides no unit test infrastructure (and I don't have the time to start doing that for the error log) I'd like to know what the general rule for Zope 2 is, If I want to make improvements to existing code that isn't covered by unit tests at all. I'd check this in to the svn trunk if you like. Cheers, Christian -- Christian Theune <ct(a)gocept.com> gocept gmbh & co. kg

3 8