- Zope-Dev - Zope lists

Re: [Zope-dev] Patch: let non-seekable streams be input for ZPublisher (updated)
by Ames Andreas (MPA/DF) 23 Aug '04

23 Aug '04

Hello, Dieter Maurer wrote: > Maybe, I have a much simpler solution: > > Something in ZServer makes all file fields seekable by > delivering them through some temporary (either a StringIO > or a temporary file). > > Maybe, you could do the same for your requests? Yes, I could. But trying to avoid it was the very reason I tried to patch HTTPRequest for. What you propose is what the HTTPServer does for instance. I have at least two objections: 1) For big requests (big means some hardcoded threshold) HTTPServer buffers the request on disk. Then ZPublisher creates a FieldStorage instance which will write the request once again to disk. And that also means the request is read from disk several times. That's a bit too much blocking disk I/O in a production webserver for my liking. And that's not counting retries; each retry will add another pair of blocking disk I/O in the current implementation. What I'm trying to do is: - avoid unneeded disk I/O in retries - make it possible that self.stdin in HTTPRequest can be non-seekable such that it doesn't have to be buffered on disk before FieldStorage is created (and does exactly that buffering). 2) I'm convinced that the one and only ZServer thread should do without any blocking (I/O) call (besides select() that is) and I want AJPServer also be an experiment to see if this improves performance. cheers, andreas

1 0

Re: Debian and Zope
by Thaddeus H. Black 23 Aug '04

23 Aug '04

This question regards the code in Zope's zpasswd.py. If you know zpasswd.py or are involved in Debian, please reply. Debian GNU/Linux 3.1 releases soon. The Debian Zope package (actually the Debian `zopectl' package) has what Debian calls a Release Critical Bug open against it [http://bugs.debian.org/251038] Normally, Debian's Zope maintainer would handle such a bug, but the maintainer seems to be inactive this year, so it falls to me to fix it. This is okay. The maintainer's work in earlier years is appreciated. The trouble is, I do not use Zope myself, so I am fixing the bug with limited understanding of the potential consequences of the fix. The part of the Debian fix which affects your Zope source is in zpasswd.py. At present, a user can invoke zpasswd in either of two ways: (1) without command-line options, in which case zpasswd prompts the user on tty for username and password; or (2) with username and password supplied on the command line. How the user cannot presently invoke zpasswd is (3) with username supplied on the command line, so that zpasswd prompts the user on tty for a password only. The Zope installation procedure on Debian requires option (3). With (3), I can cleanly eliminate the Release Critical Bug now and prevent Zope from being dropped from Debian 3.1. Thus I am adding (3) to Debian Zope. Question, please. Does my addition create some subtle problem I should know about? If Zope were coded perfectly cleanly, then the addition of option (3) should create no problem; but you and I have both done enough coding to realize that old code can sometimes depend in strange ways on the odd behavior of even older code. With my small addition, zpasswd now correctly handles (3). Is this okay, or can you think of something specific it might break, something I might not have noticed? I have checked everything I know how to check, but I do not know Zope; you guys do. Regrettably, Zope's public SVN seems to be down as I write these words, so in the unlikely event that someone had already added option (3) to zpasswd.py, I would not know about it. I do understand and respect that, like any good free-software developers, you will probably all want me to update Debian Zope now to Zope's latest version! To this, I must plead that such an update far exceeds the limits of my Zope ignorance (also, it would violate Debian Project policy, which permits me minimally to fix a Release Critical Bug for an inactive maintainer but forbids me from unilaterally hijacking his Debian package). The matter at hand now is not how to include the latest Zope (it is not Debian's usual practice to include the very latest software in any event) but rather how to keep Zope in Debian at all. This is a good cause and I am pleased to help. I do not regularly subscribe to zope-dev, so please copy your reply to me and to 251038(a)bugs.debian.org (the latter automatically attaches your reply to the relevant official Debian bug report, #251038). If there were no reply, this would be okay; I would just assume that my fix were as good as it seems to be, and would proceed accordingly. Please reply by 23 August. If any active Zope developer on this list also happens to be a registered Debian Developer, he is asked to step in and take charge of Debian bug #251038 from here: one can find my full zopectl/zope Debian patch attached to the Debian bug report. Otherwise, thank you for your attention in this matter. If you want my patch to zpasswd.py, here it is. You probably want to apply it to your own SVN Zope source. diff -u zope-2.6.4/zpasswd.py zope-2.6.4.new/zpasswd.py --- zope-2.6.4/zpasswd.py 2004-08-03 20:34:53.000000000 +0000 +++ zope-2.6.4.new/zpasswd.py 2004-08-03 18:49:11.000000000 +0000 @@ -87,6 +87,16 @@ import do; do.ch(ac_path, user, group) +def get_password(): + while 1: + password = getpass.getpass("Password: ") + verify = getpass.getpass("Verify password: ") + if verify == password: + return password + else: + password = verify = '' + print "Password mismatch, please try again..." + def main(argv): short_options = ':u:p:e:d:' @@ -150,7 +160,10 @@ # Verify that we got what we need if not username or not password: - raise "CommandLineError" + if username: + password = get_password() + else: + raise "CommandLineError" access_file.write(username + ':' + generate_passwd(password, encoding) + @@ -163,14 +176,7 @@ if username != '': break - while 1: - password = getpass.getpass("Password: ") - verify = getpass.getpass("Verify password: ") - if verify == password: - break - else: - password = verify = '' - print "Password mismatch, please try again..." + password = get_password() while 1: print """ -- Thaddeus H. Black 508 Nellie's Cave Road, Blacksburg, Virginia 24060, USA; +1 540 961 0920, t(a)b-tk.org

3 4

Re: [Zope-dev] Re: Debian and Zope
by Chris Withers 23 Aug '04

23 Aug '04

Thaddeus H. Black wrote: >>Can you tell us who that maintainer is? > > Luca De Vitis <luca(a)debian.org>. Last known > Zope activity: 4 Nov 2003. Hmmm, don't know him. Anyone else know him? Oh well, I guess we'll just have to wait for someone with the necessary Zope and Debian Zen to emerge :-/ cheers, Chris -- Simplistix - Content Management, Zope & Python Consulting - http://www.simplistix.co.uk

1 0

More unit test questions
by Lennart Regebro 20 Aug '04

20 Aug '04

OK, I got test.py running. But if I run bin/python test.py Products it will happily try to run the tests in all products. But if I type bin/python test.py Products/Productname It will find no tests at all. bin/python test.py Products/Productname/tests is no better. BUT! If I type bin/python test.py lib/python/Products/ZCTextIndex it does find the tests for ZCTextIndex. In short: Heuh? What are the difference that makes it impossible to test just one products unit test if it's located in Products instead of lib/python/Products. Or is there a difference in how the CMF tests are written? Anybody know?

3 3

Running Zope tests in 2.7 fails
by Lennart Regebro 20 Aug '04

20 Aug '04

I have finally realized that the way tests are run in 2.7 is different from earlier version of Zope, and I'm trying to get them to run. It's Zope 2.7.2 on Debian. Python 2.3.4 and 2.3.2, so I guess 2.3.3 would give the same error. Could it be a side effect of the compiling or something? Here is the traceback: Running unit tests from /home/regebro/Zopes/Zope-2.7.2-0 /home/regebro/Zopes/Zope-2.7.2-0/lib/python/DocumentTemplate/cDocumentTemplate.so: undefined symbol: PyUnicodeUCS4_DecodeLatin1 ['/home/regebro/Zopes/Zope-2.7.2-0/lib/python', '/home/regebro/Zopes/Zope-2.7.2-0', '/usr/local/lib/python23.zip', '/usr/local/lib/python2.3', '/usr/local/lib/python2.3/plat-linux2', '/usr/local/lib/python2.3/lib-tk', '/usr/local/lib/python2.3/lib-dynload', '/usr/local/lib/python2.3/site-packages'] Traceback (most recent call last): File "test.py", line 841, in ? process_args() File "test.py", line 831, in process_args bad = main(module_filter, test_filter, libdir) File "test.py", line 629, in main files = find_tests(module_filter) File "test.py", line 460, in find_tests os.path.walk(walkdir, finder.visit, rx) File "/usr/local/lib/python2.3/posixpath.py", line 290, in walk walk(name, func, arg) File "/usr/local/lib/python2.3/posixpath.py", line 290, in walk walk(name, func, arg) File "/usr/local/lib/python2.3/posixpath.py", line 282, in walk func(arg, top, names) File "test.py", line 439, in visit __import__(pkg) File "/home/regebro/Zopes/Zope-2.7.2-0/lib/python/AccessControl/__init__.py", line 16, in ? import DTML File "/home/regebro/Zopes/Zope-2.7.2-0/lib/python/AccessControl/DTML.py", line 18, in ? from DocumentTemplate import DT_Util File "/home/regebro/Zopes/Zope-2.7.2-0/lib/python/DocumentTemplate/__init__.py", line 22, in ? from DocumentTemplate import String, File, HTML, HTMLDefault, HTMLFile File "/home/regebro/Zopes/Zope-2.7.2-0/lib/python/DocumentTemplate/DocumentTemplate.py", line 112, in ? from DT_String import String, File File "/home/regebro/Zopes/Zope-2.7.2-0/lib/python/DocumentTemplate/DT_String.py", line 17, in ? from DT_Util import ParseError, InstanceDict, TemplateDict, render_blocks, str File "/home/regebro/Zopes/Zope-2.7.2-0/lib/python/DocumentTemplate/DT_Util.py", line 42, in ? from cDocumentTemplate import InstanceDict, TemplateDict, \ ImportError: /home/regebro/Zopes/Zope-2.7.2-0/lib/python/DocumentTemplate/cDocumentTemplate.so: undefined symbol: PyUnicodeUCS4_DecodeLatin1 Any ideas why this is? The Zope in itself seems to work fine. I have a 2.7.0 instance to, it also fails, but with a completely different error. //Lennart

3 3

Re: [Zope-dev] Patch: let non-seekable streams be input for ZPublisher (updated)
by Ames Andreas (MPA/DF) 20 Aug '04

20 Aug '04

Hi Dieter, *, Dieter Maurer wrote: > The "FieldStorage" must be rebuilt, because it can contain > file objects. These file objects must be reset as they may > have been (partially) read in the previous request. > This prevent reusing the previous "FieldStorage". As you may have seen in my first patch proposal I actually tried to address this problem, although halfbaken and buggy. So here's the next trial. I hope I've got it straight now. Why didn't I do something to reset the files, FiledStorage creates? Because I thought it wouldn't be needed. If you look at cgi.FieldStorage you will see, that accessing FieldStorage's 'value' field, will always reset the file before reading it and HTTPRequest's 'BODY' field does the same. Cause of your hint I looked again and realised that I was wrong. HTTPRquest's 'BODYFILE' field fails to reset the file before accessing it (as in OFS/Image.py the File class does). That could be easily fixed. What's worse is that FiledStorage can possibly create subinstances of itsellf which can possibly contain other fileobjects. This is the case for multipart-type requests. How to solve this? I can see three alternative aproaches: 1) In HTTPRequest.retry loop through self._fs.list (if it exists) and call seek(0) on every fileobject, you can find there. This can only work if the maximum depth of the FiledStorage tree is predictable, which in turn depends on the possible input. I don't know enough about the possible HTTP requests to determine such a limit or to even 'prove' its existance. 2) In HTTPRequest.retry recursively walk through the FieldStorage trees in self._fs.list (if it exists). I wouldn't do that in Python without being able to predict the (worst case) number of function calls needed to do this recursion. And without refreshing my knowledge of compiler techniques I wouldn't be able to roll the recursion out to a loop. 3) This is a somewhat drastic (and possibly conceived as hackish?) approach which OTOH is independent of the input and should perform quite well. I overwrite cgi.FieldStorage, esp. its make_file method, such that it adds all fileobjects, that it creates, to a (possibly caller provided) list. Thus the caller (i.e. HTTPRequest) has a list of all those fileobjects that need to be reset if the FieldStorage should be reused (as in ZPublisher's retries). For the patch attached I chose the third approach. Please let me know, what you think about it. If it is assessed to be ok I will update the collctor item, I opened yesterday. cheers, andreas

2 1

Re: [Zope-dev] Patch: let non-seekable streams be input for ZPublisher (updated)
by Ames Andreas (MPA/DF) 20 Aug '04

20 Aug '04

Hi, Andreas Ames wrote: > next trial. I hope I've got it straight now. Not quite so (although this may only be nitpicking ;-). fs_env = environ should have read fs_env = environ.copy() obviously. See attachment. cheers, andreas

1 0

Patch: let non-seekable streams be input for ZPublisher (updated)
by Ames Andreas (MPA/DF) 18 Aug '04

18 Aug '04

Hi, I had a thinko in my previous patch (http://mail.zope.org/pipermail/zope-dev/2004-August/023630.html) which is corrected in the attached version of the patch. Sorry for any inconvenience, I might have caused. As I'm sort of protocol challenged (thanks to corporate firewall), svn unfortunately isn't an option for me. Thus I created the patch against lib/python/ZPublisher/HTTPRequest.py as released in 2.7.2-0. As the patch is almost trivial it shouldn't be a problem to apply it even manually against other vesions (at least 2.7.0, 2.7.1 and 2.7.2 should work OOTB i.e. by using /usr/bin/patch). To sum up its purpose again: It's meant to avoid the stdin.seek() call in the HTTPRequest class so that non-seekable streams can be used as stdin. It also potentially avoids unnecessary (blocking) disk I/O in retries (this depends on the according request that's being worked on) and repeated request parsing in cgi.FieldStorage. The immediate cause for the patch is that AJPServer uses non-seekable streams as input for the ZPublisher. To change this locally I'd need to go with disk buffering. Nevertheless I think this could provide a general performance improvement. To this end I just save the cgi.FieldStorage instance, that is created by HTTPRequest.processInputs, when the request is worked upon for the first time, across retry boundaries. Questions: - Is this the right forum/place to send patches to? - Is there any chance that this could be applied to Zope's mainline? (If not I will proceed with a local disk buffering scheme in the long term.) cheers, andreas

3 2

Zope 2.7.2, restrictedTraverse or hasattr
by Kai Hoppert 18 Aug '04

18 Aug '04

Hi all, i actually have a Problem with restrictedTraverse and hasattr. Object1 Object2 Object3 I want to test if there exists a Object in Object3 wich has the same id and meta type how Object2 (id=myobject). so i ask Object3.restrictedTraverse('myobject',None) or hasattr(Object3,'myobject') i every time get's true back. I know that he jump's to Object2. i testet this with absolute_url, but this makes no sense i think. Either i never noticed that this both funktions works in this way or it is a bug. If it is no bug has anybody a solution for my problem? Greetings Kai Hoppert ___________________________________________________ tomcom Gesellschaft für Informationstechnologie mbH Heuriedweg 14 | D-88131 Lindau Fon +49 (0)8382 975844 | Fax +49 (0)8382 975855 kai.hoppert(a)tomcom.de | http://www.tomcom.de This document should only be read by those persons to whom it is addressed and is not intended to be relied upon by any person without subsequent written confirmation of its contents. tomcom disclaims all responsibility and accepts no liability for the consequences of any person acting, or refraining from acting on the contents of this document. Any unauthorised form of dissemination, copying, disclosure, modification, distribution and/or publication of this message is strictly prohibited. For information about tomcom please contact us on +49.8382.975844 or visit our web site at www.tomcom.de

3 2

Re: [Zope-dev] Re: "hasattr" geddon
by Shane Hathaway 18 Aug '04

18 Aug '04

On Saturday 10 July 2004 20:54 pm, Casey Duncan wrote: > Shane Hathaway <shane(a)hathawaymix.org> wrote: > > Why are > > we fixing hasattr and bare excepts when the real problem is ZODB? > > Well, the real problem is *probably* people, but I digress... http://c2.com/cgi/wiki?TheProcessIsTheProblem ;-) > Still, swallowing read conflicts can lead to bugs, even in read-only > tranactions. Consider the following function: > > def getStuffFrom(ob, default): > if hasattr(ob, 'stuff'): > return ob.stuff > return default > > I have seen this idiom about in some zope application/framework code. If > the hasattr results in a read conflict, then it returns false and > default is returned by the function even if ob does have the attribute. > In the example I saw (in DCWorkflow) this would result in the wrong UI > elements being displayed for ob mysteriously. Reloading the page would > likely fix the problem, baffling both the user and app developer, who > now both think Zope is playing with their minds. The change to ZODB that I'm suggesting would fix this. The suppressed conflict error would still mark the transaction as uncommitable. When the publisher calls commit() (which it does on every request even if nothing changed), it would result in a ConflictError, then the publisher would call abort() and the transaction would be retried. Users would never see results from a transaction that had any conflict. Shane

3 2