funky side-effects, possible bug in HTTPRequest.py
The last few days I've been working on patch to Zope 2 that will clean up the textarea size preference handling in the ZMI. Right now, its a mess. I kept running into this really irritating behavior such that, when ever I'd go to pull something out of a request object, it'd end up being found in the 'other' dictionary instead of where I expected to be (which happened to be 'form'). After getting really curious as to why this was happening I've managed to track it down to subtle a change in the way that the HTTPRequest.get method worked between revisions 1.75 and 1.74 (1.75 being mj's merge of the value tainting code). The code responsible (assuming the value isn't tainted): # Untrusted data *after* trusted data v = self.form.get(key, _marker) if v is not _marker: other[key] = v # *boom* return v That magical promotion of the key & value to the other dictionary is what tripped me up. This technique, while originally used only for known convenience variables like URLx or BASEx and for promoting lazy data, now applies to all variables after revision 1.75. (This isn't the only spot it gets used now, the tainting changes made it affect form values, cookies, tainted form values, etc.) It would increase the speed at which variables are found--but I'm not sure its really an intended side effect, and now I'll explain how I was bitten by it. Its not unusual to see people write methods to be published that are similar too: manage_main = DTMLFile("edit", globals()) def manage_changeFoo(self, REQUEST, something=None, or_other=628): # stuff return self.manage_main(self, REQUEST) They're a dime a dozen, yay neat. What sometimes makes them more interesting is when people modify the REQUEST dictionaries to pull off various behind the scenes fake-like-we-requested-it-that-way stuff. This is what several of the methods that handled the Taller, Wider, Narrower, etc. buttons on text area prefs did, and thats why I ran into it. One of these methods did: REQUEST.form.update({'something':'x', 'or_other':'y'}) and then it returned a DTMLFile plied with the snazzy new request that had been tricked out with these fake values now stuck into the form dictionary. This explains the TALES I saw that was "request/form/something | request/something | default" which when I first saw made wonder if the author was on acid. It was clear to me that 'something' would never be in other, it wasn't a special variable and it wasn't being explicitly stuck into the other dictionary by the code, so why the redundant TALES I wondered? I removed the first statement, and it broke the functionality. Hmmm! Until about 20 minutes ago I didn't understand exactly how object methods got published, but I tracked down the code in Publish.py and found the mapply() function and now its all clear to me what was happening. The request object gets mapply'd to the published method, this means that any positional arguments or any keyword arguments will get HTTPRequest.get'd out of the request object when they are applied to the published method, and thanks to the side-effects from revision 1.75 on, it also means any named variables in a method definition will be promoted into the HTTPRequest.other dictionary. Now, let me just say - if thats how its supposed to be, so be it - but spin my nipple nuts and call me Frank, this does NOT strike me as terribly intuitive behavior. Anyway, it was a learning experience for me, but I'm not convinced this isn't a bug. What do you think? (the patches I spoke of should be ready sometime tomorrowish assuming I don't run into any other funkyness, I'll post them to the collector) -- Jamie Heilman http://audible.transient.net/~jamie/ "We must be born with an intuition of mortality. Before we know the words for it, before we know there are words, out we come bloodied and squalling with the knowledge that for all the compasses in the world, there's only one direction, and time is its only measure." -Rosencrantz
On Fri, Jun 20, 2003 at 03:07:18AM -0700, Jamie Heilman wrote:
Anyway, it was a learning experience for me, but I'm not convinced this isn't a bug. What do you think?
From the POV of a poor web monkey, my opinion is that this is purely awful. It's the kind of thing that can easily waste hours of debug time.
I was just saying to Andy McKay yesterday that sometimes I think zope 2 is designed around the principle of greatest consternation rather than the principle of least suprise. Case in point. -- Paul Winkler http://www.slinkp.com Look! Up in the sky! It's ANTHROPOLOGIST MAN! (random hero from isometric.spaceninja.com)
Jamie Heilman wrote:
[major snippage]
Hmmm, that means that this changes break exactly these applications, which, in order to be on the secure side, explicitly use REQUEST.form['bla'] more than once in a request, right. Ironic. cheers, oliver
Oliver Bleutgen wrote:
Jamie Heilman wrote:
[major snippage]
Hmmm, that means that this changes break exactly these applications, which, in order to be on the secure side, explicitly use REQUEST.form['bla'] more than once in a request, right.
Naw, it doesn't break them, promotion doesn't mean the vars are actually moved from the form dictionary to the other dictionary, only copied. What it does is humble the poor shmuck who thought he knew the dangers of REQUEST.get well enough to use it without shooting himself in the foot. -- Jamie Heilman http://audible.transient.net/~jamie/ "You came all this way, without saying squat, and now you're trying to tell me a '56 Chevy can beat a '47 Buick in a dead quarter mile? I liked you better when you weren't saying squat kid." -Buddy
Jamie Heilman wrote at 2003-6-20 03:07 -0700:
... That magical promotion of the key & value to the other dictionary is what tripped me up. This technique, while originally used only for known convenience variables like URLx or BASEx and for promoting lazy data, now applies to all variables after revision 1.75.
Zope promotes the variables from "cookies" and "form" to "other" at least since Zope 2.1.6 (the first version I worked with). I think this is for efficiency reasons. You have a single dictionary lookup instead of three ("other", "form", "cookie"). Dieter
Dieter Maurer wrote:
Zope promotes the variables from "cookies" and "form" to "other" at least since Zope 2.1.6 (the first version I worked with).
I think this is for efficiency reasons. You have a single dictionary lookup instead of three ("other", "form", "cookie").
Yeah most promotion is for efficiency, but since 2.1.6 eh? Interesting, that means its happening elsewhere too. Hmm. Yeah, I see, it happens in __init__ for cookies, mmm and in processInputs for form variables I guess. The difference I can see is two fold, first, before 1.75 I believe promotion of form and cookies was only done during request object creation, interestingly, its not done there anymore from the looks of it; cookies and form vars are kept in their seperate tainted & non-tainted buckets until they are fetched. Second the promotion of cookies wouldn't clobber pre-existing keys in other, interestingly, in 2.1.6 times, form vars would clober normal other vars (URLx, BASEx, etc.). I must admit, this isn't the biggest deal to me, I just blew quite a bit of time tracking it down because I couldn't understand why the promotion was happening and I don't like surprises, and nothing about "def foo(self, bar, baz):" immediately jumped out and said to me, "promote bar and baz form vars and cookies to the other dictionary." Now that I have an appreciation for exactly how methods are published, I see things in a different light of course. -- Jamie Heilman http://audible.transient.net/~jamie/ "...thats the metaphorical equivalent of flopping your wedding tackle into a lion's mouth and flicking his lovespuds with a wet towel, pure insanity..." -Rimmer
On June 20, Jamie Heilman wrote:
and I don't like surprises
Zope 2 is probably not the most suitable choice, then. ;-) More seriously, though, my colleagues and I will often find a few of these sorts of *surprising* things in Zope 2 every week. It's really quite demoralising building for Zope 2 when it feels like such a dead effort, with many (seemingly rather deep) problems in Zope 2 that nobody has either the time or motivation to fix, because all the interesting work is happening on Zope 3. Are there, or will there be, any guidelines for developing for Zope 2 with the view to migrating to Zope 3? And given the logevity of Zope 2 suggested by the roadmap, will there be any effort to clean up the Zope 2 code in the medium/long term? FWIW I use Zope for its webblication features over its CMF features, which is why Zope 3 is far more interesting than most of the work happening around Zope 2. I understand that things are moving quite well with Zope 3 but the current state of affairs does put places like ours, with a commitment to producing a large, complicated Zope 2 extranet/portal in a number of months, in a confusing headspace. a. -- Adrian van den Dries avdd@flow.com.au Development team www.dev.flow.com.au FLOW Communications Pty. Ltd. www.flow.com.au
I'll quote the (seemingly late) Andrew Kenneth Milton wrt Zope 2: http://www.zope.org/Members/mcdonc/Silly/newzopelogo ;-) On Fri, 2003-06-20 at 20:35, Adrian van den Dries wrote:
On June 20, Jamie Heilman wrote:
and I don't like surprises
Zope 2 is probably not the most suitable choice, then. ;-)
More seriously, though, my colleagues and I will often find a few of these sorts of *surprising* things in Zope 2 every week. It's really quite demoralising building for Zope 2 when it feels like such a dead effort, with many (seemingly rather deep) problems in Zope 2 that nobody has either the time or motivation to fix, because all the interesting work is happening on Zope 3.
Are there, or will there be, any guidelines for developing for Zope 2 with the view to migrating to Zope 3?
And given the logevity of Zope 2 suggested by the roadmap, will there be any effort to clean up the Zope 2 code in the medium/long term?
FWIW I use Zope for its webblication features over its CMF features, which is why Zope 3 is far more interesting than most of the work happening around Zope 2.
I understand that things are moving quite well with Zope 3 but the current state of affairs does put places like ours, with a commitment to producing a large, complicated Zope 2 extranet/portal in a number of months, in a confusing headspace.
a.
Jamie Heilman wrote:
(the patches I spoke of should be ready sometime tomorrowish assuming I don't run into any other funkyness, I'll post them to the collector)
These are done, there's a synopsis available at http://collector.zope.org/Zope/628 and the patches are available at http://audible.transient.net/zope/resized.diff and http://audible.transient.net/zope/sqlsized.diff they apply against the HEAD branch, and they do use python 2.2 constructs as I don't think the issue is really critical enough to warrant making the next 2.6 release. I'm currious what opinions are on the idea of joining the dtpref_* usage and the sql_pref__* usage together. Good idea? Bad idea? (I think its a good idea, but then I don't use zsql methods anyway so I kinda don't care either.) -- Jamie Heilman http://audible.transient.net/~jamie/ "I was in love once -- a Sinclair ZX-81. People said, "No, Holly, she's not for you." She was cheap, she was stupid and she wouldn't load -- well, not for me, anyway." -Holly
In article <20030620100717.GE8282@audible.transient.net> you write:
# Untrusted data *after* trusted data v = self.form.get(key, _marker) if v is not _marker: other[key] = v # *boom* return v
That magical promotion of the key & value to the other dictionary is what tripped me up.
Wouldn't other.setdefault(key, v) be better? So a variable already existing in other wouldn't get clobbered. Florent -- Florent Guillaume, Nuxeo (Paris, France) +33 1 40 33 79 87 http://nuxeo.com mailto:fg@nuxeo.com
participants (7)
-
Adrian van den Dries -
Chris McDonough -
Dieter Maurer -
Florent Guillaume -
Jamie Heilman -
Oliver Bleutgen -
Paul Winkler