Testing Zope applications (was Re: [Zope-ZEO] Advice)
Note that this conversation hasn't had anything to do with ZEO for some time, so I'm moving it over to zope-dev. Toby Dickenson wrote:
(snip)
I think it is really much easier to use ZPublisher/Test (which is also available as Zope.debug:
import Zope Zope.debug(url)
This provides a much thinner and more easily controlled environment.
... but not a complete replication of the deployed environment. For example, you cant test RESPONSE.write.
This sounds like a bug that needs to be fixed, not a limitation in the approach. Would you mind submitting a collector item on it?
From a philosophical point of view, using Zope.debug is wrong because the whole purpose is an integration test of the interactions of your pre-tested components with the Zope environment (depending on the complexity of your glue, you may also think of it as a unit test of that glue).
But Zope.debug uses the Zope environment. Unless you consider ZServer a critical part of the Zope environment. It is extremely rare for the difference between something like ZServer and ZPublisher.Test to have any noticable impact on the application behavior. (snip)
The key to this is (as always) ensuring that your design process considers the testability of the glue.
What do you mean by glue?
The layer which allows Zope-ignorant code to be used as a Zope product. ZCatalog.py is an example of glue for Catalog.py.
I think that ZPublisher.Test provides a pretty good test harness for anything below the publisher level, such as ZCatalog. (snip)
How does this make anything untestable? You certainly can decide correct and incorrect behavior. I think there is much more possibility of problems due to UI changes or dynamism that make analysis of test results difficult.
I suspect our difference in opinion is the scope of what we are testing. Is it a bug if a product can be broken by a Manager using the management interface of an instance of a different product?
Should this 'bug' be tested for?
If so, then a test plan must verify that every dtml-namespace and every acquisition name lookup can never be subverted by an ObjectManager or PropertyManager instance, no matter how the instances are arranged in containers.
The alternative is that the Manager take responsibility for not creating any objects with a conflicting name - but who decides which names are disallowed. How do you test that this list is complete?
First, as an aside, you can't prove that non-trivial software is correct via testing. :) Second, part of a test is deciding what the interface to the tested software is. One way to limit the scope of testing is to limit the promises made, for example to leave the behavior undefined under some circumstances.
I dont see a way to test this constraint, and it has proven impossible to avoid the problems using design rules. I recently checked some of our recent products using strategically placed debugging __getattr__ hooks - with initially horrifying results.
It sounds like this is a problem that needs to be addressed. It's not really a testing problem, but an architecture problem I'm not sure exactly what problem you are refering to. It sound's like an issue of depending on a specific acquired name and having the name overridden with something bogus. Is that it? There have been a couple of recent developments that I think could help this: 1. It's much easier than it used to be to name objects using physical paths and converting the physical paths to objects. This would allow you to refer to a particular object using an absolute reference. 2. There will be a new interface in Zope 2.3 that will allow you to prevent a name from being used lower in a containement hierarchy. This change is documented at: http://dev.zope.org/Wikis/DevSite/Proposals/ReplaceableProperty. The work has already been done and checked into CVS. I've asked Shane, the author, to update the interfaces wiki to capture this change. Jim -- Jim Fulton mailto:jim@digicool.com Python Powered! Technical Director (888) 344-4332 http://www.python.org Digital Creations http://www.digicool.com http://www.zope.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
On Mon, 25 Sep 2000 12:41:21 -0400, Jim Fulton <jim@digicool.com> wrote:
But Zope.debug uses the Zope environment. Unless you consider ZServer a critical part of the Zope environment. It is extremely rare for the difference between something like ZServer and ZPublisher.Test to have any noticable impact on the application behavior.
This skepticism has served me well; I have already submitted several reports & patches for bugs in Zope's ZServer.HTTPResponse.write implementation. Having said that, I do agree that for most applications this difference is not critical. [much snippage; I agree with it all]
I dont see a way to test this constraint, and it has proven impossible to avoid the problems using design rules. I recently checked some of our recent products using strategically placed debugging __getattr__ hooks - with initially horrifying results.
It sounds like this is a problem that needs to be addressed. It's not really a testing problem, but an architecture problem
Agreed. Don't all testability problems originate with architecture problems? (apart from the 'I cant be bothered; testing is boring' problem ;-)
I'm not sure exactly what problem you are refering to. It sound's like an issue of depending on a specific acquired name and having the name overridden with something bogus. Is that it?
There are two related issues that conspire to make the problem hard: A. What you descibe above, that looks like it will be fixed in part by NO_SUBOBJECTS_OVERRIDE (which looks great). The outstanding issue is what happens when a new version of a product wants to add a new NO_SUBOBJECTS_OVERRIDE name (when objects of that name may already exist in old subobjects). In many cases these to-be-protected names are part of a product's API. http://dev.zope.org/Wikis/DevSite/Proposals/APINamingConvention suggests a naming convention for new API attributes. I suggested in the discussion Wiki that it should not be possible to create subobjects with an id of the form reserved for API attributes - but it seems to have got lost in the Wiki noise :-(
2. There will be a new interface in Zope 2.3 that will allow you to prevent a name from being used lower in a containement hierarchy. This change is documented at: http://dev.zope.org/Wikis/DevSite/Proposals/ReplaceableProperty. The work has already been done and checked into CVS. I've asked Shane, the author, to update the interfaces wiki to capture this change.
B. A common mistake in DTML authoring is to look up a name in the DTML namespace when the context expected to contain the name is not at the top of the namespace stack. The bug goes undetected if the context at the top of the stack does not contain that name. I have found that this happens too frequently even for skilled authors, and is rarely detected in review. In summary: one namespace isnt enough. I suspect that this is the problem percieved by Andrew Kuchling (who started this thread back on zope-zeo). Our solution to this (successful so far) has been to replace DTML in any use other than Document Templates. (For me, the greatest horror is the name 'DTML Method'. The name 'method' implies a tight coupling to the instance that it is a method of, however it is possible for a DTML Method to execute in a context where all of it's instances attributes are masked by those of other objects. Urgh) Toby Dickenson tdickenson@geminidataloggers.com
Toby Dickenson wrote:
(snip)
I'm not sure exactly what problem you are refering to. It sound's like an issue of depending on a specific acquired name and having the name overridden with something bogus. Is that it?
There are two related issues that conspire to make the problem hard:
A. What you descibe above, that looks like it will be fixed in part by NO_SUBOBJECTS_OVERRIDE (which looks great). The outstanding issue is what happens when a new version of a product wants to add a new NO_SUBOBJECTS_OVERRIDE name (when objects of that name may already exist in old subobjects).
I think that there should be some discussion of this design pattern. Specifically, I'm not sure I like the idea of an application that depends on fixed names in a hierarchy. In fact, I know I don't. :) I think it would be a good to have a discussion of this pattern. In particular, I'd like to see: - Problems it solves - Alternative approaches *I* would like to see this discussion happen in a Wiki, but I won't insist. :) FWIW, it will be much more likely for me to make comments in a wiki, especially if someone sends me the wiki link when they are ready for comments.
In many cases these to-be-protected names are part of a product's API. http://dev.zope.org/Wikis/DevSite/Proposals/APINamingConvention suggests a naming convention for new API attributes. I suggested in the discussion Wiki that it should not be possible to create subobjects with an id of the form reserved for API attributes - but it seems to have got lost in the Wiki noise :-(
I thought, until a couple of days ago, that this proposal had been rejected. I found out recently that it hadn't. I'll look at your comments when I move the proposal forward. <aside> On Wikis, while they need improvement (see http://dev.zope.org/Wikis/DevSite/Proposals/WikiNG) I personally find them far less noisy than mail lists. Hopefully, we'll make the wiki's provide the best of both worlds soon. </aside>
2. There will be a new interface in Zope 2.3 that will allow you to prevent a name from being used lower in a containement hierarchy. This change is documented at: http://dev.zope.org/Wikis/DevSite/Proposals/ReplaceableProperty. The work has already been done and checked into CVS. I've asked Shane, the author, to update the interfaces wiki to capture this change.
B. A common mistake in DTML authoring is to look up a name in the DTML namespace when the context expected to contain the name is not at the top of the namespace stack. The bug goes undetected if the context at the top of the stack does not contain that name.
But if the name isn't found in the inner (aka top ;) namespace, then the search should proceed further in.
I have found that this happens too frequently even for skilled authors, and is rarely detected in review. In summary: one namespace isnt enough.
I suspect that this is the problem percieved by Andrew Kuchling (who started this thread back on zope-zeo). Our solution to this (successful so far) has been to replace DTML in any use other than Document Templates.
I think that there is broad agreement that DTML should be uses soley for templates and shouldn't contain complex logic.
(For me, the greatest horror is the name 'DTML Method'. The name 'method' implies a tight coupling to the instance that it is a method of, however it is possible for a DTML Method to execute in a context where all of it's instances attributes are masked by those of other objects. Urgh)
This is a great point. The intent was that DTML Methods would be bound most tightly to the instances they were accessed in, but this is not what's done. Jim -- Jim Fulton mailto:jim@digicool.com Technical Director (888) 344-4332 Python Powered! Digital Creations http://www.digicool.com http://www.python.org Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email address may not be added to any commercial mail list with out my permission. Violation of my privacy with advertising or SPAM will result in a suit for a MINIMUM of $500 damages/incident, $1500 for repeats.
On Tue, 26 Sep 2000 07:42:29 -0400, Jim Fulton <jim@digicool.com> wrote:
Toby Dickenson wrote:
I'm not sure exactly what problem you are refering to. It sound's like an issue of depending on a specific acquired name and having the name overridden with something bogus. Is that it?
There are two related issues that conspire to make the problem hard:
A. What you descibe above, that looks like it will be fixed in part by NO_SUBOBJECTS_OVERRIDE (which looks great). The outstanding issue is what happens when a new version of a product wants to add a new NO_SUBOBJECTS_OVERRIDE name (when objects of that name may already exist in old subobjects).
I think that there should be some discussion of this design pattern. Specifically, I'm not sure I like the idea of an application that depends on fixed names in a hierarchy. In fact, I know I don't. :)
The technique isnt nice, but its hardly unconventional. the REQUEST object (as in self.REQUEST) is a good example.
*I* would like to see this discussion happen in a Wiki, but I won't insist. :) FWIW, it will be much more likely for me to make comments in a wiki, especially if someone sends me the wiki link when they are ready for comments.
I start something later this week. Toby Dickenson tdickenson@geminidataloggers.com
[on the subject of dtml picking the wrong context to look in first]
I dont see a way to test this constraint, and it has proven impossible to avoid the problems using design rules. I recently checked some of our recent products using strategically placed debugging __getattr__ hooks - with initially horrifying results.
Try applying this patch, which highlights every instance where it may be possible to subvert the behaviour of a Zope application by adding a carefully named property or subobject to the root folder. A few minutes browsing through the management interface picked up over 200 incidents, listed at http://www.zope.org/Members/htrd/names Many of these may be innocuous (and most are variations on a theme), however I am sure that many are undiscovered bugs. *** Application.py 2000/07/21 09:45:37 1.1.1.3 --- Application.py 2000/09/26 14:36:04 *************** *** 194,201 **** --- 194,207 ---- def title_and_id(self): return self.title def title_or_id(self): return self.title + def __getattr__(self,id,reg={}): + if not reg.has_key(id): + print `id` + reg[id]=id + raise AttributeError(id) + def __init__(self): # Initialize users uf=UserFolder() self.__allow_groups__=uf Toby Dickenson tdickenson@geminidataloggers.com
participants (2)
-
Jim Fulton -
Toby Dickenson