[Zope-dev] Testing Zope applications (was Re: [Zope-ZEO] Advice)

[Jim Fulton]

Note that this conversation hasn't had anything to do
with ZEO for some time, so I'm moving it over to zope-dev.

Toby Dickenson wrote:
> 
(snip)
> >I think it is really much easier to use ZPublisher/Test
> >(which is also available as Zope.debug:
> >
> >   import Zope
> >   Zope.debug(url)
> >
> >This provides a much thinner and more easily
> >controlled environment.
> 
> ... but not a complete replication of the deployed environment. For
> example, you cant test RESPONSE.write.

This sounds like a bug that needs to be fixed, not a limitation in the
approach. Would you mind submitting a collector item on it?

> From a philosophical point of view, using Zope.debug is wrong because
> the whole purpose is an integration test of the interactions of your
> pre-tested components with the Zope environment (depending on the
> complexity of your glue, you may also think of it as a unit test of
> that glue).

But Zope.debug uses the Zope environment.  Unless you consider ZServer
a critical part of the Zope environment.  It is extremely
rare for the difference between something like ZServer and
ZPublisher.Test to have any noticable impact on the application
behavior.
 
(snip)
> >> The key to this is (as always) ensuring that your design process
> >> considers the testability of the glue.
> >
> >What do you mean by glue?
> 
> The layer which allows Zope-ignorant code to be used as a Zope
> product. ZCatalog.py is an example of glue for Catalog.py.

I think that ZPublisher.Test provides a pretty good test harness
for anything below the publisher level, such as ZCatalog.

(snip)
> >How does this make anything untestable?  You certainly can decide
> >correct and incorrect behavior. I think there is much more possibility
> >of problems due to UI changes or dynamism that make analysis of
> >test results difficult.
> 
> I suspect our difference in opinion is the scope of what we are
> testing. Is it a bug if a product can be broken by a Manager using the
> management interface of an instance of a different product?
> 
> Should this 'bug' be tested for?
> 
> If so, then a test plan must verify that every dtml-namespace and
> every acquisition name lookup can never be subverted by an
> ObjectManager or PropertyManager instance, no matter how the instances
> are arranged in containers.
> 
> The alternative is that the Manager take responsibility for not
> creating any objects with a conflicting name - but who decides which
> names are disallowed. How do you test that this list is complete?

First, as an aside, you can't prove that non-trivial software 
is correct via testing. :)

Second, part of a test is deciding what the interface to the tested
software is. One way to limit the scope of testing is to limit the
promises made, for example to leave the behavior undefined under
some circumstances.
 
> I dont see a way to test this constraint, and it has proven impossible
> to avoid the problems using design rules. I recently checked some of
> our recent products using strategically placed debugging __getattr__
> hooks - with initially horrifying results.

It sounds like this is a problem that needs to be addressed.
It's not really a testing problem, but an architecture problem

I'm not sure exactly what problem you are refering to. It sound's
like an issue of depending on a specific acquired name and having
the name overridden with something bogus. Is that it?

There have been a couple of recent developments that I think could
help this:

  1. It's much easier than it used to be to name objects using 
     physical paths and converting the physical paths to objects.
     This would allow you to refer to a particular object using an 
     absolute reference.

  2. There will be a new interface in Zope 2.3 that will allow
     you to prevent a name from being used lower in a containement
     hierarchy. This change is documented at: 
     http://dev.zope.org/Wikis/DevSite/Proposals/ReplaceableProperty.
     The work has already been done and checked into CVS. I've asked Shane,
     the author, to update the interfaces wiki to capture this change.

Jim

--
Jim Fulton           mailto:jim@digicool.com   Python Powered!        
Technical Director   (888) 344-4332            http://www.python.org  
Digital Creations    http://www.digicool.com   http://www.zope.org    

Under US Code Title 47, Sec.227(b)(1)(C), Sec.227(a)(2)(B) This email
address may not be added to any commercial mail list with out my
permission.  Violation of my privacy with advertising or SPAM will
result in a suit for a MINIMUM of $500 damages/incident, $1500 for
repeats.