[Zope-dev] Test fixture concepts (was: Zope test layers, pytest, and test isolation)

[Wolfgang Schnerring]

Hello,

part two. :)

* Uli Fouquet <uli at gnufix.de> [2011-03-24 01:05]:
> A big advantage of test layers over `pytest` testing scopes might be
> that you can spread your tests associated to a certain layer over many
> files/modules/packages as you like and the setup/teardown will
> nevertheless only be performed once for each layer
> 
> Compared to Zope test layers I came to the conclusion that there is not
> much like this concept already in `pytest` and the behaviour of test
> layers can not easily be faked.

I fully agree, the test layer concept of zope.testrunner provides some
very nice properties, most notably:

1) Shared fixture setup. It is shared across many TestCases and
created only once per test run. (And it is independent of how the
tests are organized into modules or whatever.)

2) Stacking of fixture setups. My canonical example: First, we
initialize the application, load ZCML, prepare databases, etc. and
talk to it with zope.testbrowser. That's the first layer. Then we add
a HTTP port with gocept.selenium on another layer and do the rest with
a real browser -- but the whole, possibly expensive, application setup
*does not have to be done again*.

3) Orthogonal composition, i.e. you can have small, well defined
fixture units that you can then combine in a separate step to form the
actual setup for your tests. This enables for example that library
packages like zope.component could provide test infrastructure for
themselves, and application packages could pull all the test
infrastructure together that they need -- not like zope.app.testing
for example, which is a giant monolith that sets up everything at
once.

I think that these properties are very valuable, and I haven't seen them
anywhere else so far. I haven't looked at py.test closely, yet, but
from what I've seen, and what you wrote, it doesn't seem to offer
these things right now.

> Would it make sense to bring the Zope layer concept into `pytest`?
> 
> Are there already possibilities to mimic testlayer-like behaviour in
> `pytest` which we simply overlooked?

An honest counter question: Why not use zope.testrunner? What
advantages does py.test offer?

But your email brings into focus a question that I've been thinking
about lately: What's the future direction for test fixture setup?

Because, while test layers are nice because they have the above
properties, I'm not too happy with the current implementation, namely
the use (or is it abuse? ;-) of __bases__ and __name__, which leads
very naturally to not-so-helpful implementation patterns. As a
concrete example, some of the test layers of ZTK packages, most
notably the successors of zope.app.testing.functional in
zope.component, zope.app.appsetup, zope.app.wsgi, are implemented
right now in a way that does not have properties 2+3.

The most straightforward way to improve that situation would be to use
plone.testing, since that provides a layer implementation that is both
sane and easy to use, and has all the nice properties. (Maybe even
fold some of it into zope.testrunner itself.)

But then I realized, that's a major undertaking, so we better think
about what the actual goals are before changing lots of stuff. And
then I heard people expressing interest in other test runners.
(py.test and nose seem to be the biggest players, stdlib's
unittest/unittest2 seems to be woefully behind in terms of test
selection and other features like coverage or debugging. I for one
happen to like zope.testrunner, but if there's a good alternative, why
not combine our efforts?)
I also found that it is rather hard to explain how to use test layers
in a way so you actually get the above properties, so I started
wondering whether there was a cleaner way to get them.

So I guess I have three questions:

1. (The starting point:) How can we improve the test infrastructure
situation of the ZTK packages?

2. (The main thing:) Are test layers, both the concept and the
implementation as epitomized by plone.testing, the way to go or is
there a cleaner alternative?

3. (But only tangentially, I don't want to sidetrack the discussion at
hand nor start a flamewar:) Is there a compelling alternative to
zope.testrunner that would make it worthwile to think about either
generalizing the fixture support or even migrating away from
zope.testrunner?

Wolfgang

-- 
Wolfgang Schnerring · ws at gocept.com
gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany
http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1
Zope and Plone consulting and development