[Zope-dev] Test fixture concepts (was: Zope test layers, pytest, and test isolation)

[Martin Aspeli]

Hi,

On 27 March 2011 15:54, Uli Fouquet <uli at gnufix.de> wrote:

> The (limited) experiences with py.test, however, were awesome. Some
> points that are quite cool IMHO:
>
> - Easy finding of tests: just write some ``test_function`` in a
>  ``test_module`` and it will be found and executed. That also makes
>  py.test tests more readable and maybe more intuitive.

I'm not sure this is always a good idea. It feels a bit implicit, and
having a base class isn't really a big problem, IMHO. It seems a bit
like the kind of thing that sounds cool (look, it's even easier!), but
in practice makes little difference.

> - Lots of setup code (unrelated to fixtures) can simply be skipped. No
>  need to do the ``testsuite = <complex-testcase-collecting>`` over and
>  over again. Maybe the main point of py.test.

You don't need that for zope.testrunner either, of course, at least
not when using unittest base classes.

> - py.test is more widespread in the Python community (that's my
>  impression; I can't proof it)

What about nose?

> - Support of unittest/unittest2: you can write standard lib setups
>  (defining TestCases; no need to also write testsuite-setup stuff) and
>  they will be found/executed. zope.testrunner for instance does not
>  support the new `setUpClass`/`tearDownClass` concept of unittest2
>  (yes, you would use layers in that case; but it might be nice if
>  zope.testrunner  would support also class-wide fixtures in
>  unittest2-style; people from other worlds might expect that to work).

zope.testing should definitely gain support for the new unittest2
hooks. That wouldn't be very hard, though. ;-)

> Main drawbacks I see on py.test side are:
>
> - Lack of layer support (yet). Maybe we can do something about that in
>  `zope.pytest` based on `plone.testing.layer`.
>
> - Limited doctest support. It is quite difficult (AFAIK) to define
>  fixtures for doctests or to even set the usual doctest options
>  (``ELLIPSIS``, ``NORMALIZE_WHITESPACE``, ...) at setup time. Doctests
>  are simply collected and executed and not much finetuning is possible.

With zope.testrunner, you *do* need a test_suite method to run
doctests. I think that's a good thing. Look at plone.testing's README
for examples.

> My very own concern about the latter point is: if this cannot be solved
> satisfactorily (in terms of readability and ease of use at least),
> py.test will not become a really hot candidate in the testing framework
> game on the Zope side at all. There are too many doctests used already
> and whether you like them or not: every testing framework used should
> offer at least minimal support for doctest fixtures and doctest options.
> But I might just have missed these features in py.test and they are
> provided already.

FWIW, I think we should stop using .txt doctests for unit tests.
Doctests should be used to test *documentation* ("the examples are
valid"). For actual unit tests, writing tests in a unittest class is
almost always better in the long run. doctests don't scale well and
discourage the kind of ad-hoc "this seems broken, I'll just write a
quick test" or "I just fixed a bug, better add a regression test"
testing.

> For now I think that there is absolutely no need to think about a
> general move to py.test for the ztk.

I think there's benefit in unifying the concepts and support for
concepts like layers so that people can use the test runner they
prefer.

Maritn