[Zope-dev] Test fixture concepts (was: Zope test layers, pytest, and test isolation)

[Jim Fulton]

On Sun, Mar 27, 2011 at 11:13 AM, Martin Aspeli
<optilude+lists at gmail.com> wrote:
> Hi,
>
> On 27 March 2011 15:54, Uli Fouquet <uli at gnufix.de> wrote:
>
>> The (limited) experiences with py.test, however, were awesome. Some
>> points that are quite cool IMHO:
>>
>> - Easy finding of tests: just write some ``test_function`` in a
>>  ``test_module`` and it will be found and executed. That also makes
>>  py.test tests more readable and maybe more intuitive.
>
> I'm not sure this is always a good idea. It feels a bit implicit, and
> having a base class isn't really a big problem, IMHO. It seems a bit
> like the kind of thing that sounds cool (look, it's even easier!), but
> in practice makes little difference.

+1
+1
+1

This is especially important for doctests (and manuel) or any
situation where setup is important and where you can't really guess.

>> - py.test is more widespread in the Python community (that's my
>>  impression; I can't proof it)
>
> What about nose?

It looks to me like a layerish mechanism might be possible in nose, or
at least like
zope.testing layers could be integrated with nose.

>
>> - Support of unittest/unittest2: you can write standard lib setups
>>  (defining TestCases; no need to also write testsuite-setup stuff) and
>>  they will be found/executed. zope.testrunner for instance does not
>>  support the new `setUpClass`/`tearDownClass` concept of unittest2
>>  (yes, you would use layers in that case; but it might be nice if
>>  zope.testrunner  would support also class-wide fixtures in
>>  unittest2-style; people from other worlds might expect that to work).
>
> zope.testing should definitely gain support for the new unittest2
> hooks. That wouldn't be very hard, though. ;-)

I assume you mean zope.testrunner.

>
>> Main drawbacks I see on py.test side are:
>>
>> - Lack of layer support (yet). Maybe we can do something about that in
>>  `zope.pytest` based on `plone.testing.layer`.
>>
>> - Limited doctest support. It is quite difficult (AFAIK) to define
>>  fixtures for doctests or to even set the usual doctest options
>>  (``ELLIPSIS``, ``NORMALIZE_WHITESPACE``, ...) at setup time. Doctests
>>  are simply collected and executed and not much finetuning is possible.
>
> With zope.testrunner, you *do* need a test_suite method to run
> doctests. I think that's a good thing. Look at plone.testing's README
> for examples.

Again, +1

If I were to use nose or py.tests, I would want to adopt an explicit
style, which I believe is possible w nose.

...

> FWIW, I think we should stop using .txt doctests for unit tests.

I disagree, of course.

> Doctests should be used to test *documentation* ("the examples are
> valid").

Manuel is *much* better for that.  (Of course, manuel is arguably a form of
doctest.)


> For actual unit tests, writing tests in a unittest class is
> almost always better in the long run. doctests don't scale well and
> discourage the kind of ad-hoc "this seems broken, I'll just write a
> quick test" or "I just fixed a bug, better add a regression test"
> testing.

You're just not using them correctly. :)

One thing I hate about unittest is the javiotic ceremony it involves.
Doctests can cut down on the clutter a lot. I believe that that is
py.test's strength as well..

More generally, I'd love to see us adopt another test runner so that
we can stop maintianing zope.testrunner.  When it was written at
the turn of the century, there weren't good alternatives.  Personally,
I think maintaining it is boring.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton