[Zope-dev] Test fixture concepts

[Jim Fulton]

On Mon, Mar 28, 2011 at 5:27 PM, Martin Aspeli <optilude+lists at gmail.com> wrote:
> On 28 March 2011 15:45, Tres Seaver <tseaver at palladion.com> wrote:
>> The vast majority of the doctest testcases in zope.* packages fall into
>> this category:  poor isolation, lots of edge cases which would obscure
>> any real narrative docs, of which there are almost none.  I believe the
>> conflict is intrinsic, here, and not just an accident of careless /
>> naive implementation:  exercising all the preconditions of the contract
>> of the function-under-test makes for really poor documentation, but is
>> essential to good testing.
>
> Agree. Here's what I've found and learned the hard way:
>
> doctests are sometimes easier / more fun to write the *first* time I
> write tests. It's easy to just start writing, and the narrative helps
> me think about what I'm doing.

Yup.

> Plus, it feels like I'm saving time on
> writing docs.

And it doesn't because tests != docs.

I've found that docs need to be written from an entirely different
point of view.

> They are much worse all subsequent times. Maintenance becomes a soup.

I've only found this to be the case if all tests are put in the same document.
Keep in mind that unit tests can be a maintenance disaster too.  I speak from
experience.

Some key factors in maintainability:

- Is the intent clear. Unit tests don't do anything to make intent clear.
  Doctests don't force you to make intent clear.

- Coupling between different tests. Arguably doctests encourage this by
  making it easy to glom many unrelated tests into the same document, but
  this can and does happen with unit tests as well, especially if a lot of
  setup is needed.

> Quick tests for edge cases feel like they obscure the narrative,

Yup, they do!

> so I may forgo them.

Which is a mistake. You should create separate tests.  I typically put
large tests, dealing with main use cases where there is a definite
flow of activity in '.test' files.  I do these in separate files
because they're easier to write that way.  I use a '.test' suffix to
avoid the pretense that these are documentation.  I put edge-case
tests in small docstrings in testing modules.  I'm not really
religious about using doctests for this, but I find small edge-case
doctests easier to read than traditional unit tests. It's possible
that I'd like py.test tests as much.

> Refactoring is painful.

It all depends on how well the tests are written and this applies
equally no matter what format is used.

> Tool support is poorer,

True

...

> And people find the docs
> underwhelming.

tests != docs

This was a lesson learned.

> If I'm doing it wrong, I'd like to see it done "right".

You might check out the bobo and zc.ngi docs and tests and let me know
what you think.

> Manuel is kind
> of cool, but I'm not sure it really addresses the issue. It's a better
> doctest, not a better unittest.

You may be missing the point.

When writing docs, the primary goal is to guide the user to
understanding as effectively as possible.  The focus is on the user
and their goals.  A secondary, but important goal is making sure what
you say in documentation is true.  Manuel helps with the second goal
without compromising the first goal.

When writing tests, the focus is on the software and assuring that it
is working correctly.
Manuel can help here too by making some things easier to express and
by being a better doctests, but the main goal of manuel is to enable
writing good documentation that is still executable.

> One of the main objections to unittest(2) seems to be that it's
> "Javaesque". Yes, JUnit was the first library to solidify many of the
> current, cross-language testing patterns. We shouldn't be so
> NIH-stricken that we can't see the benefit of being consistent with
> patterns used in most other languages and test frameworks these days.

It's not about NIH, it's about ceremony.  Java and it's culture are all about
ceremony that just get in the way. py.test and (modern uses of) doctest are
in many ways a reaction to that ceremony.

Jim

-- 
Jim Fulton
http://www.linkedin.com/in/jimfulton