[Zope-dev] Why I dislike narrative doctests

23 Apr 2008

      Suppose I find a bug (say, zope.testing.testrunner.StartUpFailure
objects make unittest.TextTestRunner cry).  Being a good developer I
want to start the bug fix with a unit test.

Now if zope.testing used old-style isolated unit tests, I could open the
tests.py (or tests/test_testrunner.py) in a text editor and start
writing:

    class OptionsStub(object):
        post_mortem = False

    def doctest_StartUpFailure():
        """Test that StartUpFailure is a proper unittest.TestCase

        StartUpFailure is a TestCase-lookalike that is inserted into the
        test suite to indicate that a failed import of a test module
        won't be unnoticed.

            >>> from zope.testing.testrunner import StartUpFailure
            >>> options = OptionsStub()
            >>> module_name = 'test_me'
            >>> exc_info = (ImportError, ImportError('zonkulator'), None)
            >>> test = StartUpFailure(options, module_name, exc_info)
            >>> unittest.TextTestRunner(verbosity=2).run(test)

        This is a regression test for http://launchpad.net/bugs/221151
        """

There, I'm almost done.  Now I can fix the bug and paste the correct
output into the doctest.

But zope.testing uses narrative doctests instead of isolated unit tests.
There are 22 plain-text files that together comprise over 4500 lines of
text.  Now instead of mechanically opening the appropriate test module
and grepping for the class/function name I have to figure out which of
the txt files is the appropriate place for the new test.  After that I
have to figure out how to insert it into the narrative seamlessly.  Or
decide to create a new file---but a hundred of .txt files each 20-lines
long is not a good idea, in my opinion.

Suddenly I almost don't want to fix the bug any more.

While I was writing this, I suddenly realised that almost all
zope.testing's txt files are actually functional tests.  There are
nearly no unit tests there.  Perhaps that's what causing me pain.

I still maintain that:

  * adding a new unit test should be a trivial task

  * extending a narrative doctest is not a trivial task

  * the only way to solve this conflict is to explicitly separate unit
    tests (test_*.py) from doctest-enhanced API documentation (*.txt).

Marius Gedminas
-- 
UNIX is user friendly. It's just selective about who its friends are.