[Zope-dev] Re: zope.sendmail Retry fixes and new state machine.

[Marius Gedminas]

On Mon, Mar 10, 2008 at 06:09:50PM +0100, Wichert Akkerman wrote:
[...the usual doctest vs unittest discussion ...]
> Lack of isolation is a very convincing argument to me.

For me as well.  Which is why I put my doctests into a traditional
tests/test_foo.py module like this:

    # ... imports etc ...

    def doctest_FooClass_somemethod():
        """Test FooClass.somemethod

            >>> foo = FooClass(ContextStub())

        When you frob the groddle, by default you get zlotniks

            >>> foo.frob(GroddleStub())
            '42 zlotniks'

        """

    def doctest_FooClass_somemethod_special_case():
        """Test FooClass.somemethod

            >>> foo = FooClass(ContextStub())

        There's a very special corner case that happens when the system runs
        out of zlotniks

            >>> foo.frob(GroddleStub(zlotniks_required=1000))
            Traceback (most recent call last):
              ...
            OutOfZlotniksError: 1000 zlotniks required, but only 50 are available

        """

    def test_suite():
        return doctest.DocTestSuite(setUp=setup.placelessSetUp,
                                    tearDown=setup.placelessTearDown,
                                    optionflags=doctest.NORMALIZE_WHITESPACE)


I find that this style helps me write much more readable tests without
sacrificing isolation.

Instead I sacrifice pdb debuggability, that I don't value much (I find
the pdb user interface horrible and prefer debugging with print
statements).

I also sacrifice test reusability -- you can't invoke one doctest from
the middle of another one.  (Footnotes don't count -- invoking a doctest
footnote more than once is just *evil*, because you never know which
invocation failed.).

I find that for me personally, the advantages of test readability
outweigh the disadvantages:

  * doctests without English test explaining each block of related
    assertions look naked.  Unittests without comments look normal.  As
    a result I feel the urge to explain my doctests when I never felt
    the urge to explain my unit tests.  Don't know why that is, but
    that's how I feel.

  * The ability to nicely format data for presentation is extremely
    helpful.  Compare

        >>> print_table([view.column_titles] + view.getTabularData())
        +------------------+------------------+--------------+
        | Month            | Widgets produced | Widgets sold |
        +------------------+------------------+--------------+
        | January          | 42               | 17           |
        | February         | 33               | 25           |
        +------------------+------------------+--------------+

    with

        self.assertEquals(view.getTabularData(),
                          [['January', 42, 17],
                           ['February', 33, 25]])

    Now compare the error messages when the test fails:

        Failed example:
            print_table([view.column_titles] + view.getTabularData())
        Differences (ndiff with -expected +actual):
              +------------------+------------------+--------------+
              | Month            | Widgets produced | Widgets sold |
              +------------------+------------------+--------------+
            - | January          | 42               | 17           |
                                    ^
            + | January          | 43               | 17           |
                                    ^
              | February         | 33               | 25           |
              +------------------+------------------+--------------+

    versus

        AssertionError: [['January', 42, 17], ['February', 33, 25]] != [['January', 43, 17], ['February', 33, 25]]

    I'm not saying that it's impossible to write a better table
    comparison function and use that instead of assertEquals.  I'm
    saying that with doctests it's easier and feels more natural.

> Perhaps more personal taste but I also find python unittests to be much
> more readable. You don't suffer from mixing lots of test setup/teardown
> being repeated through the file. As Tres mentioned this is especially
> true when testing corner cases.

My solution for repeating setup/teardown:

  * Put things common to all tests in the module (like placelessSetUp,
    setup.setUpAnnotations, registering your custom adapters needed for
    these tests) into global setUp/tearDown functions, and pass those to
    the DocTestSuite constructor.

  * Extract object creation into global reusable helper functions.  Then tests
    look like

        >>> context = createFrombulator()
        >>> view = FrombulatorEditView(context, TestRequest())

    and you don't have to care in every test that your Frombulator needs
    seventeen Wodgets in a nested tree of Grompets for correct
    functioning.  These helper functions are parametrizable, so that if
    I find out that I need a very custom wodget in a particular test, I
    can do

        >>> my_wodget = Wodget('foo', 'bar')
        >>> my_wodget.someSetUpOnlyUsedInThisTest(42, -17j)
        >>> context = createFrombulator(main_wodget=my_wodget)
        ...

> Being able to debug tests by stepping over them with pdb is incredibly
> useful. With doctests that doesn't work.

Right.  In this case doctests aren't the right choice for you.

> Being able to run a single test easily allows for quick testing and
> debugging. I can't tell the testrunner 'start running at line 52 but
> also include all the test setup magic from before, but skip the rest'.

This is one of the reasons why I prefer doctest modules to long
documentation-and-all-tests-rolled-into-one .txt files, despite best
efforts of Stephan Richter to convince me otherwise.  ;-)

> With
> unittests I can simple run zopectl test -s <module> -t <test function>.

Heh.  I've got a vim macro that (1) finds the dotted name of the module
I'm editing and (2) finds the nearest function definition above the
cursor and (3) constructs a command line to run

  bin/test -s subdir -m package.module -t testnamee

Yummy.  Sadly, vim is completely incapable of sanely executing a
long-running (which I define as "longer than 2 seconds") shell process
in the background, so I have to alt-tab to a terminal window and paste
the constructed command line that my vim macro puts into the clipboard.

> doctests hurt my productivity badly.

I'm happy to say that's not the case for me.

Now, *functional* tests do hurt my productivity sometimes.  They also
happen to be long-running slow doctests that compare multi-thousand-line
HTML pages to ad-hoc patterns built with the doctest.ELLIPSIS feature.
Pain.  A recent decision to bite the bullet and introduce a dependency
on lxml so that we could compare just the interesting snippets of HTML,
extracted with xpath queries, helps somewhat.  The abysmal rate of ~6
zope.testbrowser requests per second on a modern 1.8 GHz Core 2 Duo
machine doesn't, but that's a different topic.

Marius Gedminas
-- 
I code in vi because I don't want to learn another OS. :)
                -- Robert Love
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080319/1b350096/attachment.bin