On Mon, Mar 10, 2008 at 06:09:50PM +0100, Wichert Akkerman wrote: [...the usual doctest vs unittest discussion ...]
Lack of isolation is a very convincing argument to me.
For me as well. Which is why I put my doctests into a traditional tests/test_foo.py module like this: # ... imports etc ... def doctest_FooClass_somemethod(): """Test FooClass.somemethod >>> foo = FooClass(ContextStub()) When you frob the groddle, by default you get zlotniks >>> foo.frob(GroddleStub()) '42 zlotniks' """ def doctest_FooClass_somemethod_special_case(): """Test FooClass.somemethod >>> foo = FooClass(ContextStub()) There's a very special corner case that happens when the system runs out of zlotniks >>> foo.frob(GroddleStub(zlotniks_required=1000)) Traceback (most recent call last): ... OutOfZlotniksError: 1000 zlotniks required, but only 50 are available """ def test_suite(): return doctest.DocTestSuite(setUp=setup.placelessSetUp, tearDown=setup.placelessTearDown, optionflags=doctest.NORMALIZE_WHITESPACE) I find that this style helps me write much more readable tests without sacrificing isolation. Instead I sacrifice pdb debuggability, that I don't value much (I find the pdb user interface horrible and prefer debugging with print statements). I also sacrifice test reusability -- you can't invoke one doctest from the middle of another one. (Footnotes don't count -- invoking a doctest footnote more than once is just *evil*, because you never know which invocation failed.). I find that for me personally, the advantages of test readability outweigh the disadvantages: * doctests without English test explaining each block of related assertions look naked. Unittests without comments look normal. As a result I feel the urge to explain my doctests when I never felt the urge to explain my unit tests. Don't know why that is, but that's how I feel. * The ability to nicely format data for presentation is extremely helpful. Compare >>> print_table([view.column_titles] + view.getTabularData()) +------------------+------------------+--------------+ | Month | Widgets produced | Widgets sold | +------------------+------------------+--------------+ | January | 42 | 17 | | February | 33 | 25 | +------------------+------------------+--------------+ with self.assertEquals(view.getTabularData(), [['January', 42, 17], ['February', 33, 25]]) Now compare the error messages when the test fails: Failed example: print_table([view.column_titles] + view.getTabularData()) Differences (ndiff with -expected +actual): +------------------+------------------+--------------+ | Month | Widgets produced | Widgets sold | +------------------+------------------+--------------+ - | January | 42 | 17 | ^ + | January | 43 | 17 | ^ | February | 33 | 25 | +------------------+------------------+--------------+ versus AssertionError: [['January', 42, 17], ['February', 33, 25]] != [['January', 43, 17], ['February', 33, 25]] I'm not saying that it's impossible to write a better table comparison function and use that instead of assertEquals. I'm saying that with doctests it's easier and feels more natural.
Perhaps more personal taste but I also find python unittests to be much more readable. You don't suffer from mixing lots of test setup/teardown being repeated through the file. As Tres mentioned this is especially true when testing corner cases.
My solution for repeating setup/teardown: * Put things common to all tests in the module (like placelessSetUp, setup.setUpAnnotations, registering your custom adapters needed for these tests) into global setUp/tearDown functions, and pass those to the DocTestSuite constructor. * Extract object creation into global reusable helper functions. Then tests look like >>> context = createFrombulator() >>> view = FrombulatorEditView(context, TestRequest()) and you don't have to care in every test that your Frombulator needs seventeen Wodgets in a nested tree of Grompets for correct functioning. These helper functions are parametrizable, so that if I find out that I need a very custom wodget in a particular test, I can do >>> my_wodget = Wodget('foo', 'bar') >>> my_wodget.someSetUpOnlyUsedInThisTest(42, -17j) >>> context = createFrombulator(main_wodget=my_wodget) ...
Being able to debug tests by stepping over them with pdb is incredibly useful. With doctests that doesn't work.
Right. In this case doctests aren't the right choice for you.
Being able to run a single test easily allows for quick testing and debugging. I can't tell the testrunner 'start running at line 52 but also include all the test setup magic from before, but skip the rest'.
This is one of the reasons why I prefer doctest modules to long documentation-and-all-tests-rolled-into-one .txt files, despite best efforts of Stephan Richter to convince me otherwise. ;-)
With unittests I can simple run zopectl test -s <module> -t <test function>.
Heh. I've got a vim macro that (1) finds the dotted name of the module I'm editing and (2) finds the nearest function definition above the cursor and (3) constructs a command line to run bin/test -s subdir -m package.module -t testnamee Yummy. Sadly, vim is completely incapable of sanely executing a long-running (which I define as "longer than 2 seconds") shell process in the background, so I have to alt-tab to a terminal window and paste the constructed command line that my vim macro puts into the clipboard.
doctests hurt my productivity badly.
I'm happy to say that's not the case for me. Now, *functional* tests do hurt my productivity sometimes. They also happen to be long-running slow doctests that compare multi-thousand-line HTML pages to ad-hoc patterns built with the doctest.ELLIPSIS feature. Pain. A recent decision to bite the bullet and introduce a dependency on lxml so that we could compare just the interesting snippets of HTML, extracted with xpath queries, helps somewhat. The abysmal rate of ~6 zope.testbrowser requests per second on a modern 1.8 GHz Core 2 Duo machine doesn't, but that's a different topic. Marius Gedminas -- I code in vi because I don't want to learn another OS. :) -- Robert Love