Previously Tres Seaver wrote:

Gary Poster wrote:

...

I did a five-minute skim of the checkin but hope to look a bit more tomorrow. Hopefully Marius, Benji, Albertas, or someone else who has actually done work on this package will take a look and chime in.

I did have one somewhat trivial thought. I generally prefer durations and intervals expressed as datetime.timedeltas myself, because they convey their meaning without having to look it up docs (is that number value a number of seconds? milliseconds? minutes?). There might even be a zcml built in for schema field for that; I believe I remember that there is in ZConfig.

Also, some variety of doctest would be nice. Even when a package is not using doctests, I add new tests as doctest unless there's a really good reason not to.

Becuase they make for poor unit tests? Using them to document the "mainline" use cases for an API is one thing: using them to do thorough coverage of edge cases is quite another. I find that for the latter, they fail *both* as documentation *and* as tests: their value as documentation drops as the amount of scaffoldiing rises, and the lack of isolation between tests sharply reduces their value for testing the corners.

I realize I have said this before, but then others keep urging the "doctests everywhere" meme.

Indeed, and for that reason this can't be said enough. Doctests are useful to create testable documentation. They are not the right tool to create isolated, debuggable tests. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.

Previously Benji York wrote:

Wichert Akkerman wrote:

...

Previously Tres Seaver wrote:

...

I realize I have said this before, but then others keep urging the "doctests everywhere" meme.

Indeed, and for that reason this can't be said enough. Doctests are useful to create testable documentation. They are not the right tool to create isolated, debuggable tests.

Not everyone agrees with this assertion. I for one vastly prefer both to read and write "unity" doctests over classic unit tests. And as "doctests everywhere" is the current status quo for just about the entire code base, I would expect a very convincing argument to be required to change.

The fact that something is popular does not necessarily mean it is the right thing :) Lack of isolation is a very convincing argument to me. Perhaps more personal taste but I also find python unittests to be much more readable. You don't suffer from mixing lots of test setup/teardown being repeated through the file. As Tres mentioned this is especially true when testing corner cases. Being able to debug tests by stepping over them with pdb is incredibly useful. With doctests that doesn't work. Being able to run a single test easily allows for quick testing and debugging. I can't tell the testrunner 'start running at line 52 but also include all the test setup magic from before, but skip the rest'. With unittests I can simple run zopectl test -s <module> -t <test function>. doctests hurt my productivity badly. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.

On Monday 10 March 2008, Martijn Pieters wrote:

I completely agree with Tres' and Wichert's statements on this. I only use doctests where they actually would make sense as documentation, the corner cases I always write as unit tests. The tools for dealing with pure Python code are so much more powerful than for python-embedded-in-text-with-prefixes, as well.

In my opinion it is as necessary to describe corner cases with words. In fact, corner cases often need even more documentation, because they are not obvious. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"

Wichert Akkerman wrote:

The fact that something is popular does not necessarily mean it is the right thing :)

Very true. It does, however, mean that you have to have a convincing argument to change popular opinion.

Lack of isolation is a very convincing argument to me.

This almost never bites me (if I'm understanding you correctly).

Perhaps more personal taste but I also find python unittests to be much more readable. You don't suffer from mixing lots of test setup/teardown being repeated through the file. As Tres mentioned this is especially true when testing corner cases.

Footnotes and good test structure alleviate the need for in-line setup/teardown.

Being able to debug tests by stepping over them with pdb is incredibly useful. With doctests that doesn't work.

As a heavy pdb user, this rarely bothers me, but seems to be more of an implementation issue than an argument that doctests are bad for unit tests.

Being able to run a single test easily allows for quick testing and debugging. I can't tell the testrunner 'start running at line 52 but also include all the test setup magic from before, but skip the rest'. With unittests I can simple run zopectl test -s <module> -t <test function>.

I don't see a huge advantage over bin/test -s <package> -t <doctest file name>

doctests hurt my productivity badly.

<shrug> I guess I'm the odd-man-out. -- Benji York Senior Software Engineer Zope Corporation

On Mon, Mar 10, 2008 at 06:09:50PM +0100, Wichert Akkerman wrote: [...the usual doctest vs unittest discussion ...]

Lack of isolation is a very convincing argument to me.

For me as well. Which is why I put my doctests into a traditional tests/test_foo.py module like this: # ... imports etc ... def doctest_FooClass_somemethod(): """Test FooClass.somemethod >>> foo = FooClass(ContextStub()) When you frob the groddle, by default you get zlotniks >>> foo.frob(GroddleStub()) '42 zlotniks' """ def doctest_FooClass_somemethod_special_case(): """Test FooClass.somemethod >>> foo = FooClass(ContextStub()) There's a very special corner case that happens when the system runs out of zlotniks >>> foo.frob(GroddleStub(zlotniks_required=1000)) Traceback (most recent call last): ... OutOfZlotniksError: 1000 zlotniks required, but only 50 are available """ def test_suite(): return doctest.DocTestSuite(setUp=setup.placelessSetUp, tearDown=setup.placelessTearDown, optionflags=doctest.NORMALIZE_WHITESPACE) I find that this style helps me write much more readable tests without sacrificing isolation. Instead I sacrifice pdb debuggability, that I don't value much (I find the pdb user interface horrible and prefer debugging with print statements). I also sacrifice test reusability -- you can't invoke one doctest from the middle of another one. (Footnotes don't count -- invoking a doctest footnote more than once is just *evil*, because you never know which invocation failed.). I find that for me personally, the advantages of test readability outweigh the disadvantages: * doctests without English test explaining each block of related assertions look naked. Unittests without comments look normal. As a result I feel the urge to explain my doctests when I never felt the urge to explain my unit tests. Don't know why that is, but that's how I feel. * The ability to nicely format data for presentation is extremely helpful. Compare >>> print_table([view.column_titles] + view.getTabularData()) +------------------+------------------+--------------+ | Month | Widgets produced | Widgets sold | +------------------+------------------+--------------+ | January | 42 | 17 | | February | 33 | 25 | +------------------+------------------+--------------+ with self.assertEquals(view.getTabularData(), [['January', 42, 17], ['February', 33, 25]]) Now compare the error messages when the test fails: Failed example: print_table([view.column_titles] + view.getTabularData()) Differences (ndiff with -expected +actual): +------------------+------------------+--------------+ | Month | Widgets produced | Widgets sold | +------------------+------------------+--------------+ - | January | 42 | 17 | ^ + | January | 43 | 17 | ^ | February | 33 | 25 | +------------------+------------------+--------------+ versus AssertionError: [['January', 42, 17], ['February', 33, 25]] != [['January', 43, 17], ['February', 33, 25]] I'm not saying that it's impossible to write a better table comparison function and use that instead of assertEquals. I'm saying that with doctests it's easier and feels more natural.

Perhaps more personal taste but I also find python unittests to be much more readable. You don't suffer from mixing lots of test setup/teardown being repeated through the file. As Tres mentioned this is especially true when testing corner cases.

My solution for repeating setup/teardown: * Put things common to all tests in the module (like placelessSetUp, setup.setUpAnnotations, registering your custom adapters needed for these tests) into global setUp/tearDown functions, and pass those to the DocTestSuite constructor. * Extract object creation into global reusable helper functions. Then tests look like >>> context = createFrombulator() >>> view = FrombulatorEditView(context, TestRequest()) and you don't have to care in every test that your Frombulator needs seventeen Wodgets in a nested tree of Grompets for correct functioning. These helper functions are parametrizable, so that if I find out that I need a very custom wodget in a particular test, I can do >>> my_wodget = Wodget('foo', 'bar') >>> my_wodget.someSetUpOnlyUsedInThisTest(42, -17j) >>> context = createFrombulator(main_wodget=my_wodget) ...

Being able to debug tests by stepping over them with pdb is incredibly useful. With doctests that doesn't work.

Right. In this case doctests aren't the right choice for you.

Being able to run a single test easily allows for quick testing and debugging. I can't tell the testrunner 'start running at line 52 but also include all the test setup magic from before, but skip the rest'.

This is one of the reasons why I prefer doctest modules to long documentation-and-all-tests-rolled-into-one .txt files, despite best efforts of Stephan Richter to convince me otherwise. ;-)

With unittests I can simple run zopectl test -s <module> -t <test function>.

Heh. I've got a vim macro that (1) finds the dotted name of the module I'm editing and (2) finds the nearest function definition above the cursor and (3) constructs a command line to run bin/test -s subdir -m package.module -t testnamee Yummy. Sadly, vim is completely incapable of sanely executing a long-running (which I define as "longer than 2 seconds") shell process in the background, so I have to alt-tab to a terminal window and paste the constructed command line that my vim macro puts into the clipboard.

doctests hurt my productivity badly.

I'm happy to say that's not the case for me. Now, *functional* tests do hurt my productivity sometimes. They also happen to be long-running slow doctests that compare multi-thousand-line HTML pages to ad-hoc patterns built with the doctest.ELLIPSIS feature. Pain. A recent decision to bite the bullet and introduce a dependency on lxml so that we could compare just the interesting snippets of HTML, extracted with xpath queries, helps somewhat. The abysmal rate of ~6 zope.testbrowser requests per second on a modern 1.8 GHz Core 2 Duo machine doesn't, but that's a different topic. Marius Gedminas -- I code in vi because I don't want to learn another OS. :) -- Robert Love

Dieter Maurer wrote:

Looks at the ZODB tests. They make heavy use of multiple inheritance to compose test classes out of component test classes

Good point; the ZODB tests make a great argument for using doctest over unittest.

and use the same tests e.g. for "FileStorage" as well as for "ZEO" tests.

I've done similar things with doctests. It's a great idea that transcends any particular testing tool. -- Benji York Senior Software Engineer Zope Corporation

On Mar 10, 2008, at 17:57 , Wichert Akkerman wrote:

Previously Tres Seaver wrote:

...

Gary Poster wrote:

...

Also, some variety of doctest would be nice. Even when a package is not using doctests, I add new tests as doctest unless there's a really good reason not to.

Becuase they make for poor unit tests? Using them to document the "mainline" use cases for an API is one thing: using them to do thorough coverage of edge cases is quite another. I find that for the latter, they fail *both* as documentation *and* as tests: their value as documentation drops as the amount of scaffoldiing rises, and the lack of isolation between tests sharply reduces their value for testing the corners.

I realize I have said this before, but then others keep urging the "doctests everywhere" meme.

Indeed, and for that reason this can't be said enough. Doctests are useful to create testable documentation. They are not the right tool to create isolated, debuggable tests.

Amen to that. Doctests are a useful testing tool, but no single tool can fit every situation. jens

--On 10. März 2008 22:33:13 -0500 Stephan Richter <srichter@cosmos.phy.tufts.edu> wrote:

On Monday 10 March 2008, Wichert Akkerman wrote:

...

Indeed, and for that reason this can't be said enough. Doctests are useful to create testable documentation. They are not the right tool to create isolated, debuggable tests.

Huh? I totally disagree. If you cannot explain your software in words, you do not understand it. As there are different kinds of programmatic tests, you can easily write different levels of doctests that have different audiences.

This sounds like writing doctests just for sake of having doctests for all and everything. I completely disagree with that. In complex algorithms edgecases are often only of interest for the person implementing the code in order for having a safety belt. For writing good doctests you have to be a good (English) writer at some point. Doctests come into the game if you want to tell other people about the usage of your module. It's basically not of interest for telling them all edgecases. Addressing edgecase is unittests is basically good enough for me. They sometime require more code to test and put this into a doctest is often just overhead. Andreas

--On 11. März 2008 09:42:11 -0300 David Pratt <fairwinds@eastlink.ca> wrote:

Hi Andreas. I think your response gets to the heart of the issue. For software to be useful, it is often more important for folks other than the author to understand it. This can only occur with communication. Sometimes it is the understanding of edgecases in particular, that gets lost over time. I find generally that I cannot keep in memory all of the details of the code I write. Several months afterwards without my own documentation, these details can be lost. So it is just important if I need to come back to it myself as for anyone else.

You can also document your edgecase in the same way as comments within your unittests. It matters that you have tests for the related edgecases. It's up to every developer to decide which test type is best for the individual case. If you're good in story telling (I am not so good), go with doctest but there can not be law that everything must be in doctests.

If the test is worth writing, it is not difficult to add a small amount of text to communicate about it. If the software is worth writing in the first place, I consider the code incomplete without doctests. For community projects, it is often the case that it is not the author maintaining the code in the future. This only strengthens the argument for doctests and the reason this has become the standard for zope.

Agreed. But unittests are for certain tests more suitable. E.g. if you need a complex and bigger fixture or something like that. Andreas

Jim Fulton wrote at 2008-3-11 09:58 -0400:

On Mar 11, 2008, at 9:46 AM, Andreas Jung wrote: ...

...

You can also document your edgecase in the same way as comments within your unittests.

This is true in theory, but in practice, it almost never happens.

There is no guarantee that doctests will contain useful text, but they encourage text which has a greater chance of being useful than no text at all.

"DocTests" essentially provide examples. I find documentation that only consists of examples very bad documentation -- not concise, at the wrong abstraction level, expecting me to induce the concepts from the examples rather than to describe the concepts and *then* (maybe) make some examples. Faced with the Zope 3/inspect incompatibility ("https://bugs.launchpad.net/zope3/+bug/181371") I have hit such a bad documentation: I have two classes "C1" and "C2", "C2" inherits from "C1". "C1" magically gets an attribute "a", but "C2" does not. Moreover, "C2.a" raises an AttributeError. There is a DocTest, that shows me that "C2.a" should indeed raise an "AttributeError". Thus, I can see that the author had something in mind when he implemented this counter intuitive behaviour. But, there is nothing that tells me what this may have been and why "C1" is treated differently from "C2". Documentation and tests have different purposes: Documentation tells relevant facts to the reader. It should be on an adequate abstraction level. It should clearly describe the concepts and include examples at places where this is helpful to unterstand the concepts. It should not consist of examples only and leave the concepts undescribed. The primary purpose of tests it to ensure that there are few undetected bugs. Some tests need to go to the same detail level as the implementation itself in order to probe for bugs introduced by implementation edgecases -- a level too detailed for a good documentation (apart from source code documentation itself). If a good test coveradge it obtained by performing the same tests over and over again on different data sets, representing relevant uses cases, the better. I do not need and I do not want essentially the same documentation for each of these sets. The ZODB test suite makes very good use of this principle, executing e.g. the "ConflictResolution" tests on both "FileStorage" and "ZEOStorage". The documentation is concise and clear: class FileStorageTests(..., ConflictResolutionTests, ...): ... By the way, a good documentation uses the same principle. It simply states "a 'FileStorage' supports conflict resolution (see XXXX for a description of the 'conflict resolution concept)" and does not show examples for conflict resolution for each an every conflict resolving storage. -- Dieter

Andreas Jung wrote at 2008-3-11 06:43 +0100:

... This sounds like writing doctests just for sake of having doctests for all and everything. I completely disagree with that. In complex algorithms edgecases are often only of interest for the person implementing the code in order for having a safety belt. For writing good doctests you have to be a good (English) writer at some point. Doctests come into the game if you want to tell other people about the usage of your module. It's basically not of interest for telling them all edgecases. Addressing edgecase is unittests is basically good enough for me. They sometime require more code to test and put this into a doctest is often just overhead.

+1 -- Dieter

Gary Poster wrote:

Perhaps your argument, Tres, is that the effort outweighs the cost in this particular case. That's a more reasonable argument to me. Maybe the scaffolding will be arduous.

I can certainly get on board with that argument.. I, myself, have added small unit tests to zope.sendmail. At the time I contemplated converting the tests to doctest, but it looked too painful because I didn't understand the meaning behind the assertions being made in the current tests. -- Benji York Senior Software Engineer Zope Corporation

On Tue, Mar 11, 2008 at 09:25:59AM -0400, Jim Fulton wrote:

It is certainly straightforward enough to create isolated doctests. For edge cases, I do typically create separate isolated short doctests that deal just with that case. The assertion that doctests don't allow isolation is simply not correct.

Can you think of a package offhand that I might look at to see a good example of this? I've got a lot of overgrown, brittle doctests at work that need gradual reorganization, and I'd love to see some good role models. -- Paul Winkler http://www.slinkp.com

On Thursday 13 March 2008, Paul Winkler wrote:

On Tue, Mar 11, 2008 at 09:25:59AM -0400, Jim Fulton wrote:

...

It is certainly straightforward enough to create isolated doctests.  For edge cases, I do typically create separate isolated short doctests that deal just with that case.  The assertion that doctests don't allow isolation is simply not correct.

Can you think of a package offhand that I might look at to see a good example of this?

I am very happy about my tests in z3c.form. They demonstrate tests with different audiences in mind. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"

On Tue, Mar 11, 2008 at 09:25:59AM -0400, Jim Fulton wrote:

I'm not sure what scaffolding you're referring to. Do you mean test set up? Or the tools for normalizing output? For test set up, I don't think there's any difference. I can sympathize with the skepticism with normalizing output. Sometimes, I think it might be better to write documented tests rather than executable documentation and use assertion of a more traditional nature, if the test is still readable, although I suspect that it will often be easier to normalize output rather than make lots of assertions about the output.

My personal experience with doctest output normalizers (as used in zope.testing's unit tests) was very painful. To quote jwz: Some people, when confronted with a problem, think "I know, I’ll use regular expressions." Now they have two problems Normalizing output might be the easier way when you initially write the test. But when some time later your test starts failing, they're a nightmare to debug. Marius Gedminas -- Any sufficiently advanced technology is indistinguishable from a rigged demo. - Andy Finkel, computer guy

Hi! On Sun, 2008-03-09 at 20:54 -0400, Gary Poster wrote:

I did have one somewhat trivial thought. I generally prefer durations and intervals expressed as datetime.timedeltas myself, because they convey their meaning without having to look it up docs (is that number value a number of seconds? milliseconds? minutes?). There might even be a zcml built in for schema field for that; I believe I remember that there is in ZConfig.

Minor detail - yes it has warts - there are other things that will need doing to, its just time that this code was put out there.

Also, some variety of doctest would be nice. Even when a package is not using doctests, I add new tests as doctest unless there's a really good reason not to.

It was the only way I could debug and introspect the problems from the Unit tests - the python debugger does not run properly from the doctests, and the original test are all unit tests...

In this case, it looks like you've made the code significantly more robust, which has added some probably necessary complexity. The code looks readable, but I recommend a maintainer-oriented overview/ introduction as a doctest, at the least. For instance, perhaps you could think about documentation about the rationale for the approach and about the dance that this code participates in (with the lock files and all the possible SMTP error conditions and the code's responses). Of course, even more friendly docs than that would be nice, but I'm only asking for what I myself tend to give, unfortunately.

A little does need adding there - I have scanned state machine diagrams and the design rationale. I will go and put those in the checkin when I have a spare moment. Regards, Matthew

On Sun, Mar 09, 2008 at 08:54:11PM -0400, Gary Poster wrote:

I did a five-minute skim of the checkin but hope to look a bit more tomorrow. Hopefully Marius, Benji, Albertas, or someone else who has actually done work on this package will take a look and chime in.

I intend to. Actually, I feel the obligation to. Sadly, the feeling of obligation usually leads me to procrastinate and do something completely different... Marius Gedminas -- The BeOS takes the best features from the major operating systems. It's got the power and flexibility of Unix, the interface and ease of use of the MacOS, and Minesweeper from Windows. -- Tyler Riti