Why I dislike narrative doctests
Suppose I find a bug (say, zope.testing.testrunner.StartUpFailure objects make unittest.TextTestRunner cry). Being a good developer I want to start the bug fix with a unit test. Now if zope.testing used old-style isolated unit tests, I could open the tests.py (or tests/test_testrunner.py) in a text editor and start writing: class OptionsStub(object): post_mortem = False def doctest_StartUpFailure(): """Test that StartUpFailure is a proper unittest.TestCase StartUpFailure is a TestCase-lookalike that is inserted into the test suite to indicate that a failed import of a test module won't be unnoticed. >>> from zope.testing.testrunner import StartUpFailure >>> options = OptionsStub() >>> module_name = 'test_me' >>> exc_info = (ImportError, ImportError('zonkulator'), None) >>> test = StartUpFailure(options, module_name, exc_info) >>> unittest.TextTestRunner(verbosity=2).run(test) This is a regression test for http://launchpad.net/bugs/221151 """ There, I'm almost done. Now I can fix the bug and paste the correct output into the doctest. But zope.testing uses narrative doctests instead of isolated unit tests. There are 22 plain-text files that together comprise over 4500 lines of text. Now instead of mechanically opening the appropriate test module and grepping for the class/function name I have to figure out which of the txt files is the appropriate place for the new test. After that I have to figure out how to insert it into the narrative seamlessly. Or decide to create a new file---but a hundred of .txt files each 20-lines long is not a good idea, in my opinion. Suddenly I almost don't want to fix the bug any more. While I was writing this, I suddenly realised that almost all zope.testing's txt files are actually functional tests. There are nearly no unit tests there. Perhaps that's what causing me pain. I still maintain that: * adding a new unit test should be a trivial task * extending a narrative doctest is not a trivial task * the only way to solve this conflict is to explicitly separate unit tests (test_*.py) from doctest-enhanced API documentation (*.txt). Marius Gedminas -- UNIX is user friendly. It's just selective about who its friends are.
Can't you start a plain-old unittests module and stick your test in there? Is there any rule that says you cannot create a plain-old test module if the project you are fixing the bug in uses doctests? On Wed, Apr 23, 2008 at 5:19 PM, Marius Gedminas <mgedmin@b4net.lt> wrote:
Suppose I find a bug (say, zope.testing.testrunner.StartUpFailure objects make unittest.TextTestRunner cry). Being a good developer I want to start the bug fix with a unit test.
Now if zope.testing used old-style isolated unit tests, I could open the tests.py (or tests/test_testrunner.py) in a text editor and start writing:
class OptionsStub(object): post_mortem = False
def doctest_StartUpFailure(): """Test that StartUpFailure is a proper unittest.TestCase
StartUpFailure is a TestCase-lookalike that is inserted into the test suite to indicate that a failed import of a test module won't be unnoticed.
>>> from zope.testing.testrunner import StartUpFailure >>> options = OptionsStub() >>> module_name = 'test_me' >>> exc_info = (ImportError, ImportError('zonkulator'), None) >>> test = StartUpFailure(options, module_name, exc_info) >>> unittest.TextTestRunner(verbosity=2).run(test)
This is a regression test for http://launchpad.net/bugs/221151 """
There, I'm almost done. Now I can fix the bug and paste the correct output into the doctest.
But zope.testing uses narrative doctests instead of isolated unit tests. There are 22 plain-text files that together comprise over 4500 lines of text. Now instead of mechanically opening the appropriate test module and grepping for the class/function name I have to figure out which of the txt files is the appropriate place for the new test. After that I have to figure out how to insert it into the narrative seamlessly. Or decide to create a new file---but a hundred of .txt files each 20-lines long is not a good idea, in my opinion.
Suddenly I almost don't want to fix the bug any more.
While I was writing this, I suddenly realised that almost all zope.testing's txt files are actually functional tests. There are nearly no unit tests there. Perhaps that's what causing me pain.
I still maintain that:
* adding a new unit test should be a trivial task
* extending a narrative doctest is not a trivial task
* the only way to solve this conflict is to explicitly separate unit tests (test_*.py) from doctest-enhanced API documentation (*.txt).
Marius Gedminas -- UNIX is user friendly. It's just selective about who its friends are.
-----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux)
iD8DBQFID5m0kVdEXeem148RAqs5AJ4qUo9IiqMSsuFOIrZBTq+xyWv/FQCePM/Q eTvxFfGfDPuGA1YzlM8CTks= =J91E -----END PGP SIGNATURE-----
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
-- Sidnei da Silva Enfold Systems http://enfoldsystems.com Fax +1 832 201 8856 Office +1 713 942 2377 Ext 214
On Wed, Apr 23, 2008 at 05:22:29PM -0300, Sidnei da Silva wrote:
Can't you start a plain-old unittests module and stick your test in there? Is there any rule that says you cannot create a plain-old test module if the project you are fixing the bug in uses doctests?
I'll probably do that (since I sort of took over the maintainership of zope.testing a while ago). The point of my message was not to whine about the state of zope.testing, but to present a new argument against the current fashion of using plain-text narrative doctests for everything. These are very fine when you initially write them, but a pain when you come back a year later and need to modify them. Trying to kill two birds with one stone (i.e. have narrative doctests serve both as documentation and as the test suite) doesn't seem to work: * users complain about boring corner cases and irrelevant test fixtures in what they expected to be documentation * programmers complain about narratives being too hard to modify Marius Gedminas -- You can't spell evil without vi.
On Apr 23, 2008, at 4:47 PM, Marius Gedminas wrote: ...
The point of my message was not to whine about the state of zope.testing, but to present a new argument against the current fashion of using plain-text narrative doctests for everything.
Except that that is not the current fashion, which has been pointed out many times in many places. Jim -- Jim Fulton Zope Corporation
Hi, On Wed, Apr 23, 2008 at 04:47:59PM -0400, Jim Fulton wrote:
On Apr 23, 2008, at 4:47 PM, Marius Gedminas wrote: ...
The point of my message was not to whine about the state of zope.testing, but to present a new argument against the current fashion of using plain-text narrative doctests for everything.
Except that that is not the current fashion, which has been pointed out many times in many places.
For my own record (I must have missed this many times in many places), is the current fashion something along the lines: "Use the various test styles as reasonable, text-file narrative doctests are preferred." ? Christian -- gocept gmbh & co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - ct@gocept.com - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development
On Apr 24, 2008, at 1:12 AM, Christian Theune wrote:
Hi,
On Wed, Apr 23, 2008 at 04:47:59PM -0400, Jim Fulton wrote:
On Apr 23, 2008, at 4:47 PM, Marius Gedminas wrote: ...
The point of my message was not to whine about the state of zope.testing, but to present a new argument against the current fashion of using plain-text narrative doctests for everything.
Except that that is not the current fashion, which has been pointed out many times in many places.
For my own record (I must have missed this many times in many places), is the current fashion something along the lines:
"Use the various test styles as reasonable, text-file narrative doctests are preferred."
No. WRT doc tests: - If a file is documentation and a test, make sure it is good documentation. In that case, documentation comes first. Don't add so many tests that it ruins the documentation. - Test edge cases in separate tests. These are typically short-ish strings in test modules. - A variation is to have a narrative that doesn't try hard to be documentation. The narrative can be convenient, up to a point, even in a test. These should be clearly marked as not being documentation. Jim -- Jim Fulton Zope Corporation
On Thu, Apr 24, 2008 at 10:07 PM, Jim Fulton <jim@zope.com> wrote:
- If a file is documentation and a test, make sure it is good documentation. In that case, documentation comes first. Don't add so many tests that it ruins the documentation.
- Test edge cases in separate tests. These are typically short-ish strings in test modules.
- A variation is to have a narrative that doesn't try hard to be documentation. The narrative can be convenient, up to a point, even in a test. These should be clearly marked as not being documentation.
I'ö happy to here this from an authorative source. I've been saying it for years, and I have usually been contradicted by people saying "no all tests should be doctests", Lets kill that sillyness once and for all now. :-) -- Lennart Regebro: Zope and Plone consulting. http://www.colliberty.com/ +33 661 58 14 64
Lennart Regebro wrote:
On Thu, Apr 24, 2008 at 10:07 PM, Jim Fulton <jim@zope.com> wrote:
- If a file is documentation and a test, make sure it is good documentation. In that case, documentation comes first. Don't add so many tests that it ruins the documentation.
- Test edge cases in separate tests. These are typically short-ish strings in test modules.
- A variation is to have a narrative that doesn't try hard to be documentation. The narrative can be convenient, up to a point, even in a test. These should be clearly marked as not being documentation.
I'ö happy to here this from an authorative source. I've been saying it for years, and I have usually been contradicted by people saying "no all tests should be doctests", Lets kill that sillyness once and for all now. :-)
I've written such rules down: http://svn.zope.org/Sandbox/philikon/foundation. If it's missing something, feel free to suggest a patch.
On Apr 25, 2008, at 1:59 AM, Lennart Regebro wrote:
On Thu, Apr 24, 2008 at 10:07 PM, Jim Fulton <jim@zope.com> wrote:
- If a file is documentation and a test, make sure it is good documentation. In that case, documentation comes first. Don't add so many tests that it ruins the documentation.
- Test edge cases in separate tests. These are typically short-ish strings in test modules.
- A variation is to have a narrative that doesn't try hard to be documentation. The narrative can be convenient, up to a point, even in a test. These should be clearly marked as not being documentation.
I'ö happy to here this from an authorative source. I've been saying it for years, and I have usually been contradicted by people saying "no all tests should be doctests", Lets kill that sillyness once and for all now. :-)
Everything I said above refers to doctests. I only pointed out that long narrative doctests aren't always desirable. Having said that, I can live with regular unit tests that are easy to follow and documented. Jim -- Jim Fulton Zope Corporation
On Thu, Apr 24, 2008 at 04:07:12PM -0400, Jim Fulton wrote:
- If a file is documentation and a test, make sure it is good documentation. In that case, documentation comes first. Don't add so many tests that it ruins the documentation.
- Test edge cases in separate tests. These are typically short-ish strings in test modules.
- A variation is to have a narrative that doesn't try hard to be documentation. The narrative can be convenient, up to a point, even in a test. These should be clearly marked as not being documentation.
Thanks for summing that up. -- gocept gmbh & co. kg - forsterstrasse 29 - 06112 halle (saale) - germany www.gocept.com - ct@gocept.com - phone +49 345 122 9889 7 - fax +49 345 122 9889 1 - zope and plone consulting and development
On Thu, Apr 24, 2008 at 3:07 PM, Jim Fulton <jim@zope.com> wrote:
On Apr 24, 2008, at 1:12 AM, Christian Theune wrote:
Hi,
On Wed, Apr 23, 2008 at 04:47:59PM -0400, Jim Fulton wrote:
On Apr 23, 2008, at 4:47 PM, Marius Gedminas wrote: ...
The point of my message was not to whine about the state of zope.testing, but to present a new argument against the current fashion of using plain-text narrative doctests for everything.
Except that that is not the current fashion, which has been pointed out many times in many places.
For my own record (I must have missed this many times in many places), is the current fashion something along the lines:
"Use the various test styles as reasonable, text-file narrative doctests are preferred."
No. WRT doc tests:
- If a file is documentation and a test, make sure it is good documentation. In that case, documentation comes first. Don't add so many tests that it ruins the documentation.
- Test edge cases in separate tests. These are typically short-ish strings in test modules.
- A variation is to have a narrative that doesn't try hard to be documentation. The narrative can be convenient, up to a point, even in a test. These should be clearly marked as not being documentation.
However, as Sphinx lowers the barrier to cross-referencing, they will become important as link destinations from 'real' documentation.
Jim
-- Jim Fulton Zope Corporation
_______________________________________________
Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists -http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
On Apr 25, 2008, at 10:20 AM, Kent Tenney wrote:
However, as Sphinx lowers the barrier to cross-referencing,
Sounds like I need to check that out. :)
they will become important as link destinations from 'real' documentation.
Maaaybe. I like doctests for readability even if they aren't documentation. I'd be hesitant to link to d=non-documentation from documentation. Jim -- Jim Fulton Zope Corporation
On Friday 25 April 2008, Jim Fulton wrote:
However, as Sphinx lowers the barrier to cross-referencing,
Sounds like I need to check that out. :)
Paul just diud that work the last two days for z3c.form: http://docs.carduner.net/z3c.form/ Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"
On Fri, Apr 25, 2008 at 9:39 AM, Jim Fulton <jim@zope.com> wrote:
On Apr 25, 2008, at 10:20 AM, Kent Tenney wrote:
However, as Sphinx lowers the barrier to cross-referencing,
Sounds like I need to check that out. :)
they will become important as link destinations from 'real' documentation.
Maaaybe. I like doctests for readability even if they aren't documentation. I'd be hesitant to link to d=non-documentation from documentation.
I think even non-documentation doctests are documentation since they demonstrate correct usage. They need to be explained in the linking document, but they are invaluable as examples. Seems like a natural separation of concerns.
Jim
-- Jim Fulton Zope Corporation
On Fri, Apr 25, 2008 at 10:11 AM, Kent Tenney <ktenney@gmail.com> wrote:
On Fri, Apr 25, 2008 at 9:39 AM, Jim Fulton <jim@zope.com> wrote:
On Apr 25, 2008, at 10:20 AM, Kent Tenney wrote:
However, as Sphinx lowers the barrier to cross-referencing,
Sounds like I need to check that out. :)
they will become important as link destinations from 'real' documentation.
Maaaybe. I like doctests for readability even if they aren't documentation. I'd be hesitant to link to d=non-documentation from documentation.
I think even non-documentation doctests are documentation since they demonstrate correct usage. They need to be explained in the linking document, but they are invaluable as examples.
Seems like a natural separation of concerns.
To make this work well, the doctests would want to have some markup added to identify specific tests, so the link would be to the test, not the file. This would be a good entry level chore for folks wanting to contribute.
Jim
-- Jim Fulton Zope Corporation
On Thursday 24 April 2008, Jim Fulton wrote:
- If a file is documentation and a test, make sure it is good documentation. In that case, documentation comes first. Don't add so many tests that it ruins the documentation.
- Test edge cases in separate tests. These are typically short-ish strings in test modules.
I tend to put low level-documentation into text files that have the same name as the module it is testing. I usually writing for the advanced user or extension writer target audience. It is usually somewhere in between API and technical reference documentation. If I can I provide a narrative there too, though this is usually hard. But at least I am motivating the code and API (which I think is missing in a lot of the doctests that we have.) I personally hate these strings in test modules and do not use them.
- A variation is to have a narrative that doesn't try hard to be documentation. The narrative can be convenient, up to a point, even in a test. These should be clearly marked as not being documentation.
Why is this not documentation? I think we frequently forget to talk about different audiences. There is documentation for people wanting to use a particular module, extend the features or fix problems. In application code, I write functional tests for the product/project manager and even the end user. Regards, Stephan -- Stephan Richter Web Software Design, Development and Training Google me. "Zope Stephan Richter"
On Apr 25, 2008, at 11:22 AM, Stephan Richter wrote:
- A variation is to have a narrative that doesn't try hard to be documentation. The narrative can be convenient, up to a point, even in a test. These should be clearly marked as not being documentation.
Why is this not documentation?
Because it's written primarily as a test, not as documentation. For a file to be useable as documentation, greater thought needs to be given to the user reading it. Sometimes, I can't afford the extra time. Jim -- Jim Fulton Zope Corporation
On Apr 23, 2008, at 4:19 PM, Marius Gedminas wrote:
Suppose I find a bug (say, zope.testing.testrunner.StartUpFailure objects make unittest.TextTestRunner cry). Being a good developer I want to start the bug fix with a unit test.
Now if zope.testing used old-style isolated unit tests, I could open the tests.py (or tests/test_testrunner.py) in a text editor and start writing:
class OptionsStub(object): post_mortem = False
def doctest_StartUpFailure(): """Test that StartUpFailure is a proper unittest.TestCase
StartUpFailure is a TestCase-lookalike that is inserted into the test suite to indicate that a failed import of a test module won't be unnoticed.
from zope.testing.testrunner import StartUpFailure options = OptionsStub() module_name = 'test_me' exc_info = (ImportError, ImportError('zonkulator'), None) test = StartUpFailure(options, module_name, exc_info) unittest.TextTestRunner(verbosity=2).run(test)
This is a regression test for http://launchpad.net/bugs/221151 """
There, I'm almost done. Now I can fix the bug and paste the correct output into the doctest.
But zope.testing uses narrative doctests instead of isolated unit tests. There are 22 plain-text files that together comprise over 4500 lines of text. Now instead of mechanically opening the appropriate test module and grepping for the class/function name I have to figure out which of the txt files is the appropriate place for the new test. After that I have to figure out how to insert it into the narrative seamlessly. Or decide to create a new file---but a hundred of .txt files each 20- lines long is not a good idea, in my opinion.
Suddenly I almost don't want to fix the bug any more.
You don't have to modify any narrative tests. You can add isolated tests to your heart's content. Just add tests to tests.py. These can be isolated doctests or unit tests, although I prefer the former. I haven't done this yet in zope.testing, but I have in lots of other projects. See, for example: http://svn.zope.org/zc.buildout/trunk/src/zc/buildout/tests.py?rev=85041&vie... I like narrative tests when the tests are reasonably a part of a story. Not all tests are, or many tests are there own stories unto themselves. In cases like this, small isolated tests are a lot better. Jim -- Jim Fulton Zope Corporation
On Wed, Apr 23, 2008 at 10:19 PM, Marius Gedminas <mgedmin@b4net.lt> wrote:
Suppose I find a bug (say, zope.testing.testrunner.StartUpFailure objects make unittest.TextTestRunner cry). Being a good developer I want to start the bug fix with a unit test.
...
I still maintain that:
* adding a new unit test should be a trivial task
* extending a narrative doctest is not a trivial task
Yup. And then, try to debug it when it fails. Oups, you can't pdb through it. I repeat, and will continue to repeat until everybody gets it: ;-) Doctests are great. For writing testable documentation. They are NOT good for writing tests.
* the only way to solve this conflict is to explicitly separate unit tests (test_*.py) from doctest-enhanced API documentation (*.txt).
Yup. -- Lennart Regebro: Zope and Plone consulting. http://www.colliberty.com/ +33 661 58 14 64
participants (8)
-
Christian Theune -
Jim Fulton -
Kent Tenney -
Lennart Regebro -
Marius Gedminas -
Philipp von Weitershausen -
Sidnei da Silva -
Stephan Richter