RFC: Proposed new style for documenting and testing ZTK packages
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 This kind of goes with Lennart's frustration about trying to port the ZTK packages, or a core subset, to Python 3. I would like to see the ZTK packages have really excellent documentation, as well as 100% test coverage. I think we need to look at refactoring how the testing story is done to get both needs satisfied: trying to achieve both with a single set of doctests is pretty near impossible: - - Testing setup and teardown code obscures the narrative flow / intent of the documentation. - - Edge case testing *really* obscures the same. - - Test coverage suffers, because it is hard to write edge-case tests in doctest. - - Porting the doctests where people have tried to juggle both goals across Python versions is a mess. The refactoring I would like to see happen is to move the main narrative documentation into a separate, Sphinx-driven 'docs' directory in each ZTK package. As part of this move, we can start adding some of the really nice Sphinx features (cross references, indexing, auto-generation of API reference information, etc.). We can still leave the "main line" of the code illustrated using doctest-style blocks: Sphinx has nice support for testing those blocks *while building the docs*. At the end of this part of the change, each package would have a much improved documentation set, with tested examples. At the end of the process, we would be able to put the docs for each package into an "intersphinx" tree underneath the main ZTK docs, which could serve as the gateway into them. The trickier testing bits we would re-write as super thorough, no shortcuts-taken unit tests: one testcase class per class (or API function) under test, at least one method per class-under-test method, with more preferred to get all code paths / preconditions covered. The goal here would be to comment out the doctests from the test_suites and get the code to 100% coverage using pure unit tests. Meanwhile, the doctests would be refactored into the Sphinx documentation, losing all the bits which obscure their intent as documentation. I would say that the risks here are slight, given that aiming for 100% test coverage is likely to catch any oversights made during porting to the new style. The amount of work for any given package is fairly-well contained, I think: most of the effort will be in reverse-engineering the intent of the "edge-case" tests. I made a sketch at what these changes would look for the zope.event package like on a branch in Subversion: http://svn.zope.org/zope.event/branches/tseaver-new_style_docs I select zope.event largely because it is small enough that the scope of the structural changes wouldn't be lost in the diffs for the actual tests (and also because I didn't want to do the work for a bigger package before getting feedback). The rendered docs from that branch are here: http://palladion.com/static/zope.event-docs/ I pushed the bzr branch to Launchpad, to let you see the kinds of changes I made to get there (our viewcvs story is not nearly as easy to review as the Launchpad changeset view): https://code.launchpad.net/~tseaver/zope.event/new_style_docs I am attaching an annotated diff for those who would prefer to see the changes in that format. I would say that this cleanup effort should start at the "bottom" layers of the ZTK, dependency-wise (zope.interface, zope.component, etc.) and move "upwards" (in the depenency sense) over time. Thoughts? Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvJEakACgkQ+gerLs4ltQ6X/wCgueT8RD10bKrM4OInF0sIj/Tk rs4AnAiQlBWZoHd35ti0tPlae/JsVgvC =K8N0 -----END PGP SIGNATURE-----
On 17 April 2010 09:41, Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
This kind of goes with Lennart's frustration about trying to port the ZTK packages, or a core subset, to Python 3.
I would like to see the ZTK packages have really excellent documentation, as well as 100% test coverage. Thoughts?
+1000, to both aim and approach. :) Martin
On Sat, Apr 17, 2010 at 03:41, Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
This kind of goes with Lennart's frustration about trying to port the ZTK packages, or a core subset, to Python 3.
I would like to see the ZTK packages have really excellent documentation, as well as 100% test coverage. I think we need to look at refactoring how the testing story is done to get both needs satisfied: trying to achieve both with a single set of doctests is pretty near impossible:
- - Testing setup and teardown code obscures the narrative flow / intent of the documentation.
- - Edge case testing *really* obscures the same.
- - Test coverage suffers, because it is hard to write edge-case tests in doctest.
- - Porting the doctests where people have tried to juggle both goals across Python versions is a mess.
The refactoring I would like to see happen is to move the main narrative documentation into a separate, Sphinx-driven 'docs' directory in each ZTK package. As part of this move, we can start adding some of the really nice Sphinx features (cross references, indexing, auto-generation of API reference information, etc.).
We can still leave the "main line" of the code illustrated using doctest-style blocks: Sphinx has nice support for testing those blocks *while building the docs*. At the end of this part of the change, each package would have a much improved documentation set, with tested examples. At the end of the process, we would be able to put the docs for each package into an "intersphinx" tree underneath the main ZTK docs, which could serve as the gateway into them.
The trickier testing bits we would re-write as super thorough, no shortcuts-taken unit tests: one testcase class per class (or API function) under test, at least one method per class-under-test method, with more preferred to get all code paths / preconditions covered. The goal here would be to comment out the doctests from the test_suites and get the code to 100% coverage using pure unit tests. Meanwhile, the doctests would be refactored into the Sphinx documentation, losing all the bits which obscure their intent as documentation.
I would say that the risks here are slight, given that aiming for 100% test coverage is likely to catch any oversights made during porting to the new style. The amount of work for any given package is fairly-well contained, I think: most of the effort will be in reverse-engineering the intent of the "edge-case" tests.
I made a sketch at what these changes would look for the zope.event package like on a branch in Subversion:
http://svn.zope.org/zope.event/branches/tseaver-new_style_docs
I select zope.event largely because it is small enough that the scope of the structural changes wouldn't be lost in the diffs for the actual tests (and also because I didn't want to do the work for a bigger package before getting feedback).
The rendered docs from that branch are here:
http://palladion.com/static/zope.event-docs/
I pushed the bzr branch to Launchpad, to let you see the kinds of changes I made to get there (our viewcvs story is not nearly as easy to review as the Launchpad changeset view):
https://code.launchpad.net/~tseaver/zope.event/new_style_docs
I am attaching an annotated diff for those who would prefer to see the changes in that format.
I would say that this cleanup effort should start at the "bottom" layers of the ZTK, dependency-wise (zope.interface, zope.component, etc.) and move "upwards" (in the depenency sense) over time.
Thoughts?
+lots.
(Although possible we could call it "developing" instead of "hacking" but that's not a big issue. ;) ) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On 2010-4-17 03:41, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
This kind of goes with Lennart's frustration about trying to port the ZTK packages, or a core subset, to Python 3.
I would like to see the ZTK packages have really excellent documentation, as well as 100% test coverage. I think we need to look at refactoring how the testing story is done to get both needs satisfied: trying to achieve both with a single set of doctests is pretty near impossible:
- - Testing setup and teardown code obscures the narrative flow / intent of the documentation.
- - Edge case testing *really* obscures the same.
- - Test coverage suffers, because it is hard to write edge-case tests in doctest.
- - Porting the doctests where people have tried to juggle both goals across Python versions is a mess.
The refactoring I would like to see happen is to move the main narrative documentation into a separate, Sphinx-driven 'docs' directory in each ZTK package. As part of this move, we can start adding some of the really nice Sphinx features (cross references, indexing, auto-generation of API reference information, etc.).
We can still leave the "main line" of the code illustrated using doctest-style blocks: Sphinx has nice support for testing those blocks *while building the docs*. At the end of this part of the change, each package would have a much improved documentation set, with tested examples. At the end of the process, we would be able to put the docs for each package into an "intersphinx" tree underneath the main ZTK docs, which could serve as the gateway into them.
The trickier testing bits we would re-write as super thorough, no shortcuts-taken unit tests: one testcase class per class (or API function) under test, at least one method per class-under-test method, with more preferred to get all code paths / preconditions covered. The goal here would be to comment out the doctests from the test_suites and get the code to 100% coverage using pure unit tests. Meanwhile, the doctests would be refactored into the Sphinx documentation, losing all the bits which obscure their intent as documentation.
I would say that the risks here are slight, given that aiming for 100% test coverage is likely to catch any oversights made during porting to the new style. The amount of work for any given package is fairly-well contained, I think: most of the effort will be in reverse-engineering the intent of the "edge-case" tests.
I made a sketch at what these changes would look for the zope.event package like on a branch in Subversion:
http://svn.zope.org/zope.event/branches/tseaver-new_style_docs
I select zope.event largely because it is small enough that the scope of the structural changes wouldn't be lost in the diffs for the actual tests (and also because I didn't want to do the work for a bigger package before getting feedback).
The rendered docs from that branch are here:
http://palladion.com/static/zope.event-docs/
I pushed the bzr branch to Launchpad, to let you see the kinds of changes I made to get there (our viewcvs story is not nearly as easy to review as the Launchpad changeset view):
https://code.launchpad.net/~tseaver/zope.event/new_style_docs
I am attaching an annotated diff for those who would prefer to see the changes in that format.
I would say that this cleanup effort should start at the "bottom" layers of the ZTK, dependency-wise (zope.interface, zope.component, etc.) and move "upwards" (in the depenency sense) over time.
Thoughts?
+1 Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
Hi! Tres Seaver wrote:
The refactoring I would like to see happen is to move the main narrative documentation into a separate, Sphinx-driven 'docs' directory in each ZTK package. As part of this move, we can start adding some of the really nice Sphinx features (cross references, indexing, auto-generation of API reference information, etc.).
We can still leave the "main line" of the code illustrated using doctest-style blocks: Sphinx has nice support for testing those blocks *while building the docs*. [...] Thoughts?
+1 in general, but I'm concerned about one thing: How can we make sure docs and code don't get out of sync? Do we have to run unittests *and* build the docs before each checkin? Will someone make sure buildbots and nightly tests report broken docs as well? Cheers, Yuppie
On Sat, Apr 17, 2010 at 11:18, yuppie <y.2010@wcm-solutions.de> wrote:
How can we make sure docs and code don't get out of sync? Do we have to run unittests *and* build the docs before each checkin? Will someone make sure buildbots and nightly tests report broken docs as well?
Hm... As long as you use the >>> syntax and none of the ..test-block:: syntax of sphinx, the sphinx docs can be tested as a part of the standard testruns, but I guess part of this proposal was so we could easily use the testblock syntax as it has the possibility to hide setup etc easier. And then I don't know the best way to integrate it, but it would definitely be a drawback if you need two different testruns to run all the tests... Also I think it should be the responsibility of the one who does a release to make sure that we can build HTML docs and that the formatting isn't *too* broken. Preferably these docs should also be uploaded to packages.python.org, maybe there can be a makefile step for that in the docs? -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On 17 April 2010 10:41, Lennart Regebro <regebro@gmail.com> wrote:
On Sat, Apr 17, 2010 at 11:18, yuppie <y.2010@wcm-solutions.de> wrote:
How can we make sure docs and code don't get out of sync? Do we have to run unittests *and* build the docs before each checkin? Will someone make sure buildbots and nightly tests report broken docs as well?
Hm... As long as you use the >>> syntax and none of the ..test-block:: syntax of sphinx, the sphinx docs can be tested as a part of the standard testruns, but I guess part of this proposal was so we could easily use the testblock syntax as it has the possibility to hide setup etc easier. And then I don't know the best way to integrate it, but it would definitely be a drawback if you need two different testruns to run all the tests...
Also I think it should be the responsibility of the one who does a release to make sure that we can build HTML docs and that the formatting isn't *too* broken. Preferably these docs should also be uploaded to packages.python.org, maybe there can be a makefile step for that in the docs?
It's important that documentation is tested as part of the standard test run, that means when you change something you know to update the docs. repoze.bfg seemed to make an attempt at this, though it is currently disabled. http://svn.repoze.org/repoze.bfg/trunk/repoze/bfg/tests/test_docs.py Laurence
On Sat, Apr 17, 2010 at 16:05, Laurence Rowe <l@lrowe.co.uk> wrote:
It's important that documentation is tested as part of the standard test run, that means when you change something you know to update the docs. repoze.bfg seemed to make an attempt at this, though it is currently disabled. http://svn.repoze.org/repoze.bfg/trunk/repoze/bfg/tests/test_docs.py
How could I forget about Manuel. *facepalm*. I've written Manuel extensions to test sphinx tests, but Benji hasn't had time to review it yet: http://svn.zope.org/manuel/branches/sphinx/ So it's perfectly possible to run all types of Sphinx tests even with codeblocks, as a part of the standard testrun. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Lennart Regebro wrote:
On Sat, Apr 17, 2010 at 16:05, Laurence Rowe <l@lrowe.co.uk> wrote:
It's important that documentation is tested as part of the standard test run, that means when you change something you know to update the docs. repoze.bfg seemed to make an attempt at this, though it is currently disabled. http://svn.repoze.org/repoze.bfg/trunk/repoze/bfg/tests/test_docs.py
How could I forget about Manuel. *facepalm*. I've written Manuel extensions to test sphinx tests, but Benji hasn't had time to review it yet: http://svn.zope.org/manuel/branches/sphinx/
So it's perfectly possible to run all types of Sphinx tests even with codeblocks, as a part of the standard testrun.
I'm ambivalent about testing the Sphinx code snippets on each test run. I want those snippets to be *much* less comprehensive than they are currently, and am pretty sure that drift in the non-executable bits is at least as important a problem as drift in the snippets. Basically, bugfix changeses shouldn't need to run the documentation tests, because the "visible" / main-line behavior isn't supposed to change at all. Feature changes should *start* by writing the narrative docs about the added feature: feature branches which don't include updates to the docs shouldn't be merged. Running the snippet tests as part of making a release (or even via some sort of '--all' argument for nightly runners) seems pretty adequate to me. If we upgrade our testing style to keep each testcase tightly focused, with meaningful names for the methods and local varioables, it should be much easier to figure out what, if anything, got overlooked in the documentaion. +0 on running them every time IFF we can find an easy way to run them during 'setup.py test' as well as buildout, and we continue to be mindful of the distinction between docs that work primarily as docs and tests that cover every code path. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvJ7Q0ACgkQ+gerLs4ltQ7QAwCfUZfetKf3ZBdCa4AZ/G+02jab w5AAn2ZkjA5eevc+clc4Jre/IHSk7SH0 =rdQf -----END PGP SIGNATURE-----
On Sat, Apr 17, 2010 at 19:17, Tres Seaver <tseaver@palladion.com> wrote:
I'm ambivalent about testing the Sphinx code snippets on each test run. I want those snippets to be *much* less comprehensive than they are currently, and am pretty sure that drift in the non-executable bits is at least as important a problem as drift in the snippets.
Well, unless they take of time, but they shouldn't, really. I think it's positive to get a quick feedback if you are breaking the documentation.
+0 on running them every time IFF we can find an easy way to run them during 'setup.py test' as well as buildout, and we continue to be mindful of the distinction between docs that work primarily as docs and tests that cover every code path.
I think manuel should solve that, we'll see, I'll have to test. I need to get manuel ported to Python 3 then too. :) -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Lennart Regebro wrote:
On Sat, Apr 17, 2010 at 19:17, Tres Seaver <tseaver@palladion.com> wrote:
I'm ambivalent about testing the Sphinx code snippets on each test run. I want those snippets to be *much* less comprehensive than they are currently, and am pretty sure that drift in the non-executable bits is at least as important a problem as drift in the snippets.
Well, unless they take of time, but they shouldn't, really. I think it's positive to get a quick feedback if you are breaking the documentation.
Again, I don't mind that part, but I want to break the cycle of jamming crap (for documentation purposes) into the docs for purposes of getting test coverage.
+0 on running them every time IFF we can find an easy way to run them during 'setup.py test' as well as buildout, and we continue to be mindful of the distinction between docs that work primarily as docs and tests that cover every code path.
I think manuel should solve that, we'll see, I'll have to test. I need to get manuel ported to Python 3 then too. :)
Turtles all the way down. ;) Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvKIHgACgkQ+gerLs4ltQ7foQCfTzBh8IGTr8beH/H+vdSLrAI5 TdEAoJWNJ1w/GQaVoMITw1siOxKaRJZj =ns67 -----END PGP SIGNATURE-----
On 04/17/2010 10:56 PM, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Lennart Regebro wrote:
On Sat, Apr 17, 2010 at 19:17, Tres Seaver<tseaver@palladion.com> wrote:
I'm ambivalent about testing the Sphinx code snippets on each test run. I want those snippets to be *much* less comprehensive than they are currently, and am pretty sure that drift in the non-executable bits is at least as important a problem as drift in the snippets.
Well, unless they take of time, but they shouldn't, really. I think it's positive to get a quick feedback if you are breaking the documentation.
Again, I don't mind that part, but I want to break the cycle of jamming crap (for documentation purposes) into the docs for purposes of getting test coverage.
I think Lennart was arguing for a safety belt: if we're not supposed to break those tests *at all* within bug fixes then I want to know right away if I did so accidentally. I guess your coverage argument was that if the unit tests do the coverage anyway then we won't miss anything, but we won't be 100% anyway and as the tests in the docs do exist and should run quickly, the safety belt would be cheap. I'm +0.5 after some pondering. -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Fri, Apr 16, 2010 at 9:41 PM, Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
This kind of goes with Lennart's frustration about trying to port the ZTK packages, or a core subset, to Python 3.
I would like to see the ZTK packages have really excellent documentation, as well as 100% test coverage. I think we need to look at refactoring how the testing story is done to get both needs satisfied: trying to achieve both with a single set of doctests is pretty near impossible:
- - Testing setup and teardown code obscures the narrative flow / intent of the documentation.
- - Edge case testing *really* obscures the same.
- - Test coverage suffers, because it is hard to write edge-case tests in doctest.
- - Porting the doctests where people have tried to juggle both goals across Python versions is a mess.
The refactoring I would like to see happen is to move the main narrative documentation into a separate, Sphinx-driven 'docs' directory in each ZTK package. As part of this move, we can start adding some of the really nice Sphinx features (cross references, indexing, auto-generation of API reference information, etc.).
We can still leave the "main line" of the code illustrated using doctest-style blocks: Sphinx has nice support for testing those blocks *while building the docs*. At the end of this part of the change, each package would have a much improved documentation set, with tested examples. At the end of the process, we would be able to put the docs for each package into an "intersphinx" tree underneath the main ZTK docs, which could serve as the gateway into them.
The trickier testing bits we would re-write as super thorough, no shortcuts-taken unit tests: one testcase class per class (or API function) under test, at least one method per class-under-test method, with more preferred to get all code paths / preconditions covered. The goal here would be to comment out the doctests from the test_suites and get the code to 100% coverage using pure unit tests. Meanwhile, the doctests would be refactored into the Sphinx documentation, losing all the bits which obscure their intent as documentation.
I would say that the risks here are slight, given that aiming for 100% test coverage is likely to catch any oversights made during porting to the new style. The amount of work for any given package is fairly-well contained, I think: most of the effort will be in reverse-engineering the intent of the "edge-case" tests.
I made a sketch at what these changes would look for the zope.event package like on a branch in Subversion:
http://svn.zope.org/zope.event/branches/tseaver-new_style_docs
I select zope.event largely because it is small enough that the scope of the structural changes wouldn't be lost in the diffs for the actual tests (and also because I didn't want to do the work for a bigger package before getting feedback).
The rendered docs from that branch are here:
http://palladion.com/static/zope.event-docs/
I pushed the bzr branch to Launchpad, to let you see the kinds of changes I made to get there (our viewcvs story is not nearly as easy to review as the Launchpad changeset view):
https://code.launchpad.net/~tseaver/zope.event/new_style_docs
I am attaching an annotated diff for those who would prefer to see the changes in that format.
I would say that this cleanup effort should start at the "bottom" layers of the ZTK, dependency-wise (zope.interface, zope.component, etc.) and move "upwards" (in the depenency sense) over time.
Thoughts?
I agree with the main thrust. (Of course, I disagree with some of your judgments of doctest, but that's mostly beside the point.) - Documentation should be written for documentation's sake. The emphasis should be on helping people understand what the software is for and how to use it, *not* on coverage. - Documentation should be executable. Manuel helps a lot for this. - I can almost guarantee that any examples that aren't tested will be wrong. I tried to do a good job on the bobo docs. I made almost all of the examples executable, and the ones I didn't had an amazing tendency to have bugs. - I agree that tests should be separate from documentation. You can get some of your coverage from the docs, but you'll need tests for edge cases and details not addressed by the docs. Jim -- Jim Fulton
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jim Fulton wrote:
On Fri, Apr 16, 2010 at 9:41 PM, Tres Seaver <tseaver@palladion.com> wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
This kind of goes with Lennart's frustration about trying to port the ZTK packages, or a core subset, to Python 3.
I would like to see the ZTK packages have really excellent documentation, as well as 100% test coverage. I think we need to look at refactoring how the testing story is done to get both needs satisfied: trying to achieve both with a single set of doctests is pretty near impossible:
- - Testing setup and teardown code obscures the narrative flow / intent of the documentation.
- - Edge case testing *really* obscures the same.
- - Test coverage suffers, because it is hard to write edge-case tests in doctest.
- - Porting the doctests where people have tried to juggle both goals across Python versions is a mess.
The refactoring I would like to see happen is to move the main narrative documentation into a separate, Sphinx-driven 'docs' directory in each ZTK package. As part of this move, we can start adding some of the really nice Sphinx features (cross references, indexing, auto-generation of API reference information, etc.).
We can still leave the "main line" of the code illustrated using doctest-style blocks: Sphinx has nice support for testing those blocks *while building the docs*. At the end of this part of the change, each package would have a much improved documentation set, with tested examples. At the end of the process, we would be able to put the docs for each package into an "intersphinx" tree underneath the main ZTK docs, which could serve as the gateway into them.
The trickier testing bits we would re-write as super thorough, no shortcuts-taken unit tests: one testcase class per class (or API function) under test, at least one method per class-under-test method, with more preferred to get all code paths / preconditions covered. The goal here would be to comment out the doctests from the test_suites and get the code to 100% coverage using pure unit tests. Meanwhile, the doctests would be refactored into the Sphinx documentation, losing all the bits which obscure their intent as documentation.
I would say that the risks here are slight, given that aiming for 100% test coverage is likely to catch any oversights made during porting to the new style. The amount of work for any given package is fairly-well contained, I think: most of the effort will be in reverse-engineering the intent of the "edge-case" tests.
I made a sketch at what these changes would look for the zope.event package like on a branch in Subversion:
http://svn.zope.org/zope.event/branches/tseaver-new_style_docs
I select zope.event largely because it is small enough that the scope of the structural changes wouldn't be lost in the diffs for the actual tests (and also because I didn't want to do the work for a bigger package before getting feedback).
The rendered docs from that branch are here:
http://palladion.com/static/zope.event-docs/
I pushed the bzr branch to Launchpad, to let you see the kinds of changes I made to get there (our viewcvs story is not nearly as easy to review as the Launchpad changeset view):
https://code.launchpad.net/~tseaver/zope.event/new_style_docs
I am attaching an annotated diff for those who would prefer to see the changes in that format.
I would say that this cleanup effort should start at the "bottom" layers of the ZTK, dependency-wise (zope.interface, zope.component, etc.) and move "upwards" (in the depenency sense) over time.
Thoughts?
I agree with the main thrust. (Of course, I disagree with some of your judgments of doctest, but that's mostly beside the point.)
I'm trying hard to keep my prejudices out of the discussion;).
- Documentation should be written for documentation's sake. The emphasis should be on helping people understand what the software is for and how to use it, *not* on coverage.
Amen.
- Documentation should be executable. Manuel helps a lot for this.
Amen. The Sphinx tools for this are pretty neat, too.
- I can almost guarantee that any examples that aren't tested will be wrong. I tried to do a good job on the bobo docs. I made almost all of the examples executable, and the ones I didn't had an amazing tendency to have bugs.
I'm not against having the snippets be executable, because I *do* want them to work. I just don't want to encourage anyone to think that they are testing the software when they write the snippets, or execute them. Executing the snippets is testing the documentation, not the software.
- I agree that tests should be separate from documentation. You can get some of your coverage from the docs, but you'll need tests for edge cases and details not addressed by the docs.
I would actually prefer to measure coverage (objectively) without reference to the snippets in the docs. Having *all* the tests for a module in one place helps think more clearly about getting good coverage, both from the "lines executed" standpoint (which can be objectively measured) and from the "semantics enforced" standpoint, which can't. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvKJg4ACgkQ+gerLs4ltQ4gFwCgsS6Vi6PL0VRZnQy5gPNZP3+U vBwAnjMqGByK6t7gkXUvQAmG7YeBfhpL =sF3w -----END PGP SIGNATURE-----
On 4/17/10 5:20 PM, Tres Seaver wrote:
- Documentation should be written for documentation's sake. The emphasis should be on helping people understand what the software is for and how to use it, *not* on coverage.
Amen.
- Documentation should be executable. Manuel helps a lot for this.
Amen. The Sphinx tools for this are pretty neat, too.
Documentation needn't be executable. Documentation should be right. But sometimes documentation is just documentation and automating the checking of its rightness isn't reasonable, especially if making it executable takes the slightest bit of meaning away from it as documentation. Sorry. No free lunch here. - C
On 18 April 2010 05:20, Tres Seaver <tseaver@palladion.com> wrote:
I'm not against having the snippets be executable, because I *do* want them to work. I just don't want to encourage anyone to think that they are testing the software when they write the snippets, or execute them. Executing the snippets is testing the documentation, not the software.
Everyone who's ever written a doctest, or is thinking about doing so: Please frame this and hang it above your monitor. :-) Martin
On 17/04/10 23:20, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Jim Fulton wrote:
<snip>
- I can almost guarantee that any examples that aren't tested will be wrong. I tried to do a good job on the bobo docs. I made almost all of the examples executable, and the ones I didn't had an amazing tendency to have bugs.
I'm not against having the snippets be executable, because I *do* want them to work. I just don't want to encourage anyone to think that they are testing the software when they write the snippets, or execute them.
I second Martin: the sentence hereunder is worth its weight in gold. (I wonder if this is an appropriate use of the translation of French "valoir son pesant d'or")
Executing the snippets is testing the documentation, not the software.
- I agree that tests should be separate from documentation. You can get some of your coverage from the docs, but you'll need tests for edge cases and details not addressed by the docs.
I would actually prefer to measure coverage (objectively) without reference to the snippets in the docs.
I think this would make a lot of sense.
Having *all* the tests for a module in one place helps think more clearly about getting good coverage, both from the "lines executed" standpoint (which can be objectively measured) and from the "semantics enforced" standpoint, which can't.
Tres.
With both unit and doc testing, we can get full "lines executed" coverage. Because of the insulation enforced by our unit testing framework, this can lead to better semantic coverage than with the dependent tests we write when doc testing. One of the reason that full "lines executed" coverage does not translate to full "semantics" coverage is the presence of class attributes. In fact, those attributes are holding "sub-global" state that multiplies the number of combinations that should be tested. I slowly have come to the idea that class attributes can be as bad as global variables : they allow hidden dependencies between methods. This is an insidious problem as class attributes look better until you are told other. A good sign that this problem is creeping in is when you need to write complex setup in order to write method tests. In other words, unit testing can be seen as a way of being warned early of too many attributes in our classes. PS: Hereabove, I speak only of the insulated semantics of each class. This does not remove the need for integration tests which *try* to test all interactions between those classes. -- Godefroid Chapelle (aka __gotcha) http://bubblenet.be
Hello Tres, Saturday, April 17, 2010, 3:41:02 AM, you wrote: TS> -----BEGIN PGP SIGNED MESSAGE----- TS> Hash: SHA1 <snip> +lots on more docs +lots on 100% coverage TS> The trickier testing bits we would re-write as super thorough, no TS> shortcuts-taken unit tests: one testcase class per class (or API TS> function) under test, at least one method per class-under-test method, TS> with more preferred to get all code paths / preconditions covered. The TS> goal here would be to comment out the doctests from the test_suites and TS> get the code to 100% coverage using pure unit tests. Meanwhile, the TS> doctests would be refactored into the Sphinx documentation, losing all TS> the bits which obscure their intent as documentation. I'm somewhat vary on unittests. I've seen some damn cryptic ones that took a lot of time to decipher. A doctest somehow forces you to dump your mind (well at least that, if we're not that brilliant techdoc writers). OTOH I think most unittests maybe have some comments, worst case they don't even use "speaking" variable names. I'm not sure whether we can enforce such rules (super thorough, no shortcuts-taken, well commented) if we can't easier ones. -- Best regards, Adam GROSZER mailto:agroszer@gmail.com -- Quote of the day: You look tired
Am 19.04.2010, 12:50 Uhr, schrieb Adam GROSZER <agroszer@gmail.com>: Before I start: + lots on the general idea. Zope's narrative documentation is attrocious"[1] and we have lost users and developers as a result.
I'm somewhat vary on unittests. I've seen some damn cryptic ones that took a lot of time to decipher.
All the more reason not to think of them as part of the documentation. Would it possible to come up with some metrics for the quality of tests?
A doctest somehow forces you to dump your mind (well at least that, if we're not that brilliant techdoc writers).
Narrative documentation forces you to explain yourself to someone else. Neither, however, will necessarily notice if you forget something: tests aren't documentation and docu
OTOH I think most unittests maybe have some comments, worst case they don't even use "speaking" variable names.
The key to any of this is making it easy for developers to want to go this way. Writing good, legible tests *will* encourage others to emulate them.
I'm not sure whether we can enforce such rules (super thorough, no shortcuts-taken, well commented) if we can't easier ones.
What does Lint have to say about them? Charlie [1] http://www.stereoplex.com/2010/jan/14/migrating-django-mingus/ -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226
On Mon, Apr 19, 2010 at 7:21 AM, Charlie Clark <charlie.clark@clark-consulting.eu> wrote:
Narrative documentation forces you to explain yourself to someone else. Neither, however, will necessarily notice if you forget something: tests aren't documentation and docu [sic]
Quite true. It is also true that tests should be well documented and documentation should be well tested. Some prefer doctests for both. -- Benji York
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Benji York wrote:
On Mon, Apr 19, 2010 at 7:21 AM, Charlie Clark <charlie.clark@clark-consulting.eu> wrote:
Narrative documentation forces you to explain yourself to someone else. Neither, however, will necessarily notice if you forget something: tests aren't documentation and docu [sic]
Quite true. It is also true that tests should be well documented and documentation should be well tested. Some prefer doctests for both.
I would disagree that "tests should be well documented", at least in a thread whose main purpose is to point up the very grave shortcomings in the narrative documentation for the ZTK package set as a whole. Writing clean, understandable code, with good class / method / function / variable names, comments where appropriate, etc., is not the same thing as documenting that code. Overloading the word "well-documented" to describe code which conforms to such (completely uncontroversial) standards muddies the point here. I am not arguing that writing good doctests is impossible, any more than writing good unit tests. What I am arguing is that we have lots of mediocre-to-poor tests (of both kinds), and that we have basically no documentation of the sort that most people mean when they use the term. I want us to take off the "I'm testing the code" hat when writing documentation, and focus on producing a much more useful documentation set, one which actually encourages folks who haven't already drunk our Kool-Aid to use the code. For instance, the ratio of lines of narrative text to lines of examples in the documentation ought to be *much* higher than the near-parity which is the rule in most ZTK doctests. I don't think we can get there by continuing to pretend that our existing doctests are working as documenation, overall, which was my rationale for moving the ones which might form the basis for real docs *out* of the code, and manage them separately as Sphinx documentation. As I have said elasewhere in the thread, I also believe that the snipptes in that documentation should be testable, where possible: I think that the Sphinx addons which allow running the snippets as part of building the tests are a really good fit for ensuring that the snippets don't bit-rot. Unfortunately, no tool is going to help ensure that the narrative itself hasn't bit-rotted: that takes a dedicated investment over time by the team maintaining the packages. I also believe that the goal of testability of document snippets remains subordinate to the goals of comprehensibility and of completness *as documentation*. Doumentation writers must work to remove any artifacts of testability which shows up in the rendered docs, where such artifacts don't actively aid comprehension. When we wear the testing hat, as opposed to the documenation hat, we should come to some kind of agreement about how to achieve the goal of getting our code thoroughly tested. I am not adamantly opposed to using doctests to aid in this goal, particularly for covering the "main line" of usage of an API. However, because they frequently test non-essential artifacts (hash ordering, exception rendering, etc.), they tend to be the most fragile of our tests across things like major Python version changes, platforms, etc. Also, because they take a narrative form, doctests alos encourage test writere to use shared state between logically-discrete tests, which leads to really ugly coupling and reduces the ability of the test writer to reason about semantic completeness of the tests. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvMjPAACgkQ+gerLs4ltQ5X0QCcDdeRhjUKLQtjcNZVXW7fGulv WLMAoLbNRu9SMximbOrqnsV2mqkWrbxz =SExz -----END PGP SIGNATURE-----
On Mon, Apr 19, 2010 at 12:50:23PM +0200, Adam GROSZER wrote:
Hello Tres,
Saturday, April 17, 2010, 3:41:02 AM, you wrote:
TS> -----BEGIN PGP SIGNED MESSAGE----- TS> Hash: SHA1
<snip>
+lots on more docs +lots on 100% coverage
TS> The trickier testing bits we would re-write as super thorough, no TS> shortcuts-taken unit tests: one testcase class per class (or API TS> function) under test, at least one method per class-under-test method, TS> with more preferred to get all code paths / preconditions covered. The TS> goal here would be to comment out the doctests from the test_suites and TS> get the code to 100% coverage using pure unit tests. Meanwhile, the TS> doctests would be refactored into the Sphinx documentation, losing all TS> the bits which obscure their intent as documentation.
I'm somewhat vary on unittests. I've seen some damn cryptic ones that took a lot of time to decipher. A doctest somehow forces you to dump your mind (well at least that, if we're not that brilliant techdoc writers).
That's true, but if the doctest gets too long, any readability advantages are negated. If you've the discipline to keep the doctests short, I don't see why you shouldn't continue writing them instead of unit tests -- and by "short" I mean 1-7 statements: import unittest import doctest from mypackage.mymodule import MyClass class TestMyClass(unittest.TestCase): def test_foo(): x = MyClass() self.assertEquals(x.foo(), 42) def doctest_MyClass_bar(): """Test MyClass.bar >>> y = MyClass() The bar method peforms a bar calculation that typically returns twenty-three: >>> y.bar() 23 """ def doctest_MyClass_bar_raises_if_incorrect_setup(): """Test MyClass.bar >>> y = MyClass(setup='incorrect') If you haven't supplied the correct setup, you get an exception: >>> y.bar() Traceback (most recent call last): ... IncorrectSetupError: the setup was incorrect """ def test_suite(): return unittest.TestSuite([ unittest.makeSuite(TestMyClass), doctest.DocTestSuite(), ]) The downside of this style is that it's difficult to refactor common bits from the doctests into shared methods, so you end up with a lot of duplicated code.
OTOH I think most unittests maybe have some comments, worst case they don't even use "speaking" variable names. I'm not sure whether we can enforce such rules (super thorough, no shortcuts-taken, well commented) if we can't easier ones.
Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development
On 4/19/10 15:48 , Marius Gedminas wrote:
def doctest_MyClass_bar(): """Test MyClass.bar
>>> y = MyClass()
The bar method peforms a bar calculation that typically returns twenty-three:
>>> y.bar() 23
"""
What is the advantage of that over: def test_something(self): # Test MyClass.bar y=MyClass() # The bar method peforms a bar calculation that typically # returns 23. self.assertEqual(y.bar(), 23) It reads the same, and as a bonus you can step through it with pdb and syntax highlighting works normally in most editors. Wichert.
On Mon, Apr 19, 2010 at 03:56:02PM +0200, Wichert Akkerman wrote:
On 4/19/10 15:48 , Marius Gedminas wrote:
def doctest_MyClass_bar(): """Test MyClass.bar
>>> y = MyClass()
The bar method peforms a bar calculation that typically returns twenty-three:
>>> y.bar() 23
"""
What is the advantage of that over:
def test_something(self): # Test MyClass.bar
y=MyClass()
*cringe* Sorry, I've this reflex to cringe every time I see a PEP-8 violation.
# The bar method peforms a bar calculation that typically # returns 23.
self.assertEqual(y.bar(), 23)
It reads the same, and as a bonus you can step through it with pdb and syntax highlighting works normally in most editors.
The "advantage" is that I've rarely seen comments in unit tests and personally I never felt compelled to write a comment when writing a unit test. A doctest without any text preceding a >>> line feels Wrong(TM) to me, and, judging from our test suite, the other programmers on my team. Whether that advantage is worth the multitude of disadvantages of doctest-for-unit-testing is a different question. Over the years my opinion slowly changed from "probably" to "rarely". Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development
On Mon, Apr 19, 2010 at 6:42 PM, Marius Gedminas <marius@gedmin.as> wrote:
On Mon, Apr 19, 2010 at 03:56:02PM +0200, Wichert Akkerman wrote:
On 4/19/10 15:48 , Marius Gedminas wrote:
def doctest_MyClass_bar(): """Test MyClass.bar
>>> y = MyClass()
The bar method peforms a bar calculation that typically returns twenty-three:
>>> y.bar() 23
"""
What is the advantage of that over:
def test_something(self): # Test MyClass.bar
y=MyClass()
*cringe*
Sorry, I've this reflex to cringe every time I see a PEP-8 violation.
# The bar method peforms a bar calculation that typically # returns 23.
self.assertEqual(y.bar(), 23)
It reads the same, and as a bonus you can step through it with pdb and syntax highlighting works normally in most editors.
The "advantage" is that I've rarely seen comments in unit tests and personally I never felt compelled to write a comment when writing a unit test.
Twisted and Launchpad both have policies that require new or modified unit tests to have comments, enforced as a part of our respective review processes. This seems to work well, since the reviewer can block silly comments. The documentation you get is more like a list of requirements / behaviours (e.g. "We raise an error when we try to create a branch in a namespace where there is already a branch of the same name") than a story, but sometimes that's a good thing. It's surprising how easy it is to slip into the habit. It's no substitute for tutorial / howto documentation, but is often a great supplement to API docs. jml
On 04/19/2010 03:56 PM, Wichert Akkerman wrote:
On 4/19/10 15:48 , Marius Gedminas wrote:
def doctest_MyClass_bar(): """Test MyClass.bar
>>> y = MyClass()
The bar method peforms a bar calculation that typically returns twenty-three:
>>> y.bar() 23
"""
What is the advantage of that over:
def test_something(self): # Test MyClass.bar
y=MyClass()
# The bar method peforms a bar calculation that typically # returns 23.
self.assertEqual(y.bar(), 23)
It reads the same, and as a bonus you can step through it with pdb and syntax highlighting works normally in most editors.
I had some good experiences in the last 6 months or so when switching from doctests that were tests (not documentation) to this style of testing. Also the review process mentioned (or pairing) helps to put comments in places where the code isn't easily understood by itself (after that, refactor the code so you don't need comments and put the real comments in). Christian -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Mon, Apr 19, 2010 at 15:48, Marius Gedminas <marius@gedmin.as> wrote:
If you've the discipline to keep the doctests short, I don't see why you shouldn't continue writing them instead of unit tests
Because they are a bitch to debug, relies on details of output, which makes them brittle, hard to port to Python 3, and can be tricky to set up with loads of subtle things like inheriting the environment from where the DocTest was created, etc. DocTests are for testing documentation, not documenting tests. Sorry guys, all other usage of doctests than making documentation testable is evil. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On Mon, Apr 19, 2010 at 16:03, Lennart Regebro <regebro@gmail.com> wrote:
On Mon, Apr 19, 2010 at 15:48, Marius Gedminas <marius@gedmin.as> wrote:
If you've the discipline to keep the doctests short, I don't see why you shouldn't continue writing them instead of unit tests
Because they are a bitch to debug, relies on details of output, which makes them brittle, hard to port to Python 3, and can be tricky to set up with loads of subtle things like inheriting the environment from where the DocTest was created, etc.
And from __future__ imports doesn't work. And.. I'm sure I'm gonna remember more reason why doctests are bad later, but I can't be bothered really. Just don't use them unless you are writing documentation. PLEASE! -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
On 04/19/2010 04:06 PM, Lennart Regebro wrote:
On Mon, Apr 19, 2010 at 16:03, Lennart Regebro<regebro@gmail.com> wrote:
On Mon, Apr 19, 2010 at 15:48, Marius Gedminas<marius@gedmin.as> wrote:
If you've the discipline to keep the doctests short, I don't see why you shouldn't continue writing them instead of unit tests
Because they are a bitch to debug, relies on details of output, which makes them brittle, hard to port to Python 3, and can be tricky to set up with loads of subtle things like inheriting the environment from where the DocTest was created, etc.
And from __future__ imports doesn't work. And.. I'm sure I'm gonna remember more reason why doctests are bad later, but I can't be bothered really.
Try defining a (persistent) class and sticking it into a ZODB. -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Mon, Apr 19, 2010 at 3:03 PM, Lennart Regebro <regebro@gmail.com> wrote:
On Mon, Apr 19, 2010 at 15:48, Marius Gedminas <marius@gedmin.as> wrote:
If you've the discipline to keep the doctests short, I don't see why you shouldn't continue writing them instead of unit tests
Because they are a bitch to debug, relies on details of output, which makes them brittle, hard to port to Python 3, and can be tricky to set up with loads of subtle things like inheriting the environment from where the DocTest was created, etc.
You may wish to read: http://bemusement.org/diary/2008/October/23/narrative-tests http://bemusement.org/diary/2008/October/24/more-doctest-problems which are an extended meditation on the whole topic. jml
On Mon, Apr 19, 2010 at 9:48 AM, Marius Gedminas <marius@gedmin.as> wrote:
On Mon, Apr 19, 2010 at 12:50:23PM +0200, Adam GROSZER wrote:
I'm somewhat vary on unittests. I've seen some damn cryptic ones that took a lot of time to decipher. A doctest somehow forces you to dump your mind (well at least that, if we're not that brilliant techdoc writers).
That's true, but if the doctest gets too long, any readability advantages are negated.
If you've the discipline to keep the doctests short, I don't see why you shouldn't continue writing them instead of unit tests -- and by "short" I mean 1-7 statements:
You might be interested in Manuel's isolation mechanism: http://packages.python.org/manuel/#test-isolation
The downside of this style is that it's difficult to refactor common bits from the doctests into shared methods, so you end up with a lot of duplicated code.
If I understand you (which I'm not sure I do), I prefer to factor out common code into test helpers that are either referenced in footnotes (to keep all the code in one place) or in a separate module. -- Benji York
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Adam GROSZER wrote:
I'm somewhat vary on unittests. I've seen some damn cryptic ones that took a lot of time to decipher. A doctest somehow forces you to dump your mind (well at least that, if we're not that brilliant techdoc writers). OTOH I think most unittests maybe have some comments, worst case they don't even use "speaking" variable names. I'm not sure whether we can enforce such rules (super thorough, no shortcuts-taken, well commented) if we can't easier ones.
I agree that the unit tests themselves often need cleaninng up: - - They should avoid importing the module-under-test at module scope, and minimize any other imports as well: import errors should cause tests to fail, not to be masked. - - They need to be broken up, so that each testcase method tests exactly one code path through each method (or function) under test. Each testcase method should be named to indicate which method/function it tests, and which set of preconditions. - - The local variables need to have *much* better names. - - They need to avoid using test fixtures from other modules: having to stop to look up exactly what the fixture means or does is a "pipeline stall" for the CPU of the programmer tryihg to understand the test. Doing this cleanup is part of the effort I am proposing we undertake: getting the tests comprehensible is as important as getting to 100% coverage, because the "lines executed" bit (syntax) that coverage analyzers report is still not sufficient to ensure that you cover all the semantics of the module. Analyzing completeness on the latter requires understanding what each method or function tested does, and how each test exercieses it. I have made a start on such cleanups for zope.interface: http://svn.zope.org/zope.interface/branches/tseaver-better_unit_tests/ Look particularly at the 'test_interface.py' changes: http://svn.zope.org/zope.interface/branches/tseaver-better_unit_tests/src/zo... and those to 'test_verify.py': http://svn.zope.org/zope.interface/branches/tseaver-better_unit_tests/src/zo... Note as well that I am deliberately leaving off the table the "integration testing" which is normally done via the zope.testbrowser layer support: as with doctests, such tests can be useful, but I would never rely on them for the "coverage" part of testing a package. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvMeB0ACgkQ+gerLs4ltQ7rXwCfVxQho7XLjsYTQITxuWHF3/3N 3hcAoLlL/LoaUlLSlaEpjANGPD3jR7rz =nqY1 -----END PGP SIGNATURE-----
On 04/17/2010 03:41 AM, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
This kind of goes with Lennart's frustration about trying to port the ZTK packages, or a core subset, to Python 3.
I would like to see the ZTK packages have really excellent documentation, as well as 100% test coverage. I think we need to look at refactoring how the testing story is done to get both needs satisfied: trying to achieve both with a single set of doctests is pretty near impossible:
- - Testing setup and teardown code obscures the narrative flow / intent of the documentation.
- - Edge case testing *really* obscures the same.
- - Test coverage suffers, because it is hard to write edge-case tests in doctest.
- - Porting the doctests where people have tried to juggle both goals across Python versions is a mess.
The refactoring I would like to see happen is to move the main narrative documentation into a separate, Sphinx-driven 'docs' directory in each ZTK package. As part of this move, we can start adding some of the really nice Sphinx features (cross references, indexing, auto-generation of API reference information, etc.).
We can still leave the "main line" of the code illustrated using doctest-style blocks: Sphinx has nice support for testing those blocks *while building the docs*. At the end of this part of the change, each package would have a much improved documentation set, with tested examples. At the end of the process, we would be able to put the docs for each package into an "intersphinx" tree underneath the main ZTK docs, which could serve as the gateway into them.
The trickier testing bits we would re-write as super thorough, no shortcuts-taken unit tests: one testcase class per class (or API function) under test, at least one method per class-under-test method, with more preferred to get all code paths / preconditions covered. The goal here would be to comment out the doctests from the test_suites and get the code to 100% coverage using pure unit tests. Meanwhile, the doctests would be refactored into the Sphinx documentation, losing all the bits which obscure their intent as documentation.
I would say that the risks here are slight, given that aiming for 100% test coverage is likely to catch any oversights made during porting to the new style. The amount of work for any given package is fairly-well contained, I think: most of the effort will be in reverse-engineering the intent of the "edge-case" tests.
I made a sketch at what these changes would look for the zope.event package like on a branch in Subversion:
http://svn.zope.org/zope.event/branches/tseaver-new_style_docs
I select zope.event largely because it is small enough that the scope of the structural changes wouldn't be lost in the diffs for the actual tests (and also because I didn't want to do the work for a bigger package before getting feedback).
The rendered docs from that branch are here:
http://palladion.com/static/zope.event-docs/
I pushed the bzr branch to Launchpad, to let you see the kinds of changes I made to get there (our viewcvs story is not nearly as easy to review as the Launchpad changeset view):
https://code.launchpad.net/~tseaver/zope.event/new_style_docs
I am attaching an annotated diff for those who would prefer to see the changes in that format.
I would say that this cleanup effort should start at the "bottom" layers of the ZTK, dependency-wise (zope.interface, zope.component, etc.) and move "upwards" (in the depenency sense) over time.
Thoughts?
+the-rest-of-points-my-karma-allows-me Let me raise another issue which is that we also need to figure out how to follow up with this once we start. ;) -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
Am 20.04.2010, 18:43 Uhr, schrieb Christian Theune <ct@gocept.com>:
Let me raise another issue which is that we also need to figure out how to follow up with this once we start.
Could get a lint "template" (or whatever it's called) to help here? Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226
On 04/21/2010 09:05 PM, Charlie Clark wrote:
Am 20.04.2010, 18:43 Uhr, schrieb Christian Theune<ct@gocept.com>:
Let me raise another issue which is that we also need to figure out how to follow up with this once we start.
Could get a lint "template" (or whatever it's called) to help here?
Not sure what that means, can you elaborate a bit? Christian -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
Am 22.04.2010, 05:48 Uhr, schrieb Christian Theune <ct@gocept.com>:
Could get a lint "template" (or whatever it's called) to help here? Not sure what that means, can you elaborate a bit?
Lint is a tool that analyses code for all kinds of languages, including Python, and scores it against various definable metrics. The idea is to encourage code readability and best practices. I imagine that for more "predictable" parts of code like unit tests it might be possible to use it to encourage a uniform style such as TestMock seems to promise for VisualStudio http://site.typemock.com/test-lint/ I'll admit to not using PyLint (from Nuxeo) myself on a regular basis but this is as much down to not knowing how to configure it to work with Zope projects as much as anything else. But maybe we can add it to anything the repository policy checker? Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226
Am 17.04.2010, 03:41 Uhr, schrieb Tres Seaver <tseaver@palladion.com>:
The trickier testing bits we would re-write as super thorough, no shortcuts-taken unit tests: one testcase class per class (or API function) under test, at least one method per class-under-test method, with more preferred to get all code paths / preconditions covered. The goal here would be to comment out the doctests from the test_suites and get the code to 100% coverage using pure unit tests. Meanwhile, the doctests would be refactored into the Sphinx documentation, losing all the bits which obscure their intent as documentation.
I'm all for this but in the past I have struggled mightily (this could be an ironic use of the word ;-) with unit tests for views. Is this 1) Because I've overlooked something? 2) Because views depend on too many things, setting up test cases is a lot of work? 3) Because it's not been done a lot in the past? Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226
participants (15)
-
Adam GROSZER -
Benji York -
Charlie Clark -
Chris McDonough -
Christian Theune -
Godefroid Chapelle -
Jim Fulton -
Jonathan Lange -
Laurence Rowe -
Lennart Regebro -
Marius Gedminas -
Martin Aspeli -
Tres Seaver -
Wichert Akkerman -
yuppie