[Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Benji York wrote:
> On Mon, Apr 19, 2010 at 7:21 AM, Charlie Clark
> <charlie.clark at clark-consulting.eu> wrote:
>> Narrative documentation forces you to explain yourself to someone else.
>> Neither, however, will necessarily notice if you forget something: tests
>> aren't documentation and docu [sic]
> 
> Quite true.  It is also true that tests should be well documented and
> documentation should be well tested.  Some prefer doctests for both.

I would disagree that "tests should be well documented", at least in a
thread whose main purpose is to point up the very grave shortcomings in
the narrative documentation for the ZTK package set as a whole.  Writing
clean, understandable code, with good class / method / function /
variable names, comments where appropriate, etc., is not the same thing
as documenting that code.  Overloading the word "well-documented" to
describe code which conforms to such (completely uncontroversial)
standards muddies the point here.

I am not arguing that writing good doctests is impossible, any more than
writing good unit tests.  What I am arguing is that we have lots of
mediocre-to-poor tests (of both kinds), and that we have basically no
documentation of the sort that most people mean when they use the term.

I want us to take off the "I'm testing the code" hat when writing
documentation, and focus on producing a much more useful documentation
set, one which actually encourages folks who haven't already drunk our
Kool-Aid to use the code.

For instance, the ratio of lines of narrative text to lines of examples
in the documentation ought to be *much* higher than the near-parity
which is the rule in most ZTK doctests.  I don't think we can get there
by continuing to pretend that our existing doctests are working as
documenation, overall, which was my rationale for moving the ones which
might form the basis for real docs *out* of the code, and manage them
separately as Sphinx documentation.

As I have said elasewhere in the thread, I also believe that the
snipptes in that documentation should be testable, where possible:  I
think that the Sphinx addons which allow running the snippets as part of
building the tests are a really good fit for ensuring that the snippets
don't bit-rot.  Unfortunately, no tool is going to help ensure that the
narrative itself hasn't bit-rotted:  that takes a dedicated investment
over time by the team maintaining the packages.

I also believe that the goal of testability of document snippets remains
subordinate to the goals of comprehensibility and of completness *as
documentation*.  Doumentation writers must work to remove any artifacts
of testability which shows up in the rendered docs, where such artifacts
don't actively aid comprehension.

When we wear the testing hat, as opposed to the documenation hat, we
should come to some kind of agreement about how to achieve the goal of
getting our code thoroughly tested.

I am not adamantly opposed to using doctests to aid in this goal,
particularly for covering the "main line" of usage of an API.  However,
because they frequently test non-essential artifacts (hash ordering,
exception rendering, etc.), they tend to be the most fragile of our
tests across things like major Python version changes, platforms, etc.
Also, because they take a narrative form, doctests alos encourage test
writere to use shared state between logically-discrete tests, which
leads to really ugly coupling and reduces the ability of the test writer
to reason about semantic completeness of the tests.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkvMjPAACgkQ+gerLs4ltQ5X0QCcDdeRhjUKLQtjcNZVXW7fGulv
WLMAoLbNRu9SMximbOrqnsV2mqkWrbxz
=SExz
-----END PGP SIGNATURE-----