[Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jim Fulton wrote:
> On Fri, Apr 16, 2010 at 9:41 PM, Tres Seaver <tseaver at palladion.com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA1
>>
>> This kind of goes with Lennart's frustration about trying to port the
>> ZTK packages, or a core subset, to Python 3.
>>
>> I would like to see the ZTK packages have really excellent
>> documentation, as well as 100% test coverage.  I think we need to look
>> at refactoring how the testing story is done to get both needs
>> satisfied:  trying to achieve both with a single set of doctests is
>> pretty near impossible:
>>
>> - - Testing setup and teardown code obscures the narrative flow / intent
>>  of the documentation.
>>
>> - - Edge case testing *really* obscures the same.
>>
>> - - Test coverage suffers, because it is hard to write edge-case tests
>>  in doctest.
>>
>> - - Porting the doctests where people have tried to juggle both goals
>>  across Python versions is a mess.
>>
>> The refactoring I would like to see happen is to move the main narrative
>> documentation into a separate, Sphinx-driven 'docs' directory in each
>> ZTK package.  As part of this move, we can start adding some of the
>> really nice Sphinx features (cross references, indexing, auto-generation
>> of API reference information, etc.).
>>
>> We can still leave the "main line" of the code illustrated using
>> doctest-style blocks:  Sphinx has nice support for testing those blocks
>> *while building the docs*.  At the end of this part of the change, each
>> package would have a much improved documentation set, with tested
>> examples.  At the end of the process, we would be able to put the docs
>> for each package into an "intersphinx" tree underneath the main ZTK
>> docs, which could serve as the gateway into them.
>>
>> The trickier testing bits we would re-write as super thorough, no
>> shortcuts-taken unit tests:  one testcase class per class (or API
>> function) under test, at least one method per class-under-test method,
>> with more preferred to get all code paths / preconditions covered.  The
>> goal here would be to comment out the doctests from the test_suites and
>> get the code to 100% coverage using pure unit tests.  Meanwhile, the
>> doctests would be refactored into the Sphinx documentation, losing all
>> the bits which obscure their intent as documentation.
>>
>> I would say that the risks here are slight, given that aiming for 100%
>> test coverage is likely to catch any oversights made during porting to
>> the new style.  The amount of work for any given package is fairly-well
>> contained, I think:  most of the effort will be in reverse-engineering
>> the intent of the "edge-case" tests.
>>
>> I made a sketch at what these changes would look for the zope.event
>> package like on a branch in Subversion:
>>
>>  http://svn.zope.org/zope.event/branches/tseaver-new_style_docs
>>
>> I select zope.event largely because it is small enough that the scope of
>> the structural changes wouldn't be lost in the diffs for the actual
>> tests (and also because I didn't want to do the work for a bigger
>> package before getting feedback).
>>
>> The rendered docs from that branch are here:
>>
>>  http://palladion.com/static/zope.event-docs/
>>
>> I pushed the bzr branch to Launchpad, to let you see the kinds of
>> changes I made to get there (our viewcvs story is not nearly as easy to
>> review as the Launchpad changeset view):
>>
>>  https://code.launchpad.net/~tseaver/zope.event/new_style_docs
>>
>> I am attaching an annotated diff for those who would prefer to see the
>> changes in that format.
>>
>> I would say that this cleanup effort should start at the "bottom" layers
>> of the ZTK, dependency-wise (zope.interface, zope.component, etc.) and
>> move "upwards" (in the depenency sense) over time.
>>
>> Thoughts?
> 
> I agree with the main thrust. (Of course, I disagree with some of your
> judgments of doctest, but that's mostly beside the point.)

I'm trying hard to keep my prejudices out of the discussion;).

> - Documentation should be written for documentation's sake. The
> emphasis should be on helping people understand what the software is
> for and how to use it, *not* on coverage.

Amen.

> - Documentation should be executable.  Manuel helps a lot for this.

Amen.  The Sphinx tools for this are pretty neat, too.

> - I can almost guarantee that any examples that aren't tested will be
> wrong. I tried to do a good job on the bobo docs.  I made almost all
> of the examples executable, and the ones I didn't had an amazing
> tendency to have bugs.

I'm not against having the snippets be executable, because I *do* want
them to work.  I just don't want to encourage anyone to think that they
are testing the software when they write the snippets, or execute them.
  Executing the snippets is testing the documentation, not the software.

> - I agree that tests should be separate from documentation. You can
> get some of your coverage from the docs, but you'll need tests for
> edge cases and details not addressed by the docs.

I would actually prefer to measure coverage (objectively) without
reference to the snippets in the docs.  Having *all* the tests for a
module in one place helps think more clearly about getting good
coverage, both from the "lines executed" standpoint (which can be
objectively measured) and from the "semantics enforced" standpoint,
which can't.



Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkvKJg4ACgkQ+gerLs4ltQ4gFwCgsS6Vi6PL0VRZnQy5gPNZP3+U
vBwAnjMqGByK6t7gkXUvQAmG7YeBfhpL
=sF3w
-----END PGP SIGNATURE-----