[Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages

[Jim Fulton]

On Fri, Apr 16, 2010 at 9:41 PM, Tres Seaver <tseaver at palladion.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> This kind of goes with Lennart's frustration about trying to port the
> ZTK packages, or a core subset, to Python 3.
>
> I would like to see the ZTK packages have really excellent
> documentation, as well as 100% test coverage.  I think we need to look
> at refactoring how the testing story is done to get both needs
> satisfied:  trying to achieve both with a single set of doctests is
> pretty near impossible:
>
> - - Testing setup and teardown code obscures the narrative flow / intent
>  of the documentation.
>
> - - Edge case testing *really* obscures the same.
>
> - - Test coverage suffers, because it is hard to write edge-case tests
>  in doctest.
>
> - - Porting the doctests where people have tried to juggle both goals
>  across Python versions is a mess.
>
> The refactoring I would like to see happen is to move the main narrative
> documentation into a separate, Sphinx-driven 'docs' directory in each
> ZTK package.  As part of this move, we can start adding some of the
> really nice Sphinx features (cross references, indexing, auto-generation
> of API reference information, etc.).
>
> We can still leave the "main line" of the code illustrated using
> doctest-style blocks:  Sphinx has nice support for testing those blocks
> *while building the docs*.  At the end of this part of the change, each
> package would have a much improved documentation set, with tested
> examples.  At the end of the process, we would be able to put the docs
> for each package into an "intersphinx" tree underneath the main ZTK
> docs, which could serve as the gateway into them.
>
> The trickier testing bits we would re-write as super thorough, no
> shortcuts-taken unit tests:  one testcase class per class (or API
> function) under test, at least one method per class-under-test method,
> with more preferred to get all code paths / preconditions covered.  The
> goal here would be to comment out the doctests from the test_suites and
> get the code to 100% coverage using pure unit tests.  Meanwhile, the
> doctests would be refactored into the Sphinx documentation, losing all
> the bits which obscure their intent as documentation.
>
> I would say that the risks here are slight, given that aiming for 100%
> test coverage is likely to catch any oversights made during porting to
> the new style.  The amount of work for any given package is fairly-well
> contained, I think:  most of the effort will be in reverse-engineering
> the intent of the "edge-case" tests.
>
> I made a sketch at what these changes would look for the zope.event
> package like on a branch in Subversion:
>
>  http://svn.zope.org/zope.event/branches/tseaver-new_style_docs
>
> I select zope.event largely because it is small enough that the scope of
> the structural changes wouldn't be lost in the diffs for the actual
> tests (and also because I didn't want to do the work for a bigger
> package before getting feedback).
>
> The rendered docs from that branch are here:
>
>  http://palladion.com/static/zope.event-docs/
>
> I pushed the bzr branch to Launchpad, to let you see the kinds of
> changes I made to get there (our viewcvs story is not nearly as easy to
> review as the Launchpad changeset view):
>
>  https://code.launchpad.net/~tseaver/zope.event/new_style_docs
>
> I am attaching an annotated diff for those who would prefer to see the
> changes in that format.
>
> I would say that this cleanup effort should start at the "bottom" layers
> of the ZTK, dependency-wise (zope.interface, zope.component, etc.) and
> move "upwards" (in the depenency sense) over time.
>
> Thoughts?

I agree with the main thrust. (Of course, I disagree with some of your
judgments of doctest, but that's mostly beside the point.)

- Documentation should be written for documentation's sake. The
emphasis should be on helping people understand what the software is
for and how to use it, *not* on coverage.

- Documentation should be executable.  Manuel helps a lot for this.

- I can almost guarantee that any examples that aren't tested will be
wrong. I tried to do a good job on the bobo docs.  I made almost all
of the examples executable, and the ones I didn't had an amazing
tendency to have bugs.

- I agree that tests should be separate from documentation. You can
get some of your coverage from the docs, but you'll need tests for
edge cases and details not addressed by the docs.

Jim

-- 
Jim Fulton