[Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages

[Lennart Regebro]

On Sat, Apr 17, 2010 at 03:41, Tres Seaver <tseaver at palladion.com> wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> This kind of goes with Lennart's frustration about trying to port the
> ZTK packages, or a core subset, to Python 3.
>
> I would like to see the ZTK packages have really excellent
> documentation, as well as 100% test coverage.  I think we need to look
> at refactoring how the testing story is done to get both needs
> satisfied:  trying to achieve both with a single set of doctests is
> pretty near impossible:
>
> - - Testing setup and teardown code obscures the narrative flow / intent
>  of the documentation.
>
> - - Edge case testing *really* obscures the same.
>
> - - Test coverage suffers, because it is hard to write edge-case tests
>  in doctest.
>
> - - Porting the doctests where people have tried to juggle both goals
>  across Python versions is a mess.
>
> The refactoring I would like to see happen is to move the main narrative
> documentation into a separate, Sphinx-driven 'docs' directory in each
> ZTK package.  As part of this move, we can start adding some of the
> really nice Sphinx features (cross references, indexing, auto-generation
> of API reference information, etc.).
>
> We can still leave the "main line" of the code illustrated using
> doctest-style blocks:  Sphinx has nice support for testing those blocks
> *while building the docs*.  At the end of this part of the change, each
> package would have a much improved documentation set, with tested
> examples.  At the end of the process, we would be able to put the docs
> for each package into an "intersphinx" tree underneath the main ZTK
> docs, which could serve as the gateway into them.
>
> The trickier testing bits we would re-write as super thorough, no
> shortcuts-taken unit tests:  one testcase class per class (or API
> function) under test, at least one method per class-under-test method,
> with more preferred to get all code paths / preconditions covered.  The
> goal here would be to comment out the doctests from the test_suites and
> get the code to 100% coverage using pure unit tests.  Meanwhile, the
> doctests would be refactored into the Sphinx documentation, losing all
> the bits which obscure their intent as documentation.
>
> I would say that the risks here are slight, given that aiming for 100%
> test coverage is likely to catch any oversights made during porting to
> the new style.  The amount of work for any given package is fairly-well
> contained, I think:  most of the effort will be in reverse-engineering
> the intent of the "edge-case" tests.
>
> I made a sketch at what these changes would look for the zope.event
> package like on a branch in Subversion:
>
>  http://svn.zope.org/zope.event/branches/tseaver-new_style_docs
>
> I select zope.event largely because it is small enough that the scope of
> the structural changes wouldn't be lost in the diffs for the actual
> tests (and also because I didn't want to do the work for a bigger
> package before getting feedback).
>
> The rendered docs from that branch are here:
>
>  http://palladion.com/static/zope.event-docs/
>
> I pushed the bzr branch to Launchpad, to let you see the kinds of
> changes I made to get there (our viewcvs story is not nearly as easy to
> review as the Launchpad changeset view):
>
>  https://code.launchpad.net/~tseaver/zope.event/new_style_docs
>
> I am attaching an annotated diff for those who would prefer to see the
> changes in that format.
>
> I would say that this cleanup effort should start at the "bottom" layers
> of the ZTK, dependency-wise (zope.interface, zope.component, etc.) and
> move "upwards" (in the depenency sense) over time.
>
> Thoughts?

+lots.