[Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Lennart Regebro wrote:
> On Sat, Apr 17, 2010 at 16:05, Laurence Rowe <l at lrowe.co.uk> wrote:
>> It's important that documentation is tested as part of the standard
>> test run, that means when you change something you know to update the
>> docs. repoze.bfg seemed to make an attempt at this, though it is
>> currently disabled.
>> http://svn.repoze.org/repoze.bfg/trunk/repoze/bfg/tests/test_docs.py
> 
> How could I forget about Manuel. *facepalm*. I've written Manuel
> extensions to test sphinx tests, but Benji hasn't had time to review
> it yet: http://svn.zope.org/manuel/branches/sphinx/
> 
> So it's perfectly possible to run all types of Sphinx tests even with
> codeblocks, as a part of the standard testrun.

I'm ambivalent about testing the Sphinx code snippets on each test run.
 I want those snippets to be *much* less comprehensive than they are
currently, and am pretty sure that drift in the non-executable bits is
at least as important a problem as drift in the snippets.

Basically, bugfix changeses shouldn't need to run the documentation
tests, because the "visible" / main-line behavior isn't supposed to
change at all.  Feature changes should *start* by writing the narrative
docs about the added feature:  feature branches which don't include
updates to the docs shouldn't be merged.

Running the snippet tests as part of making a release (or even via some
sort of '--all' argument for nightly runners) seems pretty adequate to
me.  If we upgrade our testing style to keep each testcase tightly
focused, with meaningful names for the methods and local varioables, it
should be much easier to figure out what, if anything, got overlooked in
the documentaion.

+0 on running them every time IFF we can find an easy way to run them
during 'setup.py test' as well as buildout, and we continue to be
mindful of the distinction between docs that work primarily as docs and
tests that cover every code path.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkvJ7Q0ACgkQ+gerLs4ltQ7QAwCfUZfetKf3ZBdCa4AZ/G+02jab
w5AAn2ZkjA5eevc+clc4Jre/IHSk7SH0
=rdQf
-----END PGP SIGNATURE-----