[Zope-dev] RFC: Proposed new style for documenting and testing ZTK packages

[Godefroid Chapelle]

On 17/04/10 23:20, Tres Seaver wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> Jim Fulton wrote:

<snip>

>
>> - I can almost guarantee that any examples that aren't tested will be
>> wrong. I tried to do a good job on the bobo docs.  I made almost all
>> of the examples executable, and the ones I didn't had an amazing
>> tendency to have bugs.
>
> I'm not against having the snippets be executable, because I *do* want
> them to work.  I just don't want to encourage anyone to think that they
> are testing the software when they write the snippets, or execute them.

I second Martin: the sentence hereunder is worth its weight in gold.

(I wonder if this is an appropriate use of the translation of French 
"valoir son pesant d'or")

>    Executing the snippets is testing the documentation, not the software.

>
>> - I agree that tests should be separate from documentation. You can
>> get some of your coverage from the docs, but you'll need tests for
>> edge cases and details not addressed by the docs.
>
> I would actually prefer to measure coverage (objectively) without
> reference to the snippets in the docs.

I think this would make a lot of sense.

> Having *all* the tests for a
> module in one place helps think more clearly about getting good
> coverage, both from the "lines executed" standpoint (which can be
> objectively measured) and from the "semantics enforced" standpoint,
> which can't.
 >
 >
 > Tres.

With both unit and doc testing, we can get full "lines executed" coverage.

Because of the insulation enforced by our unit testing framework, this 
can lead to better semantic coverage than with the dependent tests we 
write when doc testing.

One of the reason that full "lines executed" coverage does not translate 
to full "semantics" coverage is the presence of class attributes. In 
fact, those attributes are holding "sub-global" state that multiplies 
the number of combinations that should be tested.

I slowly have come to the idea that class attributes can be as bad as 
global variables : they allow hidden dependencies between methods. This 
is an insidious problem as class attributes look better until you are 
told other.

A good sign that this problem is creeping in is when you need to write 
complex setup in order to write method tests.

In other words, unit testing can be seen as a way of being warned early 
of too many attributes in our classes.

PS: Hereabove, I speak only of the insulated semantics of each class. 
This does not remove the need for integration tests which *try* to test 
all interactions between those classes.
-- 
Godefroid Chapelle (aka __gotcha) http://bubblenet.be