[Zope3-dev] The Zope Software Certification Program and Common Repository Proposal

[Lennart Regebro]

On 2/21/06, Stephan Richter <srichter at cosmos.phy.tufts.edu> wrote:
> > Most testing I do is not about handling cases at all, it's about
> > testing specific functionality, making sure that old bugs doesn't pop
> > up again and stuff like that. Most of these tests would in doctest
> > format provide no documentation at all.
>
> Ah, that is in the eye of the beholder. I would argue that bug fixes, for
> example, should be either tested by integrating text in the documentation or
> tested using a docstring test, which SchoolTool commonly does. The idea is
> not to say: "This test has been written to test the fix for issue 12234", but
> rather write something like: "A special case is XYZ, which works as follows:"

Well, possibly, although I'm not convinced.

> Any test should provide documentation.

That depends on what you mean with "provide". Of course you need in a
comment explain why you do the test, and what it tests, but that does
not mean that the test makes sense anywhere in  what would normally be
called "the documentation".

> > For my calendar, many of the
> > tests are test to do things as migrate data from the old calendar
> > product, making sure that the attendee source for CPS does what it
> > should, but it has no educational use in itself,
>
> Well, a doctest could explain the migration test and what has changed.

Nothing "changed". It's two different calendar products. It's
basically an import/export from the old calendar to the new. There is
nothing to explain.

> But for translation consistency you have a different test setup anyways.

Eh, no... I don't think I follow you here. It's a test, it's run by
the test runner. If you are referring to the setUp code, then I
actually don't understand what that has to do with it.

> Doctests are for testing code, not translations or similar things.

Well, then we at least agree on one exception to the rule. :-)

I don't agree doctest is for testing code. I think it's for testing
documentation.

> > Many tests require setup/teardown
> > functionality, which gets hard to do in the inherently linear format
> > of doctests.
>
> I used to think this way too until Jim convinced me that showing all the setup
> steps and teardown is actually useful to the reader. It shows how much is
> involved in getting a particular piece of code running. Alternatively, you
> can of course define setup and teardown functions when instantiating the
> suite.

That's not the problem, the problem is that I need to run the
setup/teardown many times. I could split the tests I have now so that
each test is a separate doctest file, and use the setup/teardown
during instantiation, but is it really reasonable to have say, ten
doctest files that do very similar things, all saying things like "The
calendar can migrate from the old calendar ever if it has french
accents", and then having the code for this (which is rather long).
That's not much of a doctest. It is definitely not any sort of
documentation. And, it also means that the setup and teardown steps
are NOT useful to the reader. In fact, it is my now repeatedly stated
opinion that several of CPSSharedCalendars tests are not useful to any
reader. They make sure it actually works, they don't provide any
insight into how to use CPSSharedCalendar. You use it by clicking
around on a website, not by writing python code.

I think you are stuck in library-development mode. ;-) But this
proposal isn't only made for libraries, right? It seems to me that
several of the requirements have to do with site integration and such,
things you get for finished products that you can install and use.
They need to be documented of course, and part of that documentation
can very well be in doctest format, but most of the documentation that
is needed (and currently sadly missing) is for end-users. A doctest
requirement wouldn't help the documentation in any reasonable way, and
it would slow down the test-writing.


And why should I pretend that the migration tests are documentation of
how to write a migration, when it is in fact a test that migration
works even with non-ascii characters? There is today no documentation
of how to write migration code from the old calendar to the new,
because that code is written. I don't need to document how to do it
again. :-)

> SchoolTool solves this by simply storing an old ZODB database and then run the
> generation scripts, because this is what happens in the real world anyways.

Well, I solve it by not using doctests, which I like better, because
it enables me to easily add and change the tests according to me
finding something new that needs testing, withouthaving to change a
binary file which can't be diffed. And how do you fit the ZODB into
the documentation? :-)

Tests are not always documentation. It's nice if they can be both, but
I don't think it's a good idea to require it to be both, because quite
often, it isn't.

--
Lennart Regebro, Nuxeo     http://www.nuxeo.com/
CPS Content Management     http://www.cps-project.org/