[Zope3-dev] The Zope Software Certification Program and Common Repository Proposal

Stephan Richter srichter at cosmos.phy.tufts.edu
Tue Feb 21 11:19:53 EST 2006


On Tuesday 21 February 2006 11:02, Lennart Regebro wrote:
> On 2/21/06, Stephan Richter <srichter at cosmos.phy.tufts.edu> wrote:
> > Why is it random? It is taken straight from the conventions now used in
> > Zope 3 for all new development. The rationale behind it is that you are
> > forced to document and reason all the cases the software handles.
>
> Most testing I do is not about handling cases at all, it's about
> testing specific functionality, making sure that old bugs doesn't pop
> up again and stuff like that. Most of these tests would in doctest
> format provide no documentation at all.

Ah, that is in the eye of the beholder. I would argue that bug fixes, for 
example, should be either tested by integrating text in the documentation or 
tested using a docstring test, which SchoolTool commonly does. The idea is 
not to say: "This test has been written to test the fix for issue 12234", but 
rather write something like: "A special case is XYZ, which works as follows:"

Any test should provide documentation.

> For my calendar, many of the 
> tests are test to do things as migrate data from the old calendar
> product, making sure that the attendee source for CPS does what it
> should, but it has no educational use in itself,

Well, a doctest could explain the migration test and what has changed. 
SchoolTool, for example, tests its migration scripts using doctests.

> the API for is is 
> tested in CalCore and CalZope and that is indeed in DocTest format (at
> least partly), we also have tests to check that the translations are
> consistent, and stuff like that.

But for translation consistency you have a different test setup anyways. 
Doctests are for testing code, not translations or similar things.

> Many tests require setup/teardown 
> functionality, which gets hard to do in the inherently linear format
> of doctests.

I used to think this way too until Jim convinced me that showing all the setup 
steps and teardown is actually useful to the reader. It shows how much is 
involved in getting a particular piece of code running. Alternatively, you 
can of course define setup and teardown functions when instantiating the 
suite.

> For example, the upgrade tests need to create old 
> calendars at the setup. Each test then fills the calendars with
> different types of data, and migrates the calendar. This migration
> includes installation of the new software and replacing of a local
> utility, which then needs to be undone before the next test. I would
> eitehr need to duplicate this code many times, or test all cases in
> one fat migration step, which would make it much harder to figure out
> exactly what failed.

SchoolTool solves this by simply storing an old ZODB database and then run the 
generation scripts, because this is what happens in the real world anyways.

> It is indeed good practice to start your development with doing
> usecases, expanding them to be doctest and using this to drive
> dvelopment. But once the code has matured past that stage, I see no
> reason to require that all tests should be doctests, or you should
> need to convince people why doctests are impossible. It  may possible
> increase the documentation quality marginally, but it may likewise
> lower the testing quality.

I don't think our testing quality has decreased in Zope 3, since we switched 
to pure doctests.

Regards,
Stephan
-- 
Stephan Richter
CBU Physics & Chemistry (B.S.) / Tufts Physics (Ph.D. student)
Web2k - Web Software Design, Development and Training


More information about the Zope3-dev mailing list