[Zope-dev] zope.testrunner misleading output

[Alan Franzoni]

On Wed, Dec 22, 2010 at 3:13 PM, Marius Gedminas <marius at gedmin.as> wrote:

> It is reported near the beginning of the output (highlighted in red, to
> stand out), mentioned in the summary at the end to ensure you don't
> miss it, and the test runner exits with a non-zero status code (I hope;
> if it doesn't, that's a bug).

Yes, the status code is correctly set - I wasn't complaining about
that, the runner works fine in our CI system - but I usually don't
pipe || while running tests at the console.

Also, there's a bit of "impedence" between "0 failures and 0 errors"
and the exit status 1. I mean, the first time it happened i thought
"WTF?". I mean: it's very easy to associate "red" with "something bad
occurred", I like that, and I'd like that to be extended.

Maybe something like

"""
Ran A tests with B failures, C errors and D other problems
"""

would satisfy us all.



> That's an interesting perspective.
>
> Note that even when there are failures, the number of tests and the
> number of seconds are highlighted in green.  (The colours there are
> mainly to make the numbers stand out so they're easier to notice in the
> output.)

Yes, you're right, it's just the number of failures that turns red, if any.

What I'd probably like is to have failed imports and syntax errors to
be counted as errors, while files defining no tests just as warnings.

>> probably not what it want.
>
> What's "it"?

it = I , sorry, my typo.

>> - I need to dig to the top in order to get the traceback; other
>> frameworks, like twisted's own trial, print all the tracebacks at the
>> bottom of the test run for easy debugging;
>
> Having used other frameworks, I appreciate zope.testrunner's eagerness
> to show me the traceback at once, so I can start examining an error
> without having to stare at an F in the middle of a sea of dots and
> wonder what it might be, while waiting 20 minutes for the rest of the
> test suite to finish running.

I realize zope.testrunner layering system is designed to run unit,
integration and maybe acceptance tests, which can be pretty consuming.
I usually run mostly unit tests, taking less than 2 seconds.

By the way you're describing nose behaviour, I think; twisted trial is
better at this; normally it just outputs the errors at the end, but if
the "-e" switch is passed, it prints the tracebacks both ASAP *and* at
the end. I guess a similar approach, using a switch, could be employed
in zope.testrunner.

> Then again I agree that having to scroll back to the first traceback is
> a necessity that's a bit bothersome.  I don't think printing the
> tracebacks at the end of the run would help, in case there were multiple
> tracebacks -- you'd want the first one anyway; the others likely being
> caused by it.  Also, tracebacks tend to be long, requiring me to scroll
> anyway.

If I'm running unit tests there's no certain connection between
tracebacks, and most probably there's no definite "first". Again,
probably a configurable switch would make everybody happy :-)


>> - while an import issue is a serious fact - meaning the test can't be
>> run, and should be reported, a test module which does not define any
>> test could just issue a warning - it could just be a placeholder test,
>> or a not-yet-finished test case, and should not be a blocking issue.
>
> An import error could be a placeholder as well.  What makes a module
> with no tests different?  If you added a module, it's reasonable to
> assume you added it for a reason, and what other reason could there be
> other than for it to have tests?

An import error is surely an error; if it can't be imported it can't
work in any way. The file might define 100 tests, and those are not
run.

A syntax error is surely an error as well; if written with the correct
syntax there might be tests inside, and they won't get executed.

A file defining no tests may be an error, but you can never be sure;
hence I think you should tell the user but you shouldn't fail by
default, IMHO.


NB:
if any of my proposal is accepted, I'm willing to contribute a patch,
I just don't want to start coding for something that might later just
be trashed.


-- 
Alan Franzoni
--
contact me at public@[mysurname].eu