[Zope-dev] zope.testrunner misleading output

[Alan Franzoni]

Hello,
I've tried submitting this as a launchpad answer but it expired
without further notice; I'm posting this here, I hope it's the right
place to discuss this.

I'm indirectly using zope.testrunner through zc.buildout zc.recipe.testrunner:

[testunits]
recipe = zc.recipe.testrunner
eggs = pydenji
defaults = [ "--auto-color", "-vv", "--tests-pattern", "^test_.*" ]

It works; but what happens if a test file matching the pattern has got
a serious failure, let's say an import or syntax error ?

Output:
Test-module import failures:

Module: pydenji.test.test_cmdline

 File "/Users/alan/Dropbox/code/pydenji_release/pydenji/pydenji/test/test_cmdline.py",
line 13

  ^
SyntaxError: invalid syntax


Running tests at level 1
Running zope.testrunner.layer.UnitTests tests:
 Set up zope.testrunner.layer.UnitTests in 0.000 seconds.
 Running:

[...] long test output [... ]

 test_resolver_retrieves_package_resource_filename
(pydenji.test.test_uriresolver.TestPackageResolver)
 Ran 73 tests with 0 failures and 0 errors in 0.188 seconds.
Tearing down left over layers:
 Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.

Test-modules with import problems:
 pydenji.test.test_cmdline


I can't show you such things here, but the "73" is green, which is the
colour for "all ok" - if any failure happens, the colors turn red,
while the "test modules with import problems" are just a tiny line
after that, and often gets overlooked.

Also, a test file matching the pattern but which not defines any test
is treated the very same way as a file with import problems, which is
probably not what it want.

The issues I can find are:

- I need to dig to the top in order to get the traceback; other
frameworks, like twisted's own trial, print all the tracebacks at the
bottom of the test run for easy debugging;
- test colors should not turn green if any test with import problem is
around; maybe an import/syntax error should count as a generic error.
- while an import issue is a serious fact - meaning the test can't be
run, and should be reported, a test module which does not define any
test could just issue a warning - it could just be a placeholder test,
or a not-yet-finished test case, and should not be a blocking issue.

Any thoughts on this?


-- 
Alan Franzoni
--
contact me at public@[mysurname].eu