[Zope-dev] [PATCH] subunit output for zope.testing

[Jonathan Lange]

On Thu, Mar 11, 2010 at 9:17 PM, Marius Gedminas <marius at gedmin.as> wrote:
> On Thu, Mar 11, 2010 at 08:52:11PM +0000, Jonathan Lange wrote:
>> For a long time, I've wanted to have a subunit output formatter for
>> zope.testing. Subunit is a language-generic streaming protocol for
>> test results. Once a test runner can display subunit output, it
>> becomes much easier to write tools to process output programmatically.
>> We want to use it in the Launchpad project for analyzing test results
>> and for distributing our test runs across multiple machines. Other
>> projects use it to get Hudson integration, or to glue their
>> multi-language test suite into one result.
>
> I like this.
>

Thanks.

> Incidentally, how do you integrate it with Hudson?  subunit2junitxml, I
> assume?  How exactly do you combine bin/test with subunit2junitxml -- I
> assume some care with respect to exit codes?
>

I haven't tried myself, but from analogy with other subunit tools:
  ./bin/test --subunit | subunit2junitxml

The formatter doesn't control exit codes, so I didn't really think
about them in this patch. Should I produce a follow up?

>> === modified file 'src/zope/testing/testrunner/formatter.py'
>> --- src/zope/testing/testrunner/formatter.py  2009-12-23 21:21:53 +0000
>> +++ src/zope/testing/testrunner/formatter.py  2010-03-11 20:29:26 +0000
> ...
>> +    def __init__(self, options):
>> +        if subunit is None:
>> +            raise Exception("Requires subunit 0.0.4 or better")
>> +        if content is None:
>> +            raise Exception("Requires testtools XXX or better")
>
> This seems a mite imprecise (unless testtools uses roman numerals for
> version numbering).
>

Oops :) I've changed it to say 0.9.2, and subunit to say 0.0.5, which
is the actual version I've tested with.

>> === modified file 'src/zope/testing/testrunner/testrunner-leaks.txt'
>> --- src/zope/testing/testrunner/testrunner-leaks.txt  2008-05-05 18:50:48 +0000
>> +++ src/zope/testing/testrunner/testrunner-leaks.txt  2010-02-20 22:23:32 +0000
>> @@ -16,7 +16,7 @@
>>      >>> from zope.testing import testrunner
>>
>>      >>> sys.argv = 'test --layer Layer11$ --layer Layer12$ -N4 -r'.split()
>> -    >>> _ = testrunner.run(defaults)
>> +    >>> _ = testrunner.run_internal(defaults)
>
> I'm curious about this change.
>

Sidnei explained it correctly. This particular file is only tested if
you're running a Python with --with-pydebug set, so it's easy to miss
when you make a change.

jml