Hello, For a long time, I've wanted to have a subunit output formatter for zope.testing. Subunit is a language-generic streaming protocol for test results. Once a test runner can display subunit output, it becomes much easier to write tools to process output programmatically. We want to use it in the Launchpad project for analyzing test results and for distributing our test runs across multiple machines. Other projects use it to get Hudson integration, or to glue their multi-language test suite into one result. More information on subunit can be found at http://launchpad.net/subunit. I've written a patch that adds subunit support to zope.testing by adding a new formatter object. The patch comes complete with tests and a willingness to jump through whatever legal hoops might be required to get it to land. The patch is attached, and can also be found as a merge proposal on Launchpad at https://code.edge.launchpad.net/~jml/zope.testing/subunit-output-formatter/+... I look forward to your replies. jml
On Thu, Mar 11, 2010 at 08:52:11PM +0000, Jonathan Lange wrote:
For a long time, I've wanted to have a subunit output formatter for zope.testing. Subunit is a language-generic streaming protocol for test results. Once a test runner can display subunit output, it becomes much easier to write tools to process output programmatically. We want to use it in the Launchpad project for analyzing test results and for distributing our test runs across multiple machines. Other projects use it to get Hudson integration, or to glue their multi-language test suite into one result.
I like this. Incidentally, how do you integrate it with Hudson? subunit2junitxml, I assume? How exactly do you combine bin/test with subunit2junitxml -- I assume some care with respect to exit codes?
=== modified file 'src/zope/testing/testrunner/formatter.py' --- src/zope/testing/testrunner/formatter.py 2009-12-23 21:21:53 +0000 +++ src/zope/testing/testrunner/formatter.py 2010-03-11 20:29:26 +0000 ... + def __init__(self, options): + if subunit is None: + raise Exception("Requires subunit 0.0.4 or better") + if content is None: + raise Exception("Requires testtools XXX or better")
This seems a mite imprecise (unless testtools uses roman numerals for version numbering).
=== modified file 'src/zope/testing/testrunner/testrunner-leaks.txt' --- src/zope/testing/testrunner/testrunner-leaks.txt 2008-05-05 18:50:48 +0000 +++ src/zope/testing/testrunner/testrunner-leaks.txt 2010-02-20 22:23:32 +0000 @@ -16,7 +16,7 @@ >>> from zope.testing import testrunner
>>> sys.argv = 'test --layer Layer11$ --layer Layer12$ -N4 -r'.split() - >>> _ = testrunner.run(defaults) + >>> _ = testrunner.run_internal(defaults)
I'm curious about this change.
Running samplelayers.Layer11 tests: Set up samplelayers.Layer1 in 0.000 seconds. Set up samplelayers.Layer11 in 0.000 seconds. @@ -60,7 +60,7 @@ Let's look at an example test that leaks:
>>> sys.argv = 'test --tests-pattern leak -N4 -r'.split() - >>> _ = testrunner.run(defaults) + >>> _ = testrunner.run_internal(defaults) Running zope.testing.testrunner.layer.UnitTests tests:... Iteration 1 Ran 1 tests with 0 failures and 0 errors in 0.000 seconds. @@ -81,7 +81,7 @@ type (or class):
>>> sys.argv = 'test --tests-pattern leak -N5 -r -v'.split() - >>> _ = testrunner.run(defaults) + >>> _ = testrunner.run_internal(defaults) Running tests at level 1 Running zope.testing.testrunner.layer.UnitTests tests:... Iteration 1
+1 for the rest. Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development
On Thu, Mar 11, 2010 at 9:17 PM, Marius Gedminas <marius@gedmin.as> wrote:
On Thu, Mar 11, 2010 at 08:52:11PM +0000, Jonathan Lange wrote:
For a long time, I've wanted to have a subunit output formatter for zope.testing. Subunit is a language-generic streaming protocol for test results. Once a test runner can display subunit output, it becomes much easier to write tools to process output programmatically. We want to use it in the Launchpad project for analyzing test results and for distributing our test runs across multiple machines. Other projects use it to get Hudson integration, or to glue their multi-language test suite into one result.
I like this.
Thanks.
Incidentally, how do you integrate it with Hudson? subunit2junitxml, I assume? How exactly do you combine bin/test with subunit2junitxml -- I assume some care with respect to exit codes?
I haven't tried myself, but from analogy with other subunit tools: ./bin/test --subunit | subunit2junitxml The formatter doesn't control exit codes, so I didn't really think about them in this patch. Should I produce a follow up?
=== modified file 'src/zope/testing/testrunner/formatter.py' --- src/zope/testing/testrunner/formatter.py 2009-12-23 21:21:53 +0000 +++ src/zope/testing/testrunner/formatter.py 2010-03-11 20:29:26 +0000 ... + def __init__(self, options): + if subunit is None: + raise Exception("Requires subunit 0.0.4 or better") + if content is None: + raise Exception("Requires testtools XXX or better")
This seems a mite imprecise (unless testtools uses roman numerals for version numbering).
Oops :) I've changed it to say 0.9.2, and subunit to say 0.0.5, which is the actual version I've tested with.
=== modified file 'src/zope/testing/testrunner/testrunner-leaks.txt' --- src/zope/testing/testrunner/testrunner-leaks.txt 2008-05-05 18:50:48 +0000 +++ src/zope/testing/testrunner/testrunner-leaks.txt 2010-02-20 22:23:32 +0000 @@ -16,7 +16,7 @@ >>> from zope.testing import testrunner
>>> sys.argv = 'test --layer Layer11$ --layer Layer12$ -N4 -r'.split() - >>> _ = testrunner.run(defaults) + >>> _ = testrunner.run_internal(defaults)
I'm curious about this change.
Sidnei explained it correctly. This particular file is only tested if you're running a Python with --with-pydebug set, so it's easy to miss when you make a change. jml
On Thu, Mar 11, 2010 at 10:38:15PM +0000, Jonathan Lange wrote:
On Thu, Mar 11, 2010 at 9:17 PM, Marius Gedminas <marius@gedmin.as> wrote:
Incidentally, how do you integrate it with Hudson? subunit2junitxml, I assume? How exactly do you combine bin/test with subunit2junitxml -- I assume some care with respect to exit codes?
I haven't tried myself, but from analogy with other subunit tools: ./bin/test --subunit | subunit2junitxml
Ah, nice. I was afraid that a naive attempt of something like bin/test --subunit > result.subunit subunit2junitxml result.subunit -o result.xml would break because Hudson runs the build step as a shell script with sh -e and therefore bin/test returning a non-zero exit code would abort the script without running subunit2junitxml. But a pipeline ignores the exit code of all but the very last process. Incidentally, does subunit2junitxml return a non-zero exit code when there are failing tests? I tried to easy_install subunit to experiment, but apparently subunit is not registered on PyPI (!?)
The formatter doesn't control exit codes, so I didn't really think about them in this patch. Should I produce a follow up?
It's not really in scope for this patch, but given the rising popularity of Hudson I think it would be nice to have a small README.hudson in zope.testing explaining the integration.
On Thu, Mar 11, 2010 at 08:52:11PM +0000, Jonathan Lange wrote:
=== modified file 'src/zope/testing/testrunner/testrunner-leaks.txt' --- src/zope/testing/testrunner/testrunner-leaks.txt 2008-05-05 18:50:48 +0000 +++ src/zope/testing/testrunner/testrunner-leaks.txt 2010-02-20 22:23:32 +0000 @@ -16,7 +16,7 @@ >>> from zope.testing import testrunner
>>> sys.argv = 'test --layer Layer11$ --layer Layer12$ -N4 -r'.split() - >>> _ = testrunner.run(defaults) + >>> _ = testrunner.run_internal(defaults)
I'm curious about this change.
Sidnei explained it correctly. This particular file is only tested if you're running a Python with --with-pydebug set, so it's easy to miss when you make a change.
Ah! That explains why it wasn't fixed on trunk. (This bit should probably be committed separately. And zope.testing could use a bit more continuous integration with various Python versions/build flags...) Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development
On Fri, Mar 12, 2010 at 3:09 PM, Marius Gedminas <marius@gedmin.as> wrote:
It's not really in scope for this patch, but given the rising popularity of Hudson I think it would be nice to have a small README.hudson in zope.testing explaining the integration.
Well, if all you want is Hudson output and you use buildout, you could just use: http://pypi.python.org/pypi/collective.xmltestreport And have a testrunner which does it all for you. Hanno
On Fri, Mar 12, 2010 at 11:09 AM, Marius Gedminas <marius@gedmin.as> wrote:
On Thu, Mar 11, 2010 at 08:52:11PM +0000, Jonathan Lange wrote:
=== modified file 'src/zope/testing/testrunner/testrunner-leaks.txt' --- src/zope/testing/testrunner/testrunner-leaks.txt 2008-05-05 18:50:48 +0000 +++ src/zope/testing/testrunner/testrunner-leaks.txt 2010-02-20 22:23:32 +0000 @@ -16,7 +16,7 @@ >>> from zope.testing import testrunner
>>> sys.argv = 'test --layer Layer11$ --layer Layer12$ -N4 -r'.split() - >>> _ = testrunner.run(defaults) + >>> _ = testrunner.run_internal(defaults)
I'm curious about this change.
Sidnei explained it correctly. This particular file is only tested if you're running a Python with --with-pydebug set, so it's easy to miss when you make a change.
Ah! That explains why it wasn't fixed on trunk.
(This bit should probably be committed separately. And zope.testing could use a bit more continuous integration with various Python versions/build flags...)
I committed this change as a separate revision, and merged the rest of the patch after that. I guess I should make a 3.9.0 release now. Any objections? -- Sidnei
On Fri, Mar 12, 2010 at 12:00:42PM -0300, Sidnei da Silva wrote:
I committed this change as a separate revision, and merged the rest of the patch after that. I guess I should make a 3.9.0 release now. Any objections?
New releases are Good. Incidentally, what happened to the zope.testing.doctest deprecation? AFAIC zope.testing itself still triggers those deprecation warnings when you import modules like zope.testing.renormalizer. I'm not entirely sure what the deprecation story is all about (well, okay, it's about reducing the number of forks and opening the way to Python 3 compatibility), but I'm a bit uneasy leaving it in such a halfway state... On the other hand people worked hard on it, and I wouldn't want to throw their work away by un-deprecating zope.testing.doctest. Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development
* 2010-03-13 16:16, Marius Gedminas wrote:
I'm not entirely sure what the deprecation story is all about (well, okay, it's about reducing the number of forks and opening the way to Python 3 compatibility), but I'm a bit uneasy leaving it in such a halfway state... On the other hand people worked hard on it, and I wouldn't want to throw their work away by un-deprecating zope.testing.doctest.
As one of the people who worked hard to clean out the situation, I'm okey with the un-deprecation. Thanks, Fabio
Hi Jonathan, On Thu, Mar 11, 2010 at 5:52 PM, Jonathan Lange <jml@mumak.net> wrote:
Hello,
For a long time, I've wanted to have a subunit output formatter for zope.testing. Subunit is a language-generic streaming protocol for test results. Once a test runner can display subunit output, it becomes much easier to write tools to process output programmatically. We want to use it in the Launchpad project for analyzing test results and for distributing our test runs across multiple machines. Other projects use it to get Hudson integration, or to glue their multi-language test suite into one result. More information on subunit can be found at http://launchpad.net/subunit.
I've written a patch that adds subunit support to zope.testing by adding a new formatter object. The patch comes complete with tests and a willingness to jump through whatever legal hoops might be required to get it to land. The patch is attached, and can also be found as a merge proposal on Launchpad at https://code.edge.launchpad.net/~jml/zope.testing/subunit-output-formatter/+...
I look forward to your replies.
The patch looks great to me, specially the testing coverage and the fact it is completely optional to use subunit. Big +1 from me. @Marius: re: run() vs run_internal(), the former would cause a sys.exit() at some point, so run_internal() was added. I didn't spot this change, but it looks fine to me. I volunteer for commiting this patch and making a new major release, if no one else volunteers. -- Sidnei
participants (5)
-
Fabio Tranchitella -
Hanno Schlichting -
Jonathan Lange -
Marius Gedminas -
Sidnei da Silva