zope/testing/testrunner/runner.py's order_by_bases() function orders layers by inheritance, basically. So if you've got a FunctionalTestLayer that inherits from some BasicTestLayer, the tests that use the basic test layer will be executed before the functional test layer ones. If there's no inheritance, the layers are sorted by name. Quite sane. Problem: I use grok. Which does some magic that probably throws off the code coverage reports. Quite some code appears untested, even though I *do* test it. The problem disappears if just import the python module first in some basic test before the grokking happened. I use z3c.testsetup: - My basic doctests end up on the zope.testing.testrunner.layer.UnitTests layer. - My functional doctests on the z3c.testsetup.functional.layer.DefaultZCMLLayer layer which inherits from zope.app.testing's ZCMLLayer. - zope.app.testing's ZCMLLayer does not inherit from anything, so also not from zope.testing's unittest layer. - 'zope....' is alphabetically behind 'z3c.testsetup....', so the functional layer is sorted in front of the basic unittest one. - So my grokked code appears untested. What's the best solution? Should zope.app.testing's ZCMLLayer inherit from zope.testing's UnitTests layer? Should z3c.testsetup do some inheriting of its own? Reinout -- Reinout van Rees - reinout@vanrees.org - http://reinout.vanrees.org Software developer at http://www.thehealthagency.com "Military engineers build missiles. Civil engineers build targets"
On Mon, Jul 13, 2009 at 09:27:54AM +0000, Reinout van Rees wrote:
zope/testing/testrunner/runner.py's order_by_bases() function orders layers by inheritance, basically. So if you've got a FunctionalTestLayer that inherits from some BasicTestLayer, the tests that use the basic test layer will be executed before the functional test layer ones.
If there's no inheritance, the layers are sorted by name. Quite sane.
Problem: I use grok. Which does some magic that probably throws off the code coverage reports. Quite some code appears untested, even though I *do* test it. The problem disappears if just import the python module first in some basic test before the grokking happened. I use z3c.testsetup:
- My basic doctests end up on the zope.testing.testrunner.layer.UnitTests layer.
- My functional doctests on the z3c.testsetup.functional.layer.DefaultZCMLLayer layer which inherits from zope.app.testing's ZCMLLayer.
- zope.app.testing's ZCMLLayer does not inherit from anything, so also not from zope.testing's unittest layer.
- 'zope....' is alphabetically behind 'z3c.testsetup....', so the functional layer is sorted in front of the basic unittest one.
- So my grokked code appears untested.
What's the best solution?
Whatever grok does that interferes with coverage should be fixed.
Should zope.app.testing's ZCMLLayer inherit from zope.testing's UnitTests layer?
Personally I would much prefer for the unit test layer to be sorted first. It is already treated specially by zope.testing; I see no harm in hardcoding its sort order.
Should z3c.testsetup do some inheriting of its own?
That to me sounds like the wrong place to fix this. Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development
On 2009-07-13, Marius Gedminas <marius@gedmin.as> wrote:
Whatever grok does that interferes with coverage should be fixed.
I did some more debugging. In the end it boils down to this: With z3c.testsetup, you can specify a zcml file at the top of your test file. z3c.testsetup creates a ZCMLLayer with that zcml file. I my test files, I had two separate zcml files (one registered a bit more than the other). So I ended up with two separate ZCMLLayer subclasses. And both did some grokking. So apparently the teardown of a ZCMLLayer subclass when going from one to the next isn't that clean. There *is* a warning in zope.testing that teardown isn't fully supported. Personally, I'm not sure where the exact problem is. Grok does decorator-like wrapping (should not throw off the coverage testing) and plain component architecture registration (should not throw off the coverage testing either). Tearing down a component architecture layer? Code coverage problems don't seem too common. Probably an interaction between two problems. How I solved it: just use one zcml layer in z3c.testsetup and do some extra grokking in the test itself instead of in the second zcml file. Reinout -- Reinout van Rees - reinout@vanrees.org - http://reinout.vanrees.org Software developer at http://www.thehealthagency.com "Military engineers build missiles. Civil engineers build targets"
On Mon, Jul 13, 2009 at 12:27:50PM +0000, Reinout van Rees wrote:
On 2009-07-13, Marius Gedminas <marius@gedmin.as> wrote:
Whatever grok does that interferes with coverage should be fixed.
I did some more debugging. In the end it boils down to this:
With z3c.testsetup, you can specify a zcml file at the top of your test file. z3c.testsetup creates a ZCMLLayer with that zcml file.
I my test files, I had two separate zcml files (one registered a bit more than the other). So I ended up with two separate ZCMLLayer subclasses. And both did some grokking. So apparently the teardown of a ZCMLLayer subclass when going from one to the next isn't that clean. There *is* a warning in zope.testing that teardown isn't fully supported.
Ah, that one. zope.testing supports test layers that muck up the global state irrepairably (by letting the layer's tearDown method raise NotImplementedError) and continues running the subsequent test layers in a fresh and squeaky-clean subprocess. Unfortunately, a separate process means separate coverage data tracking, and currently zope.testing doesn't support merging coverage data from several processes. Fortunately, the NotImplementedError in ZCMLLayer's tearDown is just a precaution, 99% of the time it is sufficient to run CleanUp.tearDown to get the global state restored to its pristine condition. You can indicate that it is safe by passing allow_teardown=True to ZCMLLayer's constructor. As a result: * the tests will run marginally faster (no subprocess overhead) * you will be able to use pdb in your tests (zope.testing disables pdb in a subprocess since it wants exclusive control over stdout) * you will be able to get accurate coverage tracing. Now, how you can convince z3c.testsetup to pass allow_teardown=True to the ZCMLLayer it constructs, I don't know. I've never used z3c.testsetup (although it sounds like an interesting library and I should check it out some day). ZCMLLayer's allow_teardown=True badly needs more publicity. Or maybe a ruthless dictator, decreeing that it shall be on by default from some near-future date.
How I solved it: just use one zcml layer in z3c.testsetup and do some extra grokking in the test itself instead of in the second zcml file.
That will work until you decide to introduce a new layer. Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development
On Mon, Jul 13, 2009 at 3:36 PM, Marius Gedminas<marius@gedmin.as> wrote:
Fortunately, the NotImplementedError in ZCMLLayer's tearDown is just a precaution, 99% of the time it is sufficient to run CleanUp.tearDown to get the global state restored to its pristine condition.
Keep in mind that a number of commonly-used ZCML directives stamp interfaces on classes; these aren't cleaned up with the general tear-down. If each layer stamps the same interfaces on the same classes, the only issue is ensuring that unit tests run first. If, however, you have different layers that may apply different sets of interfaces to different classes, you're may be relying on the process boundary as part of the required isolation. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
On 2009-07-13, Marius Gedminas <marius@gedmin.as> wrote:
On Mon, Jul 13, 2009 at 12:27:50PM +0000, Reinout van Rees wrote:
I my test files, I had two separate zcml files (one registered a bit more than the other). So I ended up with two separate ZCMLLayer subclasses. And both did some grokking. So apparently the teardown of a ZCMLLayer subclass when going from one to the next isn't that clean. There *is* a warning in zope.testing that teardown isn't fully supported.
Ah, that one.
zope.testing supports test layers that muck up the global state irrepairably (by letting the layer's tearDown method raise NotImplementedError) and continues running the subsequent test layers in a fresh and squeaky-clean subprocess.
Unfortunately, a separate process means separate coverage data tracking, and currently zope.testing doesn't support merging coverage data from several processes.
Thanks a lot, now I finally understand the cause of the problem I've been seeing. It is OK to figure out *when* something goes wrong, but the *why* is more important. Thanks! Reinout -- Reinout van Rees - reinout@vanrees.org - http://reinout.vanrees.org Software developer at http://www.thehealthagency.com "Military engineers build missiles. Civil engineers build targets"
On 2009-07-13, Marius Gedminas <marius@gedmin.as> wrote:
zope.testing supports test layers that muck up the global state irrepairably (by letting the layer's tearDown method raise NotImplementedError) and continues running the subsequent test layers in a fresh and squeaky-clean subprocess.
Unfortunately, a separate process means separate coverage data tracking, and currently zope.testing doesn't support merging coverage data from several processes.
Fortunately, the NotImplementedError in ZCMLLayer's tearDown is just a precaution, 99% of the time it is sufficient to run CleanUp.tearDown to get the global state restored to its pristine condition. You can indicate that it is safe by passing allow_teardown=3DTrue to ZCMLLayer's constructor. As a result:
* the tests will run marginally faster (no subprocess overhead)
* you will be able to use pdb in your tests (zope.testing disables pdb in a subprocess since it wants exclusive control over stdout)
* you will be able to get accurate coverage tracing.
Now, how you can convince z3c.testsetup to pass allow_teardown=3DTrue to the ZCMLLayer it constructs, I don't know. I've never used z3c.testsetup (although it sounds like an interesting library and I should check it out some day).
z3c.testsetup is great as it cuts down on the amount of repeated test setup code. (iirc stuff like nose and py.test also try this). Uli (=z3c.testsetup author), what about changing z3c.testsetup's default in this regard? Reinout -- Reinout van Rees - reinout@vanrees.org - http://reinout.vanrees.org Software developer at http://www.thehealthagency.com "Military engineers build missiles. Civil engineers build targets"
On Mon, Jul 13, 2009 at 5:55 AM, Marius Gedminas<marius@gedmin.as> wrote:
Personally I would much prefer for the unit test layer to be sorted first. It is already treated specially by zope.testing; I see no harm in hardcoding its sort order.
+1 -- Benji York Senior Software Engineer Zope Corporation
participants (4)
-
Benji York -
Fred Drake -
Marius Gedminas -
Reinout van Rees