Zope test layers, pytest, and test isolation
Hi there, Some of you might have noticed that some time ago the first version of `zope.pytest` was released: http://pypi.python.org/pypi/zope.pytest It's a try to make usage of `pytest` more comfortable in Zope-based environments. `zope.pytest` is mainly based on Martijns, Jan-Wijbrands, and Christian Klingers efforts. I put in some bits too. Right now we have a problem with pytest integration when it comes to ZCA setups: among other things `zope.pytest` offers a pytest-funcarg__ compatible function to automate ZCML based fixtures, i.e. it automatically adds setup and teardown functionality before/after test functions if requested by the test writer like this:: import my.project from zope.pytest import configure from zope.component import queryUtility def pytest_funcarg__config(request): return configure(request, my.project, 'ftesting.zcml') def test_myutil_registered(config): util = queryUtility( my.project.interfaces.ISomeIface, name = 'myutil', default = None) assert util is not None Here for test functions requiring an argument named ``config`` it is guaranteed that before the test function is run a ZCA initialization is performed based on the ZCML configuration given in the ``ftesting.zcml`` of `my.project`. This configuration is cached for the whole test session (the complete test run including all tests). If we assume that the used `ftesting.zcml` provides a named utility named ``myutil`` then the call to ``configure`` in the funcarg__-function should perform that registration and the 'myutil_registered' test should succeed. All that works very well -- in principle. The problem is: the `configure` function calls the registry setup before any real test is run and performs a teardown only after the last of all tests was run. And that leads to side-effects. In other words: all tests share the same global ZCA registrations and changes to the registrations in one test will affect other tests run thereafter. We have a lack of test isolation. Example:: import my.project from zope.pytest import configure from zope.component import queryUtility, provideUtility from zope.interface import Interface def pytest_funcarg__config(request): return configure(request, my.project, 'ftesting.zcml') def test_1(config): util = object() provideUtility(util, provides=Interface, name='a_util') def test_2(config): util = queryUtility(Interface, name='a_util', default=None) assert util is None def test_3(): util = queryUtility(Interface, name='a_util', default=None) assert util is None Here the second test will fail. Even worse, also the third test will fail, which does not require a ZCA setup at all. That's because the global registry is setup before all tests and not torn down before the last test was run. This behaviour is partly intentional: we do not want to setup/teardown the ZCA before/after single test functions as this would increase time consumed by test runs dramatically. Instead we want to reduce the number of ZCA setups/teardown as far as possible. We _could_ do a per testfunction setup/teardown by simply using a different test scope in `configure`. `pytest` offers three kinds of scope for that purpose: `session`, `module`, and `function`. `session` is the current default. Using `function` as scope we would get complete test isolation but pretty long lasting test runs. That could stop people from writing tests. Something we would like to avoid by all means. We could instead use `module` as default scope, so that all test functions inside a module would share one ZCA configuration (and for different test modules the setup would be performed from scratch). This might be a reasonable compromise. Test isolation of single tests inside a test module would of course be broken but one could say: if you want a fresh registry, just put your tests inside a new test module. With Zope test layers there might be a similar problem (each layer can create a global ZCA configuration that then will be shared amongst all applied test methods) but I think people are used to it and can cope with that specific test isolation breakage. A big advantage of test layers over `pytest` testing scopes might be that you can spread your tests associated to a certain layer over many files/modules/packages as you like and the setup/teardown will nevertheless only be performed once for each layer (well, normally at least). If you have one layer and that's enough for you, you will only have one ZCA-setup/teardown in the whole testrun. That's fast and certainly fits many real-world use-cases. Compared to Zope test layers I came to the conclusion that there is not much like this concept already in `pytest` and the behaviour of test layers can not easily be faked. `pytest` provides only the three mentioned scopes as kind of 'natural' layers. Spreading fixtures over many modules (as layers easily do) might contradict with the basic design goals of pytest and I am pretty sure that Holge Krekel wouldn't like it. Overall, we're now looking for a satisfying solution in terms of runtime and isolation. Some of the questions that arise: Would it make sense to bring the Zope layer concept into `pytest`? Are there already possibilities to mimic testlayer-like behaviour in `pytest` which we simply overlooked? Are there cheap/fast ways to cache/restore registry setups we hadn't had on the screen? Really fast setups/cache-restores could make even function-wise registrations a considerable thing. Would it simply be okay to use the 'module' scope for registration setups? Or do you have completely different ideas how to solve that issue? Any comments are really appreciated! Best regards, -- Uli
Hi. Disclaimer: I don't know anything about pytest. On Thu, Mar 24, 2011 at 1:05 AM, Uli Fouquet <uli@gnufix.de> wrote:
Are there cheap/fast ways to cache/restore registry setups we hadn't had on the screen? Really fast setups/cache-restores could make even function-wise registrations a considerable thing.
The approach we take in the Plone world (though plone.testing has no Plone or Zope 2 dependencies) is: http://pypi.python.org/pypi/plone.testing/4.0a3#configuration-registry-sandb... As part of plone.testing we have proper layer based ZCA isolation, by doing stacked component registries. This approach is extremely fast, as you only need to push/pop a registry to/from the stack for each test. If you want to use layers, you can use this. Otherwise you might be able to take the ideas and code. Hanno
Hi there, [extensive introduction by Uli of zope.pytest] Thanks Uli for that introduction to the package and its problems! I've been using zope.pytest within 2 Grok projects so far to test through webob some JSON webservices, and it's worked well for me. I hadn't had to deal with the issue of multiple configurations in those tests though, as they're basically functional/integration tests. I've also been happily using plain py.test in several other non-Zope projects, and I must say it has many nice features that make life easier, and makes a lot of test cruft go away. I'm particularly hoping that Wolfgang Schnerring will give feedback: I'm curious how his investigations on improving the layer story might also benefit py.test integration. Regards, Martijn P.S. Uli did a lot of work himself to make this package and help document it, by the way, he's doing himself injustice in his introduction. :)
Hi Uli, I am happy that some people started to look what is needed to make modular component registrations with py.test. The long explanation that you wrote made it easy for me to jump into funcargs. Those last months, I have been diving into the great plone.testing package written by Martin Aspeli. Among others, plone.testing has a very nice module zca.py to ease building zope.testing layers that need ZCA. See https://dev.plone.org/plone/browser/plone.testing/trunk/src/plone/testing/zc... Specifically, pushGlobalRegistry and popGlobalRegistry are a pair of functions that allow to make registrations step by step, while easily throwing away the latest registrations when they are not needed anymore. plone.testing does not depend on Zope2 even if it comes with support for building Zope2 layers. Le 24/03/11 01:05, Uli Fouquet a écrit :
Compared to Zope test layers I came to the conclusion that there is not much like this concept already in `pytest` and the behaviour of test layers can not easily be faked. `pytest` provides only the three mentioned scopes as kind of 'natural' layers. Spreading fixtures over many modules (as layers easily do) might contradict with the basic design goals of pytest and I am pretty sure that Holge Krekel wouldn't like it.
I do not agree with that last statement : the fact that resources can be cached during a session allows them to be reused over many modules. IOW, it does not contradict pytest design. Digging into funcargs gives me the feeling that they are much richer than layers. Because funcargs are functions that get access to a lot of context, they allow for more flexibility than layers that are only static resources.
Overall, we're now looking for a satisfying solution in terms of runtime and isolation. Some of the questions that arise:
My feeling is that the extrakey argument of cached_setup can be used in combination with code similar to plone.testing ZCA support to build something "satisfying in terms of runtime and isolation."
Would it make sense to bring the Zope layer concept into `pytest`?
I think what is already available in py.test might avoid the need of layers.
Are there already possibilities to mimic testlayer-like behaviour in `pytest` which we simply overlooked?
See above.
Are there cheap/fast ways to cache/restore registry setups we hadn't had on the screen? Really fast setups/cache-restores could make even function-wise registrations a considerable thing.
Would it simply be okay to use the 'module' scope for registration setups?
Or do you have completely different ideas how to solve that issue?
Any comments are really appreciated!
Best regards,
-- Uli
-- Godefroid Chapelle (aka __gotcha) http://bubblenet.be
Hello Uli, I've spent quite some time thinking (and partly coding) about the same issues you mention (but didn't feel ready to talk about it here, yet), so I'm glad that maybe we can start thinking about them together I think your email addresses two quite different topics, so I'll split my reply. First up: test support for zope.component. Second part: about the concept of test layers. * Uli Fouquet <uli@gnufix.de> [2011-03-24 01:05]:
Right now we have a problem with pytest integration when it comes to ZCA setups [...] all tests share the same global ZCA registrations and changes to the registrations in one test will affect other tests run thereafter. We have a lack of test isolation.
Exactly. This issue has bitten me too in various places, and as far as I know there are no solutions for it, yet. What I envision to solve this issue is that test support for zope.component should work the same way as with the ZODB. There, we have a *stack* of Databases (DemoStorages, to be precise) that wrap each other, where each one delegates reads downwards, but keeps writes for itself. So you might have one database for the layer that provides the baseline, and each test (in its setUp) gets its own database where it can do whatever it wants, because it is thrown away in its tearDown. In principle, quite a few of the mechanics to do the same thing with zope.component registries are already there (since a registry keeps a list of base registries it will delegate to when something can not be found in itself). And as Hanno and Godefroid mentioned, plone.testing does something in this direction already. (And, it bears repeating, in its core has no dependencies on Plone or Zope2.) But as far as I see, there are issues that plone.testing does not address: 1. I've been going over this stuff with my colleague Thomas Lotze, and we realized that just squeezing in a new registry and bending its bases to the previously active one is not enough for full isolation, since this does not cover *deleting* registrations (one, you can only delete the registration from the precise registry it was created in, and two, in the just-bend-the-bases approach, once you delete a registration, it's gone forever). I think to provide full isolation, we need to make *copies*. And since zope.component in general supports a chain of registries, we probably need to make copies of each of them. 2. zope.component has two entry points, the global site registry and the current registry (getGlobalSiteManager and getSiteManager). The current registry can be anything, or more precisely, you can call getSiteManager.sethook(callable) and provide a callable that returns the current registry. I think to provide test support for zope.component (i. e. generally, at the "library level"), we need to support both entry points. The global one is not hard, but the getSiteManager one gets nasty really fast, because of course we have to rely on bending getSiteManager to return the current "test registry" -- but anyone could at any time call getSiteManager.sethook to change it! Which means we need to intercept that and a) prevent our hook from being replaced and b) inject the new registry into the test stack somehow. As far as I understand, plone.testing sidesteps these issues by only dealing with the global registry, and specially munging the two known cases in the Zope world where getSiteManager is changed (zope.site and five.something). ** I'd like to know what people think about this plan. Thomas and I have been over this quite a bit and think it's sound, not overly complicated, and (after we did some experiments) definitely doable. Please do point out stuff we missed. :-) I'd very much like to put this functionality into zope.component itself, which of course raises backwards compatibility issues galore, but any code for this definitely isn't wasted since we can always package it separately if we don't find a way to integrate it. Thomas and I taken up implementing this, but we can't devote a lot of time to it (about one session per week), so realistically I'm afraid I guess it will take a few months until we have something of substance. So if there are people who want to pitch in, that'd be great. I definitely could write up a more detailed plan and maybe even formulate smaller chunks so we could go at this with more people. Wolfgang -- Wolfgang Schnerring · ws@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Fri, Mar 25, 2011 at 4:24 AM, Wolfgang Schnerring <ws@gocept.com> wrote:
Hello Uli,
I've spent quite some time thinking (and partly coding) about the same issues you mention (but didn't feel ready to talk about it here, yet), so I'm glad that maybe we can start thinking about them together
I think your email addresses two quite different topics, so I'll split my reply. First up: test support for zope.component. Second part: about the concept of test layers.
* Uli Fouquet <uli@gnufix.de> [2011-03-24 01:05]:
Right now we have a problem with pytest integration when it comes to ZCA setups [...] all tests share the same global ZCA registrations and changes to the registrations in one test will affect other tests run thereafter. We have a lack of test isolation.
Exactly. This issue has bitten me too in various places, and as far as I know there are no solutions for it, yet.
The classic solution is to start tests with empty registries, or, if you're using layers, with some baseline registries.
What I envision to solve this issue is that test support for zope.component should work the same way as with the ZODB. There, we have a *stack* of Databases (DemoStorages, to be precise) that wrap each other, where each one delegates reads downwards, but keeps writes for itself. So you might have one database for the layer that provides the baseline, and each test (in its setUp) gets its own database where it can do whatever it wants, because it is thrown away in its tearDown.
In principle, quite a few of the mechanics to do the same thing with zope.component registries are already there (since a registry keeps a list of base registries it will delegate to when something can not be found in itself). And as Hanno and Godefroid mentioned, plone.testing does something in this direction already. (And, it bears repeating, in its core has no dependencies on Plone or Zope2.)
I like the idea of stacking registries.
But as far as I see, there are issues that plone.testing does not address:
1. I've been going over this stuff with my colleague Thomas Lotze, and we realized that just squeezing in a new registry and bending its bases to the previously active one is not enough for full isolation, since this does not cover *deleting* registrations (one, you can only delete the registration from the precise registry it was created in, and two, in the just-bend-the-bases approach, once you delete a registration, it's gone forever).
I think to provide full isolation, we need to make *copies*. And since zope.component in general supports a chain of registries, we probably need to make copies of each of them.
Is deleting registrations important? This seems like an odd use case. If it's needed, I would suggest starting with a baseline (e.g. stack) that doesn't include the component you want to test deleting, then adding in setup.
2. zope.component has two entry points, the global site registry and the current registry (getGlobalSiteManager and getSiteManager). The current registry can be anything, or more precisely, you can call getSiteManager.sethook(callable) and provide a callable that returns the current registry.
I think to provide test support for zope.component (i. e. generally, at the "library level"), we need to support both entry points.
Why? Why would someone care about anything other than the current effective configuration.
The global one is not hard, but the getSiteManager one gets nasty really fast, because of course we have to rely on bending getSiteManager to return the current "test registry"
But as you point out, there's a hook for that.
-- but anyone could at any time call getSiteManager.sethook to change it!
Seriously? Nobody calls that but deep infrastructure code.
Which means we need to intercept that and a) prevent our hook from being replaced and b) inject the new registry into the test stack somehow.
I think you're making this more complicated than it needs to be.
As far as I understand, plone.testing sidesteps these issues by only dealing with the global registry, and specially munging the two known cases in the Zope world where getSiteManager is changed (zope.site and five.something).
**
I'd like to know what people think about this plan. Thomas and I have been over this quite a bit and think it's sound, not overly complicated, and (after we did some experiments) definitely doable. Please do point out stuff we missed. :-)
I think a stack-based approach is very appealing. I think anything more complex is likely to cause more problems than it solves.
I'd very much like to put this functionality into zope.component itself, which of course raises backwards compatibility issues galore,
Not sure why this would have to be backward incompatible, but I'm unconvinced that the complexity comes close to being justified by the benefit.
but any code for this definitely isn't wasted since we can always package it separately if we don't find a way to integrate it.
Thomas and I taken up implementing this, but we can't devote a lot of time to it (about one session per week), so realistically I'm afraid I guess it will take a few months until we have something of substance.
OMG and then who gets to maintain it? Not it! You and Thomas have obviously thought a lot about this and I appreciate the effort you've put into this, but I really don't think it's worth the added complexity. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton
Hi, On 25 March 2011 13:17, Jim Fulton <jim@zope.com> wrote:
On Fri, Mar 25, 2011 at 4:24 AM, Wolfgang Schnerring <ws@gocept.com> wrote:
Hello Uli,
I've spent quite some time thinking (and partly coding) about the same issues you mention (but didn't feel ready to talk about it here, yet), so I'm glad that maybe we can start thinking about them together
I think your email addresses two quite different topics, so I'll split my reply. First up: test support for zope.component. Second part: about the concept of test layers.
* Uli Fouquet <uli@gnufix.de> [2011-03-24 01:05]:
Right now we have a problem with pytest integration when it comes to ZCA setups [...] all tests share the same global ZCA registrations and changes to the registrations in one test will affect other tests run thereafter. We have a lack of test isolation.
Exactly. This issue has bitten me too in various places, and as far as I know there are no solutions for it, yet.
The classic solution is to start tests with empty registries, or, if you're using layers, with some baseline registries.
plone.testing (which is Plone non-specific and will shortly be BSD licensed) allows for stacking of ZCA registries. It has to do some ugly hacking to achieve this, since zope.component stores handles to the global registry in *three* different modules and update them in weird ways depending on the registry hooking, but it's well tested and robust now. See http://dev.plone.org/plone/browser/plone.testing/trunk/src/plone/testing/zca... and http://dev.plone.org/plone/browser/plone.testing/trunk/src/plone/testing/zca....
What I envision to solve this issue is that test support for zope.component should work the same way as with the ZODB. There, we have a *stack* of Databases (DemoStorages, to be precise) that wrap each other, where each one delegates reads downwards, but keeps writes for itself. So you might have one database for the layer that provides the baseline, and each test (in its setUp) gets its own database where it can do whatever it wants, because it is thrown away in its tearDown.
In principle, quite a few of the mechanics to do the same thing with zope.component registries are already there (since a registry keeps a list of base registries it will delegate to when something can not be found in itself). And as Hanno and Godefroid mentioned, plone.testing does something in this direction already. (And, it bears repeating, in its core has no dependencies on Plone or Zope2.)
I like the idea of stacking registries.
plone.testing implements this. If we could fix zope.component to make the implementation less ugly, that'd be a big win.
But as far as I see, there are issues that plone.testing does not address:
1. I've been going over this stuff with my colleague Thomas Lotze, and we realized that just squeezing in a new registry and bending its bases to the previously active one is not enough for full isolation, since this does not cover *deleting* registrations (one, you can only delete the registration from the precise registry it was created in, and two, in the just-bend-the-bases approach, once you delete a registration, it's gone forever).
Correct, although this is in practice extremely rare. I'd say it's much better to control setup more carefully and not have to "undo" in a child layer.
I think to provide full isolation, we need to make *copies*. And since zope.component in general supports a chain of registries, we probably need to make copies of each of them.
Copying is very hard. It was my first attempt in plone.testing and didn't work out well. You need to support pickling of registries for local/persistent component registries. I cannot begin to tell you how many weird pickling errors I found and had to work around.
Is deleting registrations important? This seems like an odd use case. If it's needed, I would suggest starting with a baseline (e.g. stack) that doesn't include the component you want to test deleting, then adding in setup.
+1
2. zope.component has two entry points, the global site registry and the current registry (getGlobalSiteManager and getSiteManager). The current registry can be anything, or more precisely, you can call getSiteManager.sethook(callable) and provide a callable that returns the current registry.
I think to provide test support for zope.component (i. e. generally, at the "library level"), we need to support both entry points.
Why? Why would someone care about anything other than the current effective configuration.
Agree. There is a problem in that provideAdapter() and friends don't use getSiteManager() - the always use the global site manager. And there are parts of zope.component that use module level variables directly, ignoring hooks.
The global one is not hard, but the getSiteManager one gets nasty really fast, because of course we have to rely on bending getSiteManager to return the current "test registry"
But as you point out, there's a hook for that.
I don't think you need to do that. plone.testing doesn't, for sure, it just changes the variable that getSiteManager() looks at. You do need to be careful about when you set hooks and when you don't. Again, plone.testing is the result of hours and hours of finding weird problems, so I'd recommend you don't discard the knowledge there. I think a best case scenario would be for plone.testing.zca to just be a delegate for something more robust in zope.component.testing.
-- but anyone could at any time call getSiteManager.sethook to change it!
Seriously? Nobody calls that but deep infrastructure code.
People do call zope.site.hooks.setHooks() sometimes, though, e.g. upon traversal. The getSiteManager.sethook() functino is of course voodoo that no-one knows about.
Which means we need to intercept that and a) prevent our hook from being replaced and b) inject the new registry into the test stack somehow.
I think you're making this more complicated than it needs to be.
Agree with Jim. Are you going to stop people monkey patching getSiteManager() too? ;-)
As far as I understand, plone.testing sidesteps these issues by only dealing with the global registry, and specially munging the two known cases in the Zope world where getSiteManager is changed (zope.site and five.something).
**
I'd like to know what people think about this plan. Thomas and I have been over this quite a bit and think it's sound, not overly complicated, and (after we did some experiments) definitely doable. Please do point out stuff we missed. :-)
I think a stack-based approach is very appealing. I think anything more complex is likely to cause more problems than it solves.
Agree. The DemoStorage analogy is a useful one. Another advantage is that the work is already done. :) I think the implementation in plone.testing has some warts caused by a lack of support in zope.component. The number one win for me would be to get rid of the proliferation of module level variables. Number two would be the __reduce__ hack to make unpickling of local componenet registries (which hold a reference to the global registry in their __bases__) work without monkey patching.
I'd very much like to put this functionality into zope.component itself, which of course raises backwards compatibility issues galore,
Not sure why this would have to be backward incompatible, but I'm unconvinced that the complexity comes close to being justified by the benefit.
I think some tidying of zope.component internals would mean pushing some of the plone.testing.zca stuff down a level to zope.component feasible, which would have no BBB issues except perhaps for plone.testing itself, which we could manage quite easily.
but any code for this definitely isn't wasted since we can always package it separately if we don't find a way to integrate it.
Thomas and I taken up implementing this, but we can't devote a lot of time to it (about one session per week), so realistically I'm afraid I guess it will take a few months until we have something of substance.
OMG and then who gets to maintain it? Not it!
You and Thomas have obviously thought a lot about this and I appreciate the effort you've put into this, but I really don't think it's worth the added complexity.
I would suggest a good starting point would be to see if you can make plone.testing.zca "clean". It should be pretty obvious what is currently a hack. If you got there, you effectively have stacking of component registries. plone.testing would be simpler, and zope.component.testing would be able to provide a simple way to stack configuration by layer without a plone.testing dependency. If you ignore the "delete" use case (which I think is a non-use case), I think this is quite achievable in a sprint, say. Martin
On Fri, Mar 25, 2011 at 9:58 AM, Martin Aspeli <optilude+lists@gmail.com> wrote:
On 25 March 2011 13:17, Jim Fulton <jim@zope.com> wrote:
On Fri, Mar 25, 2011 at 4:24 AM, Wolfgang Schnerring <ws@gocept.com> wrote:
...
2. zope.component has two entry points, the global site registry and the current registry (getGlobalSiteManager and getSiteManager). The current registry can be anything, or more precisely, you can call getSiteManager.sethook(callable) and provide a callable that returns the current registry.
I think to provide test support for zope.component (i. e. generally, at the "library level"), we need to support both entry points.
Why? Why would someone care about anything other than the current effective configuration.
Agree. There is a problem in that provideAdapter() and friends don't use getSiteManager() - the always use the global site manager. And there are parts of zope.component that use module level variables directly, ignoring hooks.
These are meant to work this way. If you want to do local configuration, you should explictly select the relevent registry/manager or call getSiteManager and use the result.
-- but anyone could at any time call getSiteManager.sethook to change it!
Seriously? Nobody calls that but deep infrastructure code.
People do call zope.site.hooks.setHooks() sometimes, though, e.g. upon traversal.
This was never meant to be an application-level feature. I find the notion that people would call these dureing traversal to be disturbing. Are you sure you're not confusing this with setSite? Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton
Hi Jim, On 25 March 2011 14:12, Jim Fulton <jim@zope.com> wrote:
Agree. There is a problem in that provideAdapter() and friends don't use getSiteManager() - the always use the global site manager. And there are parts of zope.component that use module level variables directly, ignoring hooks.
These are meant to work this way.
If you want to do local configuration, you should explictly select the relevent registry/manager or call getSiteManager and use the result.
I sortof understand, but it makes it impossible to let people use provideAdapter() & co in test setup and still retain some kind of layered test setup, without the kind of hacks we do in plone.testing.
-- but anyone could at any time call getSiteManager.sethook to change it!
Seriously? Nobody calls that but deep infrastructure code.
People do call zope.site.hooks.setHooks() sometimes, though, e.g. upon traversal.
This was never meant to be an application-level feature. I find the notion that people would call these dureing traversal to be disturbing. Are you sure you're not confusing this with setSite?
Sorry, I meant setSite() above yes. Although sometimes people call setHooks() and then setSite(site) in test setup, because setSite() doesn't work until setHooks() has been called once. I think this may sometimes just be cargo-cult, though. Martin
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/25/2011 09:58 AM, Martin Aspeli wrote:
I would suggest a good starting point would be to see if you can make plone.testing.zca "clean". It should be pretty obvious what is currently a hack. If you got there, you effectively have stacking of component registries. plone.testing would be simpler, and zope.component.testing would be able to provide a simple way to stack configuration by layer without a plone.testing dependency.
If you ignore the "delete" use case (which I think is a non-use case), I think this is quite achievable in a sprint, say.
+1 to this approach. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2MpBsACgkQ+gerLs4ltQ7QQACgycVfipd/FvF6J8swGRJpZ2M9 SzcAniu9uvBS58WDcgxTxxllY+q8UmNX =Ljpu -----END PGP SIGNATURE-----
Hello, * Martin Aspeli <optilude+lists@gmail.com> [2011-03-25 13:58]:
plone.testing (which is Plone non-specific and will shortly be BSD licensed) allows for stacking of ZCA registries. [...] Again, plone.testing is the result of hours and hours of finding weird problems, so I'd recommend you don't discard the knowledge there. I think a best case scenario would be for plone.testing.zca to just be a delegate for something more robust in zope.component.testing.
I wasn't aware plone.testing had something for the ZCA, before I read that in this thread yesterday. I'm glad to hear that, and I certainly don't want to discard anything. But from reading the code and what you wrote, I get the impression that plone.testing does not yet solve the issue completely, precisely because of the two points I brought up and that continue below:
On 25 March 2011 13:17, Jim Fulton <jim@zope.com> wrote:
On Fri, Mar 25, 2011 at 4:24 AM, Wolfgang Schnerring <ws@gocept.com> wrote:
we realized that just squeezing in a new registry and bending its bases to the previously active one is not enough for full isolation, since this does not cover *deleting* registrations
Is deleting registrations important? This seems like an odd use case. If it's needed, I would suggest starting with a baseline (e.g. stack) that doesn't include the component you want to test deleting, then adding in setup.
I've wanted to specifically nuke registrations sometimes, the pattern being, load the package's configure.zcml in the layer, and then delete something (to demonstrate some error handling behaviour). I agree that this use case is rare, but I'm not sure it is rare enough that we should ignore it. I need to think about this some more, though.
2. zope.component has two entry points, the global site registry and the current registry (getGlobalSiteManager and getSiteManager). The current registry can be anything, or more precisely, you can call getSiteManager.sethook(callable) and provide a callable that returns the current registry.
I think to provide test support for zope.component (i. e. generally, at the "library level"), we need to support both entry points.
Why? Why would someone care about anything other than the current effective configuration.
Because that's the API that zope.component offers, it conceptually deals with *two* registries: the one you get from getSiteManager and the one you get from getGlobalSiteManager. I agree that it is unlikely that client code will *read* registrations using getGlobalSiteManager -- since most everyone uses zope.component.getUtility etc. (which in turn uses getSiteManager). But *writing* is a different matter. Just one example: zope.component.provideUtility etc. uses getGlobalSiteManager, while the ZCML directives use getSiteManager. At least as long as zope.component.provide* uses getGlobalSiteManager instead of getSiteManager, I maintain that test infrastructure needs to 1. wrap the global registry 2. wrap whatever getSiteManager currently returns (Otherwise we might be able get away with not doing (1), but I think we'll always need to do (2).) plone.testing has it the other way around, doing (1) but not (2), as Martin says himself...
The global one is not hard, but the getSiteManager one gets nasty really fast, because of course we have to rely on bending getSiteManager to return the current "test registry"
I don't think you need to do that. plone.testing doesn't, for sure, it just changes the variable that getSiteManager() looks at.
... and I'm quite sure that's wrong, becasue it doesn't deal with the issue of getSiteManager at the level of zope.component, but at the level of some specific implementations (zope.site and five.localsitemanager, to be precise). When I'm using, say, Pyramid (which also uses getSiteManager.sethook), this won't help me at all. And to do this on the level of zope.component, I'm quite sure we are going to need to override getSiteManager.
but anyone could at any time call getSiteManager.sethook to change it!
Seriously? Nobody calls that but deep infrastructure code.
Agree with Jim. Are you going to stop people monkey patching getSiteManager() too? ;-)
My point is: Ensuring that test setup always and everywhere happens precisely so that the code that calls sethook (for example the setup of zope.site) is called *before* the envisioned test infrastructure code comes along to bend getSiteManager to do its will seems very fragile to me, if it's doable at all. Wolfgang -- Wolfgang Schnerring · ws@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
Hi, On 26 March 2011 08:11, Wolfgang Schnerring <ws@gocept.com> wrote:
Hello,
* Martin Aspeli <optilude+lists@gmail.com> [2011-03-25 13:58]:
plone.testing (which is Plone non-specific and will shortly be BSD licensed) allows for stacking of ZCA registries. [...] Again, plone.testing is the result of hours and hours of finding weird problems, so I'd recommend you don't discard the knowledge there. I think a best case scenario would be for plone.testing.zca to just be a delegate for something more robust in zope.component.testing.
I wasn't aware plone.testing had something for the ZCA, before I read that in this thread yesterday. I'm glad to hear that, and I certainly don't want to discard anything. But from reading the code and what you wrote, I get the impression that plone.testing does not yet solve the issue completely, precisely because of the two points I brought up and that continue below:
It certainly doesn't solve the "unregister" (non-?)use case. It has the advantage of being in use in the wild today, though. :-)
On 25 March 2011 13:17, Jim Fulton <jim@zope.com> wrote:
On Fri, Mar 25, 2011 at 4:24 AM, Wolfgang Schnerring <ws@gocept.com> wrote:
we realized that just squeezing in a new registry and bending its bases to the previously active one is not enough for full isolation, since this does not cover *deleting* registrations
Is deleting registrations important? This seems like an odd use case. If it's needed, I would suggest starting with a baseline (e.g. stack) that doesn't include the component you want to test deleting, then adding in setup.
I've wanted to specifically nuke registrations sometimes, the pattern being, load the package's configure.zcml in the layer, and then delete something (to demonstrate some error handling behaviour).
I agree that this use case is rare, but I'm not sure it is rare enough that we should ignore it. I need to think about this some more, though.
It would be better to explicitly register what you need. I think you either need to consider a package's ZCML-loaded configuration as an atomic part of test setup, or you need to break it down into individual registrations. In the first case, your test fixture is "package foo's configuration is loaded", which is of course valid. In the second case, your fixture is "the following components, some of which happen to be in package foo, are registered". I don't think a fixture of "package foo's configuration except component X and Y" is all that useful. Of course, there may be cases where an unregister is needed, but this is probably something you should solve locally. For example, you can explicitly unregister the component at the beginning of your test and reinstate it at the end of your test, or indeed in layer setup and tear-down. Automating this with stacking is not a great win. Please also take my word for it when I say that copying the whole registry is non-trivial and would rely on brittle ZCA internals. I tried. :)
2. zope.component has two entry points, the global site registry and the current registry (getGlobalSiteManager and getSiteManager). The current registry can be anything, or more precisely, you can call getSiteManager.sethook(callable) and provide a callable that returns the current registry.
I think to provide test support for zope.component (i. e. generally, at the "library level"), we need to support both entry points.
Why? Why would someone care about anything other than the current effective configuration.
Because that's the API that zope.component offers, it conceptually deals with *two* registries: the one you get from getSiteManager and the one you get from getGlobalSiteManager.
I agree that it is unlikely that client code will *read* registrations using getGlobalSiteManager -- since most everyone uses zope.component.getUtility etc. (which in turn uses getSiteManager). But *writing* is a different matter. Just one example: zope.component.provideUtility etc. uses getGlobalSiteManager, while the ZCML directives use getSiteManager.
At least as long as zope.component.provide* uses getGlobalSiteManager instead of getSiteManager, I maintain that test infrastructure needs to 1. wrap the global registry 2. wrap whatever getSiteManager currently returns (Otherwise we might be able get away with not doing (1), but I think we'll always need to do (2).)
plone.testing has it the other way around, doing (1) but not (2), as Martin says himself...
We do definitely need to allow the global site manager to be stacked (which you can achieve with __bases__ as in plone.testing, unregistration notwithstanding). But once you do that, the rest is pretty easy. The local site manager will always have the global as one of its (nested) __bases__. My preference would be: - We replace the three separate module-level references to the global site manager with a single object - This object is a holder for a registry reference that can be changed - The registry needs to have a __reduce__ that works with a local registry is pickled with the global registry as an eventual __bases__ reference - getGlobalSiteManager() always look up the current value in the holder, and nothing else looks directly at any module-level variables With this, we could remove the hacks in plone.testing and move most of the conveniences down to zope.component.testing if we wanted to. Martin
Hello, * Martin Aspeli <optilude+lists@gmail.com> [2011-03-26 11:22]:
On 26 March 2011 08:11, Wolfgang Schnerring <ws@gocept.com> wrote:
* Martin Aspeli <optilude+lists@gmail.com> [2011-03-25 13:58]: Please also take my word for it when I say that copying the whole registry is non-trivial and would rely on brittle ZCA internals. I tried. :)
I tried, too, and yup, in the end you'd have to write C to do it properly (because the AdapterLookupBase or something in zope.interface is written in C and would need to gain __deepcopy__ support). But it didn't seem unfeasible, and, when it was all said and done in Python, not very ugly either. I appreciate that you've already fought (and won!) against the whole persistence story (which hadn't even been on my radar so far), and I guess copying would necessitate reopening that can of worms.
I've wanted to specifically nuke registrations sometimes, the pattern being, load the package's configure.zcml in the layer, and then delete something (to demonstrate some error handling behaviour).
I think you either need to consider a package's ZCML-loaded configuration as an atomic part of test setup, or you need to break it down into individual registrations. In the first case, your test fixture is "package foo's configuration is loaded", which is of course valid. In the second case, your fixture is "the following components, some of which happen to be in package foo, are registered".
I don't think a fixture of "package foo's configuration except component X and Y" is all that useful.
It seems I can't convey why I think that precisely this is valuable even without taking into account the desire for non-leaky abstractions (in the ZODB-analogy it would be quite strange to have to write on top, please don't delete stuff, it's not supported). And of course I know how one could work around the limitation that unregistering is not supported. That doesn't make it any less inconvenient. The other point about getSiteManager below is *much* more important than unregistration support, though:
Because that's the API that zope.component offers, it conceptually deals with *two* registries: the one you get from getSiteManager and the one you get from getGlobalSiteManager.
I agree that it is unlikely that client code will *read* registrations using getGlobalSiteManager -- since most everyone uses zope.component.getUtility etc. (which in turn uses getSiteManager). But *writing* is a different matter. Just one example: zope.component.provideUtility etc. uses getGlobalSiteManager, while the ZCML directives use getSiteManager.
At least as long as zope.component.provide* uses getGlobalSiteManager instead of getSiteManager, I maintain that test infrastructure needs to 1. wrap the global registry 2. wrap whatever getSiteManager currently returns (Otherwise we might be able get away with not doing (1), but I think we'll always need to do (2).)
plone.testing has it the other way around, doing (1) but not (2), as Martin says himself...
We do definitely need to allow the global site manager to be stacked (which you can achieve with __bases__ as in plone.testing, unregistration notwithstanding). But once you do that, the rest is pretty easy. The local site manager will always have the global as one of its (nested) __bases__.
I'm sorry, but no, it isn't that easy. When the only local site consumer is zope.site, well, maybe. But please think of this in terms of zope.component *only*. Its API is getSiteManager.sethook(callable), and AFAICT the contract is that the return value of callable must provide IComponents (briefly: get* and register*). Nowhere does it say that you have to delegate back to the global registry, and neither it should. To bring up Pyramid once again, they explicitly don't, because they want to allow several applications (thus, several registries) coexisting in the same process. And since we can't assume this delegation, I think there is no other way to properly do the stacking than to bend getSiteManager. Wolfgang -- Wolfgang Schnerring · ws@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
Hi, it seems to me this has stalled somewhat, so I wanted to ask what people's conclusions are. * Wolfgang Schnerring <ws@gocept.com> [2011-03-26 13:41]:
* Martin Aspeli <optilude+lists@gmail.com> [2011-03-26 11:22]:
On 26 March 2011 08:11, Wolfgang Schnerring <ws@gocept.com> wrote: I don't think a fixture of "package foo's configuration except component X and Y" is all that useful.
Whether the the "unregister" use case is useful remains debatable, but I personally don't care all *that* much for it, so if the consensus is that it's overkill I'll go along I guess. I do care quite a bit for proper getSiteManager() support...
We do definitely need to allow the global site manager to be stacked (which you can achieve with __bases__ as in plone.testing, unregistration notwithstanding). But once you do that, the rest is pretty easy. The local site manager will always have the global as one of its (nested) __bases__.
I'm sorry, but no, it isn't that easy. When the only local site consumer is zope.site, well, maybe. But please think of this in terms of zope.component *only*.
Its API is getSiteManager.sethook(callable), and AFAICT the contract is that the return value of callable must provide IComponents (briefly: get* and register*). Nowhere does it say that you have to delegate back to the global registry, and neither it should. To bring up Pyramid once again, they explicitly don't, because they want to allow several applications (thus, several registries) coexisting in the same process.
And since we can't assume this delegation, I think there is no other way to properly do the stacking than to bend getSiteManager.
... as described here, though. And I wonder if I'm missing something, because to do that properly looks like quite the can of worms to me. So, how can we proceed here? Should I (and Thomas) try to get a proof-of-concept implementation of this based on plone.testing? Or should we think about what it takes to merge most of plone.testing's ZCA support into zope.component itself first? Wolfgang
Hi, On 4 April 2011 17:30, Wolfgang Schnerring <ws@gocept.com> wrote:
So, how can we proceed here? Should I (and Thomas) try to get a proof-of-concept implementation of this based on plone.testing? Or should we think about what it takes to merge most of plone.testing's ZCA support into zope.component itself first?
I think either approach is valuable, and not necessarily mutually exclusive. I do care about the plone.testing API, which is used in production, so bear that in mind. Martin
Hello, * Martin Aspeli <optilude+lists@gmail.com> [2011-04-04 18:52]:
On 4 April 2011 17:30, Wolfgang Schnerring <ws@gocept.com> wrote:
So, how can we proceed here? Should I (and Thomas) try to get a proof-of-concept implementation of this based on plone.testing? Or should we think about what it takes to merge most of plone.testing's ZCA support into zope.component itself first?
I think either approach is valuable, and not necessarily mutually exclusive.
I do care about the plone.testing API, which is used in production, so bear that in mind.
Yes, will certainly do. Wolfgang
Hello, part two. :) * Uli Fouquet <uli@gnufix.de> [2011-03-24 01:05]:
A big advantage of test layers over `pytest` testing scopes might be that you can spread your tests associated to a certain layer over many files/modules/packages as you like and the setup/teardown will nevertheless only be performed once for each layer
Compared to Zope test layers I came to the conclusion that there is not much like this concept already in `pytest` and the behaviour of test layers can not easily be faked.
I fully agree, the test layer concept of zope.testrunner provides some very nice properties, most notably: 1) Shared fixture setup. It is shared across many TestCases and created only once per test run. (And it is independent of how the tests are organized into modules or whatever.) 2) Stacking of fixture setups. My canonical example: First, we initialize the application, load ZCML, prepare databases, etc. and talk to it with zope.testbrowser. That's the first layer. Then we add a HTTP port with gocept.selenium on another layer and do the rest with a real browser -- but the whole, possibly expensive, application setup *does not have to be done again*. 3) Orthogonal composition, i.e. you can have small, well defined fixture units that you can then combine in a separate step to form the actual setup for your tests. This enables for example that library packages like zope.component could provide test infrastructure for themselves, and application packages could pull all the test infrastructure together that they need -- not like zope.app.testing for example, which is a giant monolith that sets up everything at once. I think that these properties are very valuable, and I haven't seen them anywhere else so far. I haven't looked at py.test closely, yet, but from what I've seen, and what you wrote, it doesn't seem to offer these things right now.
Would it make sense to bring the Zope layer concept into `pytest`?
Are there already possibilities to mimic testlayer-like behaviour in `pytest` which we simply overlooked?
An honest counter question: Why not use zope.testrunner? What advantages does py.test offer? But your email brings into focus a question that I've been thinking about lately: What's the future direction for test fixture setup? Because, while test layers are nice because they have the above properties, I'm not too happy with the current implementation, namely the use (or is it abuse? ;-) of __bases__ and __name__, which leads very naturally to not-so-helpful implementation patterns. As a concrete example, some of the test layers of ZTK packages, most notably the successors of zope.app.testing.functional in zope.component, zope.app.appsetup, zope.app.wsgi, are implemented right now in a way that does not have properties 2+3. The most straightforward way to improve that situation would be to use plone.testing, since that provides a layer implementation that is both sane and easy to use, and has all the nice properties. (Maybe even fold some of it into zope.testrunner itself.) But then I realized, that's a major undertaking, so we better think about what the actual goals are before changing lots of stuff. And then I heard people expressing interest in other test runners. (py.test and nose seem to be the biggest players, stdlib's unittest/unittest2 seems to be woefully behind in terms of test selection and other features like coverage or debugging. I for one happen to like zope.testrunner, but if there's a good alternative, why not combine our efforts?) I also found that it is rather hard to explain how to use test layers in a way so you actually get the above properties, so I started wondering whether there was a cleaner way to get them. So I guess I have three questions: 1. (The starting point:) How can we improve the test infrastructure situation of the ZTK packages? 2. (The main thing:) Are test layers, both the concept and the implementation as epitomized by plone.testing, the way to go or is there a cleaner alternative? 3. (But only tangentially, I don't want to sidetrack the discussion at hand nor start a flamewar:) Is there a compelling alternative to zope.testrunner that would make it worthwile to think about either generalizing the fixture support or even migrating away from zope.testrunner? Wolfgang -- Wolfgang Schnerring · ws@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
Hi, On 26 March 2011 14:18, Wolfgang Schnerring <ws@gocept.com> wrote:
Because, while test layers are nice because they have the above properties, I'm not too happy with the current implementation, namely the use (or is it abuse? ;-) of __bases__ and __name__, which leads very naturally to not-so-helpful implementation patterns. As a concrete example, some of the test layers of ZTK packages, most notably the successors of zope.app.testing.functional in zope.component, zope.app.appsetup, zope.app.wsgi, are implemented right now in a way that does not have properties 2+3.
The most straightforward way to improve that situation would be to use plone.testing, since that provides a layer implementation that is both sane and easy to use, and has all the nice properties. (Maybe even fold some of it into zope.testrunner itself.)
I'd be happy to move layer.py from plone.testing to zope.testrunner for sure.
But then I realized, that's a major undertaking, so we better think about what the actual goals are before changing lots of stuff. And then I heard people expressing interest in other test runners. (py.test and nose seem to be the biggest players, stdlib's unittest/unittest2 seems to be woefully behind in terms of test selection and other features like coverage or debugging. I for one happen to like zope.testrunner, but if there's a good alternative, why not combine our efforts?) I also found that it is rather hard to explain how to use test layers in a way so you actually get the above properties, so I started wondering whether there was a cleaner way to get them.
Personally, I rely a lot on the layer mechanism and would not want to lose it.
So I guess I have three questions:
1. (The starting point:) How can we improve the test infrastructure situation of the ZTK packages?
I'd say having more reusable test layers would help.
2. (The main thing:) Are test layers, both the concept and the implementation as epitomized by plone.testing, the way to go or is there a cleaner alternative?
When we looked at how to improve Plone's testing story, we couldn't find anything better. One thing I would like is a better way to support "python setup.py test".
3. (But only tangentially, I don't want to sidetrack the discussion at hand nor start a flamewar:) Is there a compelling alternative to zope.testrunner that would make it worthwile to think about either generalizing the fixture support or even migrating away from zope.testrunner?
It's a good question to ask, though I think this would be a really big undertaking for maybe only marginal gain (if any). Martin
Hi there, I first would like to thank everybody very much for the interesting and elaborated answers to my last questions! I learned a lot about registries and think that for now `plone.testing` (which in fact we hadn't had on the screen) will help to solve some of the most urgent issues we have with zope.pytest currently. So, thank you very much! A sidenote: IMO it would be nice to have proper deletion of registrations in the long run as well and I am willing to help as good as I can with that, even if that would mean much work. The 'just-setup-your-layers-more-carefully'-solution is a bit tricky in testing-frameworks without layers ;-) Wolfgang Schnerring wrote: [snip]
Are there already possibilities to mimic testlayer-like behaviour in `pytest` which we simply overlooked?
An honest counter question: Why not use zope.testrunner? What advantages does py.test offer?
Due to my limited experience with py.test I'm certainly not in a position to answer that comprehensively. I personally use zope.testrunner and layers happily in most of my projects. zope.pytest is currently certainly more a try to see what's possible with other approaches than zope.testrunner and some features of py.test are really promising IMO. The (limited) experiences with py.test, however, were awesome. Some points that are quite cool IMHO: - Easy finding of tests: just write some ``test_function`` in a ``test_module`` and it will be found and executed. That also makes py.test tests more readable and maybe more intuitive. - Lots of setup code (unrelated to fixtures) can simply be skipped. No need to do the ``testsuite = <complex-testcase-collecting>`` over and over again. Maybe the main point of py.test. - py.test is more widespread in the Python community (that's my impression; I can't proof it) - Support of unittest/unittest2: you can write standard lib setups (defining TestCases; no need to also write testsuite-setup stuff) and they will be found/executed. zope.testrunner for instance does not support the new `setUpClass`/`tearDownClass` concept of unittest2 (yes, you would use layers in that case; but it might be nice if zope.testrunner would support also class-wide fixtures in unittest2-style; people from other worlds might expect that to work). - `assert` works like you would expect it to work in tests. No need to use `self.assertEqual()`` and friends (but you can if you prefer). - It was very easy to run py.test from the Sphinx doctest builder which made it possible to test documentation on a quite complex level. That might be possible with zope.testrunner as well (but I never tried that; wait - I tried once and it worked, but was more complex to setup IIRC). I see that this is a very special usecase. That said I'd like to stress that I would not say that py.test is superior to zope.testrunner in general. I only enjoyed the first experiences with py.test very much and could imagine that it would be a pleasure for others too. Other things, however, meant less fun. Main drawbacks I see on py.test side are: - Lack of layer support (yet). Maybe we can do something about that in `zope.pytest` based on `plone.testing.layer`. - Limited doctest support. It is quite difficult (AFAIK) to define fixtures for doctests or to even set the usual doctest options (``ELLIPSIS``, ``NORMALIZE_WHITESPACE``, ...) at setup time. Doctests are simply collected and executed and not much finetuning is possible. My very own concern about the latter point is: if this cannot be solved satisfactorily (in terms of readability and ease of use at least), py.test will not become a really hot candidate in the testing framework game on the Zope side at all. There are too many doctests used already and whether you like them or not: every testing framework used should offer at least minimal support for doctest fixtures and doctest options. But I might just have missed these features in py.test and they are provided already. For now I think that there is absolutely no need to think about a general move to py.test for the ztk. I guess Martijn or others with more py.test experience could say more concerning the pros and cons of py.test from daily work use. [snip] Best regards, -- Uli
Hi, On 27 March 2011 15:54, Uli Fouquet <uli@gnufix.de> wrote:
The (limited) experiences with py.test, however, were awesome. Some points that are quite cool IMHO:
- Easy finding of tests: just write some ``test_function`` in a ``test_module`` and it will be found and executed. That also makes py.test tests more readable and maybe more intuitive.
I'm not sure this is always a good idea. It feels a bit implicit, and having a base class isn't really a big problem, IMHO. It seems a bit like the kind of thing that sounds cool (look, it's even easier!), but in practice makes little difference.
- Lots of setup code (unrelated to fixtures) can simply be skipped. No need to do the ``testsuite = <complex-testcase-collecting>`` over and over again. Maybe the main point of py.test.
You don't need that for zope.testrunner either, of course, at least not when using unittest base classes.
- py.test is more widespread in the Python community (that's my impression; I can't proof it)
What about nose?
- Support of unittest/unittest2: you can write standard lib setups (defining TestCases; no need to also write testsuite-setup stuff) and they will be found/executed. zope.testrunner for instance does not support the new `setUpClass`/`tearDownClass` concept of unittest2 (yes, you would use layers in that case; but it might be nice if zope.testrunner would support also class-wide fixtures in unittest2-style; people from other worlds might expect that to work).
zope.testing should definitely gain support for the new unittest2 hooks. That wouldn't be very hard, though. ;-)
Main drawbacks I see on py.test side are:
- Lack of layer support (yet). Maybe we can do something about that in `zope.pytest` based on `plone.testing.layer`.
- Limited doctest support. It is quite difficult (AFAIK) to define fixtures for doctests or to even set the usual doctest options (``ELLIPSIS``, ``NORMALIZE_WHITESPACE``, ...) at setup time. Doctests are simply collected and executed and not much finetuning is possible.
With zope.testrunner, you *do* need a test_suite method to run doctests. I think that's a good thing. Look at plone.testing's README for examples.
My very own concern about the latter point is: if this cannot be solved satisfactorily (in terms of readability and ease of use at least), py.test will not become a really hot candidate in the testing framework game on the Zope side at all. There are too many doctests used already and whether you like them or not: every testing framework used should offer at least minimal support for doctest fixtures and doctest options. But I might just have missed these features in py.test and they are provided already.
FWIW, I think we should stop using .txt doctests for unit tests. Doctests should be used to test *documentation* ("the examples are valid"). For actual unit tests, writing tests in a unittest class is almost always better in the long run. doctests don't scale well and discourage the kind of ad-hoc "this seems broken, I'll just write a quick test" or "I just fixed a bug, better add a regression test" testing.
For now I think that there is absolutely no need to think about a general move to py.test for the ztk.
I think there's benefit in unifying the concepts and support for concepts like layers so that people can use the test runner they prefer. Maritn
On Sun, Mar 27, 2011 at 11:13 AM, Martin Aspeli <optilude+lists@gmail.com> wrote:
Hi,
On 27 March 2011 15:54, Uli Fouquet <uli@gnufix.de> wrote:
The (limited) experiences with py.test, however, were awesome. Some points that are quite cool IMHO:
- Easy finding of tests: just write some ``test_function`` in a ``test_module`` and it will be found and executed. That also makes py.test tests more readable and maybe more intuitive.
I'm not sure this is always a good idea. It feels a bit implicit, and having a base class isn't really a big problem, IMHO. It seems a bit like the kind of thing that sounds cool (look, it's even easier!), but in practice makes little difference.
+1 +1 +1 This is especially important for doctests (and manuel) or any situation where setup is important and where you can't really guess.
- py.test is more widespread in the Python community (that's my impression; I can't proof it)
What about nose?
It looks to me like a layerish mechanism might be possible in nose, or at least like zope.testing layers could be integrated with nose.
- Support of unittest/unittest2: you can write standard lib setups (defining TestCases; no need to also write testsuite-setup stuff) and they will be found/executed. zope.testrunner for instance does not support the new `setUpClass`/`tearDownClass` concept of unittest2 (yes, you would use layers in that case; but it might be nice if zope.testrunner would support also class-wide fixtures in unittest2-style; people from other worlds might expect that to work).
zope.testing should definitely gain support for the new unittest2 hooks. That wouldn't be very hard, though. ;-)
I assume you mean zope.testrunner.
Main drawbacks I see on py.test side are:
- Lack of layer support (yet). Maybe we can do something about that in `zope.pytest` based on `plone.testing.layer`.
- Limited doctest support. It is quite difficult (AFAIK) to define fixtures for doctests or to even set the usual doctest options (``ELLIPSIS``, ``NORMALIZE_WHITESPACE``, ...) at setup time. Doctests are simply collected and executed and not much finetuning is possible.
With zope.testrunner, you *do* need a test_suite method to run doctests. I think that's a good thing. Look at plone.testing's README for examples.
Again, +1 If I were to use nose or py.tests, I would want to adopt an explicit style, which I believe is possible w nose. ...
FWIW, I think we should stop using .txt doctests for unit tests.
I disagree, of course.
Doctests should be used to test *documentation* ("the examples are valid").
Manuel is *much* better for that. (Of course, manuel is arguably a form of doctest.)
For actual unit tests, writing tests in a unittest class is almost always better in the long run. doctests don't scale well and discourage the kind of ad-hoc "this seems broken, I'll just write a quick test" or "I just fixed a bug, better add a regression test" testing.
You're just not using them correctly. :) One thing I hate about unittest is the javiotic ceremony it involves. Doctests can cut down on the clutter a lot. I believe that that is py.test's strength as well.. More generally, I'd love to see us adopt another test runner so that we can stop maintianing zope.testrunner. When it was written at the turn of the century, there weren't good alternatives. Personally, I think maintaining it is boring. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/28/2011 10:04 AM, Jim Fulton wrote:
On Sun, Mar 27, 2011 at 11:13 AM, Martin Aspeli <optilude+lists@gmail.com> wrote:
Hi,
On 27 March 2011 15:54, Uli Fouquet <uli@gnufix.de> wrote:
The (limited) experiences with py.test, however, were awesome. Some points that are quite cool IMHO:
- Easy finding of tests: just write some ``test_function`` in a ``test_module`` and it will be found and executed. That also makes py.test tests more readable and maybe more intuitive.
I'm not sure this is always a good idea. It feels a bit implicit, and having a base class isn't really a big problem, IMHO. It seems a bit like the kind of thing that sounds cool (look, it's even easier!), but in practice makes little difference.
+1 +1 +1
This is especially important for doctests (and manuel) or any situation where setup is important and where you can't really guess.
- py.test is more widespread in the Python community (that's my impression; I can't proof it)
What about nose?
It looks to me like a layerish mechanism might be possible in nose, or at least like zope.testing layers could be integrated with nose.
- Support of unittest/unittest2: you can write standard lib setups (defining TestCases; no need to also write testsuite-setup stuff) and they will be found/executed. zope.testrunner for instance does not support the new `setUpClass`/`tearDownClass` concept of unittest2 (yes, you would use layers in that case; but it might be nice if zope.testrunner would support also class-wide fixtures in unittest2-style; people from other worlds might expect that to work).
zope.testing should definitely gain support for the new unittest2 hooks. That wouldn't be very hard, though. ;-)
I assume you mean zope.testrunner.
Main drawbacks I see on py.test side are:
- Lack of layer support (yet). Maybe we can do something about that in `zope.pytest` based on `plone.testing.layer`.
- Limited doctest support. It is quite difficult (AFAIK) to define fixtures for doctests or to even set the usual doctest options (``ELLIPSIS``, ``NORMALIZE_WHITESPACE``, ...) at setup time. Doctests are simply collected and executed and not much finetuning is possible.
With zope.testrunner, you *do* need a test_suite method to run doctests. I think that's a good thing. Look at plone.testing's README for examples.
Again, +1
If I were to use nose or py.tests, I would want to adopt an explicit style, which I believe is possible w nose.
FWIW, I thought the same, but haven't missed 'test_suite' appreciably after a week or so of acclimatization to the nose discovery mechanism. The only wrinkle I know of is that one doesn't use base classes for test cases where the base class itself derives from unittest.TestCase: instead, the base is just used as a "pure mixin".
FWIW, I think we should stop using .txt doctests for unit tests.
I disagree, of course.
Doctests should be used to test *documentation* ("the examples are valid").
Manuel is *much* better for that. (Of course, manuel is arguably a form of doctest.)
The Sphinx doctest integration[1] ('make doctest') supports both classic "interpreter prompt" doctests and a more useful (IMHO) "code-output" form, such as: .. testcode:: 1+1 # this will give no output! print 2+2 # this will give output .. testoutput:: 4 [1] http://sphinx.pocoo.org/ext/doctest.html
For actual unit tests, writing tests in a unittest class is almost always better in the long run. doctests don't scale well and discourage the kind of ad-hoc "this seems broken, I'll just write a quick test" or "I just fixed a bug, better add a regression test" testing.
You're just not using them correctly. :)
The vast majority of the doctest testcases in zope.* packages fall into this category: poor isolation, lots of edge cases which would obscure any real narrative docs, of which there are almost none. I believe the conflict is intrinsic, here, and not just an accident of careless / naive implementation: exercising all the preconditions of the contract of the function-under-test makes for really poor documentation, but is essential to good testing.
One thing I hate about unittest is the javiotic ceremony it involves. Doctests can cut down on the clutter a lot. I believe that that is py.test's strength as well..
"I'll take explicit for $1000, Alex."
More generally, I'd love to see us adopt another test runner so that we can stop maintianing zope.testrunner. When it was written at the turn of the century, there weren't good alternatives. Personally, I think maintaining it is boring.
Nose and coverage have been pretty good to repoze.* and pyramid_*. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2QnwcACgkQ+gerLs4ltQ74FQCdGX9twswg6eAy9cnpABAZRrbo pvoAniGfpBliwnFjqFrV5uqpFcePI2tk =cqN7 -----END PGP SIGNATURE-----
On 28 March 2011 15:45, Tres Seaver <tseaver@palladion.com> wrote:
The vast majority of the doctest testcases in zope.* packages fall into this category: poor isolation, lots of edge cases which would obscure any real narrative docs, of which there are almost none. I believe the conflict is intrinsic, here, and not just an accident of careless / naive implementation: exercising all the preconditions of the contract of the function-under-test makes for really poor documentation, but is essential to good testing.
Agree. Here's what I've found and learned the hard way: doctests are sometimes easier / more fun to write the *first* time I write tests. It's easy to just start writing, and the narrative helps me think about what I'm doing. Plus, it feels like I'm saving time on writing docs. They are much worse all subsequent times. Maintenance becomes a soup. Quick tests for edge cases feel like they obscure the narrative, so I may forgo them. Refactoring is painful. Tool support is poorer, e.g. no stepping through with pdb and no pyflakes. And people find the docs underwhelming. If I'm doing it wrong, I'd like to see it done "right". Manuel is kind of cool, but I'm not sure it really addresses the issue. It's a better doctest, not a better unittest. Most zope.component packages only get away with simple doctests because they are simple/small packages (which is a good thing, mainly, of course). One of the main objections to unittest(2) seems to be that it's "Javaesque". Yes, JUnit was the first library to solidify many of the current, cross-language testing patterns. We shouldn't be so NIH-stricken that we can't see the benefit of being consistent with patterns used in most other languages and test frameworks these days. Martin
On Mon, Mar 28, 2011 at 5:27 PM, Martin Aspeli <optilude+lists@gmail.com> wrote:
On 28 March 2011 15:45, Tres Seaver <tseaver@palladion.com> wrote:
The vast majority of the doctest testcases in zope.* packages fall into this category: poor isolation, lots of edge cases which would obscure any real narrative docs, of which there are almost none. I believe the conflict is intrinsic, here, and not just an accident of careless / naive implementation: exercising all the preconditions of the contract of the function-under-test makes for really poor documentation, but is essential to good testing.
Agree. Here's what I've found and learned the hard way:
doctests are sometimes easier / more fun to write the *first* time I write tests. It's easy to just start writing, and the narrative helps me think about what I'm doing.
Yup.
Plus, it feels like I'm saving time on writing docs.
And it doesn't because tests != docs. I've found that docs need to be written from an entirely different point of view.
They are much worse all subsequent times. Maintenance becomes a soup.
I've only found this to be the case if all tests are put in the same document. Keep in mind that unit tests can be a maintenance disaster too. I speak from experience. Some key factors in maintainability: - Is the intent clear. Unit tests don't do anything to make intent clear. Doctests don't force you to make intent clear. - Coupling between different tests. Arguably doctests encourage this by making it easy to glom many unrelated tests into the same document, but this can and does happen with unit tests as well, especially if a lot of setup is needed.
Quick tests for edge cases feel like they obscure the narrative,
Yup, they do!
so I may forgo them.
Which is a mistake. You should create separate tests. I typically put large tests, dealing with main use cases where there is a definite flow of activity in '.test' files. I do these in separate files because they're easier to write that way. I use a '.test' suffix to avoid the pretense that these are documentation. I put edge-case tests in small docstrings in testing modules. I'm not really religious about using doctests for this, but I find small edge-case doctests easier to read than traditional unit tests. It's possible that I'd like py.test tests as much.
Refactoring is painful.
It all depends on how well the tests are written and this applies equally no matter what format is used.
Tool support is poorer,
True ...
And people find the docs underwhelming.
tests != docs This was a lesson learned.
If I'm doing it wrong, I'd like to see it done "right".
You might check out the bobo and zc.ngi docs and tests and let me know what you think.
Manuel is kind of cool, but I'm not sure it really addresses the issue. It's a better doctest, not a better unittest.
You may be missing the point. When writing docs, the primary goal is to guide the user to understanding as effectively as possible. The focus is on the user and their goals. A secondary, but important goal is making sure what you say in documentation is true. Manuel helps with the second goal without compromising the first goal. When writing tests, the focus is on the software and assuring that it is working correctly. Manuel can help here too by making some things easier to express and by being a better doctests, but the main goal of manuel is to enable writing good documentation that is still executable.
One of the main objections to unittest(2) seems to be that it's "Javaesque". Yes, JUnit was the first library to solidify many of the current, cross-language testing patterns. We shouldn't be so NIH-stricken that we can't see the benefit of being consistent with patterns used in most other languages and test frameworks these days.
It's not about NIH, it's about ceremony. Java and it's culture are all about ceremony that just get in the way. py.test and (modern uses of) doctest are in many ways a reaction to that ceremony. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton
On Tuesday, March 29, 2011, Jim Fulton wrote:
so I may forgo them.
Which is a mistake. You should create separate tests. I typically put large tests, dealing with main use cases where there is a definite flow of activity in '.test' files. I do these in separate files because they're easier to write that way. I use a '.test' suffix to avoid the pretense that these are documentation. I put edge-case tests in small docstrings in testing modules. I'm not really religious about using doctests for this, but I find small edge-case doctests easier to read than traditional unit tests. It's possible that I'd like py.test tests as much.
Yeah, Marius led me recently to that path too. Write a narrative in text files and use doc strings of functions to do edge cases (or when you don't have time for the narrative). I am getting used to it. I still much prefer the sort of output comparison that doctests/manuel gives me over the assertion language that unittest.TestCase requires. Regards, Stephan -- Entrepreneur and Software Geek Google me. "Zope Stephan Richter"
On 3/29/11 14:40 , Stephan Richter wrote:
On Tuesday, March 29, 2011, Jim Fulton wrote:
so I may forgo them.
Which is a mistake. You should create separate tests. I typically put large tests, dealing with main use cases where there is a definite flow of activity in '.test' files. I do these in separate files because they're easier to write that way. I use a '.test' suffix to avoid the pretense that these are documentation. I put edge-case tests in small docstrings in testing modules. I'm not really religious about using doctests for this, but I find small edge-case doctests easier to read than traditional unit tests. It's possible that I'd like py.test tests as much.
Yeah, Marius led me recently to that path too. Write a narrative in text files and use doc strings of functions to do edge cases (or when you don't have time for the narrative). I am getting used to it. I still much prefer the sort of output comparison that doctests/manuel gives me over the assertion language that unittest.TestCase requires.
FWIW unittest2 has much nicer output if you use the new assert methods. Wichert.
On 03/29/2011 02:43 PM, Wichert Akkerman wrote:
On 3/29/11 14:40 , Stephan Richter wrote:
Yeah, Marius led me recently to that path too. Write a narrative in text files and use doc strings of functions to do edge cases (or when you don't have time for the narrative). I am getting used to it. I still much prefer the sort of output comparison that doctests/manuel gives me over the assertion language that unittest.TestCase requires.
FWIW unittest2 has much nicer output if you use the new assert methods.
py.test has very nice output if you use the Python 'assert' statement. There are no assert methods to remember. Regards, Martijn
On 2011-4-20 21:59, Martijn Faassen wrote:
On 03/29/2011 02:43 PM, Wichert Akkerman wrote:
On 3/29/11 14:40 , Stephan Richter wrote:
Yeah, Marius led me recently to that path too. Write a narrative in text files and use doc strings of functions to do edge cases (or when you don't have time for the narrative). I am getting used to it. I still much prefer the sort of output comparison that doctests/manuel gives me over the assertion language that unittest.TestCase requires.
FWIW unittest2 has much nicer output if you use the new assert methods.
py.test has very nice output if you use the Python 'assert' statement. There are no assert methods to remember.
That sounds nice. I have used Catch (https://github.com/philsquared/catch) a lot for C++ testing recently which also uses a single assert statement instead of a miriad of assert* functions, and it has been a very pleasant experience. Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
Hello, * Jim Fulton <jim@zope.com> [2011-03-28 10:04]:
More generally, I'd love to see us adopt another test runner so that we can stop maintianing zope.testrunner. When it was written at the turn of the century, there weren't good alternatives. Personally, I think maintaining it is boring.
I agree, it would be nice to get out of the test runner business, just as we're getting out of the networking business more and more courtesy of WSGI. But I'm wary of throwing out the baby with the bathwater, zope.testrunner has quite a few features under the hood that are really useful, which I'm not sure other test runners have, and I definitely wouldn't want to lose. Layers are the most prominent, of course, but then there's post-mortem debugging (-D), coverage integration, ... the list goes on for a few more items, I'm certain. I guess, apart from the layer issue (see other messages in this thread), some research and write-up would be a good idea to get a feeling what the other test runners are like and how they measure up against zope.testrunner. A quick google search turns up nothing appropriate, so I might do a comparison of zope.testrunner, py.test and nose, but that's going to take a while. Wolfgang
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 03/28/2011 10:56 AM, Wolfgang Schnerring wrote:
Hello,
* Jim Fulton <jim@zope.com> [2011-03-28 10:04]:
More generally, I'd love to see us adopt another test runner so that we can stop maintianing zope.testrunner. When it was written at the turn of the century, there weren't good alternatives. Personally, I think maintaining it is boring.
I agree, it would be nice to get out of the test runner business, just as we're getting out of the networking business more and more courtesy of WSGI. But I'm wary of throwing out the baby with the bathwater, zope.testrunner has quite a few features under the hood that are really useful, which I'm not sure other test runners have, and I definitely wouldn't want to lose. Layers are the most prominent, of course, but then there's post-mortem debugging (-D), coverage integration, ... the list goes on for a few more items, I'm certain.
I guess, apart from the layer issue (see other messages in this thread), some research and write-up would be a good idea to get a feeling what the other test runners are like and how they measure up against zope.testrunner. A quick google search turns up nothing appropriate, so I might do a comparison of zope.testrunner, py.test and nose, but that's going to take a while.
nose supports post-mortem debugging, and integrates nicely with Net Batchelder's 'coverage' tool. It's extension interface uses setuptools entry points, and is reasonably well documented: http://somethingaboutorange.com/mrl/projects/nose/1.0.0/plugins/writing.html Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2QrXIACgkQ+gerLs4ltQ4SgwCfRXRljDcSF+4jh6f8Qr5Lk/gX EHwAoKsM2w8vIKG7hOriFYXQnViEt8RI =6G8m -----END PGP SIGNATURE-----
Hello, * Martin Aspeli <optilude+lists@gmail.com> [2011-03-27 16:13]:
On 27 March 2011 15:54, Uli Fouquet <uli@gnufix.de> wrote:
The (limited) experiences with py.test, however, were awesome. Some points that are quite cool IMHO: [...]
I agree wholeheartedly with what Martin has said about py.test vs. zope.testrunner.
- Lots of setup code (unrelated to fixtures) can simply be skipped. No need to do the ``testsuite = <complex-testcase-collecting>`` over and over again. Maybe the main point of py.test. You don't need that for zope.testrunner either, of course, at least not when using unittest base classes.
This is a point that bears repeating, though: test_suite() is *not needed* since zope.testing-3.8.0 (2009-07-24) for descendants of unittest.TestCase.
FWIW, I think we should stop using .txt doctests for unit tests. Doctests should be used to test *documentation* ("the examples are valid"). For actual unit tests, writing tests in a unittest class is almost always better in the long run.
+lots and lots and lots, especially since you've formulated it in quite a balanced way.
For now I think that there is absolutely no need to think about a general move to py.test for the ztk.
I think there's benefit in unifying the concepts and support for concepts like layers so that people can use the test runner they prefer.
How can we make progress here? I'm not sure whether this calls for some green field sketching, "how should test fixture setup work?" or some hands-on experimentation, "let's see how we get some existing test layers to run under py.test", or both, or something else entirely. Wolfgang
On 03/27/2011 05:13 PM, Martin Aspeli wrote:
On 27 March 2011 15:54, Uli Fouquet<uli@gnufix.de> wrote:
The (limited) experiences with py.test, however, were awesome. Some points that are quite cool IMHO:
- Easy finding of tests: just write some ``test_function`` in a ``test_module`` and it will be found and executed. That also makes py.test tests more readable and maybe more intuitive.
I'm not sure this is always a good idea. It feels a bit implicit, and having a base class isn't really a big problem, IMHO. It seems a bit like the kind of thing that sounds cool (look, it's even easier!), but in practice makes little difference.
Belatedly adding my few cents here: in practice I quite enjoy not having to copy ugly boilerplate from test modules that I always always forget. It's actually nice to just write a test_* function and have the test runner just work. So it's cool for some people, namely me. Regards, Martijn
On Sun, Mar 27, 2011 at 10:54 AM, Uli Fouquet <uli@gnufix.de> wrote:
- `assert` works like you would expect it to work in tests. No need to use `self.assertEqual()`` and friends (but you can if you prefer).
How do they deal with the fact that assert statements are dropped when Python is run with -O? -- Benji York
Hi there, Benji York wrote:
On Sun, Mar 27, 2011 at 10:54 AM, Uli Fouquet <uli@gnufix.de> wrote:
- `assert` works like you would expect it to work in tests. No need to use `self.assertEqual()`` and friends (but you can if you prefer).
How do they deal with the fact that assert statements are dropped when Python is run with -O?
I think they don't. You get a warning if you do and, of course, reports might become useless. On the other hand you have distributed-testing options (I haven't tried yet) where you should be able to finetune the enviroments (yes, plural) nicely in which your tests run. It is said that distributed testing with py.test allows parallel usage of several CPUs (if the machine has several CPUs), parallel testing on remote machines and more. AFAICS you can also run the same tests with different Python interpreters in parallel. In sum all that might give better speedups than -O, but -- I haven't tried it yet. Best regards, -- Uli
On Mon, Mar 28, 2011 at 9:33 AM, Uli Fouquet <uli@gnufix.de> wrote:
Hi there,
Benji York wrote:
On Sun, Mar 27, 2011 at 10:54 AM, Uli Fouquet <uli@gnufix.de> wrote:
- `assert` works like you would expect it to work in tests. No need to use `self.assertEqual()`` and friends (but you can if you prefer).
How do they deal with the fact that assert statements are dropped when Python is run with -O?
I think they don't. You get a warning if you do and, of course, reports might become useless.
I don't think they're that foolish. At least I hope they're not. I think they use a custom compiler. Jim -- Jim Fulton http://www.linkedin.com/in/jimfulton
participants (11)
-
Benji York -
Godefroid Chapelle -
Hanno Schlichting -
Jim Fulton -
Martijn Faassen -
Martin Aspeli -
Stephan Richter -
Tres Seaver -
Uli Fouquet -
Wichert Akkerman -
Wolfgang Schnerring