<This is to a large part an angry rant. Sorry for that.> In my effort to port zope.component to Python 3 I constantly encounter one problem that is very hard to solve: Circular dependencies. It's hard to port module A, when running the tests require module B to be ported, and module B depends on module A. Or, as with setuptools, it actually uses itself, so it has to work before you can test if it works. (Sorry, that's just insane). So far the main circularity was that everything depended on zope.testing as a testrunner, zc.buildout for making the development environment, and zope.testing obviously depended on zope.interface etc. I solved that by also adding support for setuptools/distributes testrunner and using that instead. That fixed zope.interface, zope.event and zope.exception. These modules still have buildout configurations if you want them, but you don't need buildout anymore. zope.interface, zope.event and zope.exception can now be developed and tested with only setuptools, you can run the tests with "python setup.py test" both under Python 2 and 3. But then the time came to zope.testing itself. It, rather annoyingly, depends on itself. I did the same there, but almost all tests are even more annoyingly, doctests, and depends on a custom version of doctest included, etc, etc, etc. The result was hours and hours of minor fiddling to find exactly what character in a doctest that made it fail. So I've spent an inordinate amount of time on zope.testing, and I have come to a conclusion about it: == Suggestion 1 == I think we should deprecate zope.testing. Completely. There has been some discussion about deprecating it for something else, but I think we should just deprecate it. Just say "Don't use zope.testing, it's pure evil". We can recommend another testrunner, and it seems nose is winning the wars in the Python world there, but I don't think we necessarily need to do that. Most testrunners will find the tests by themselves, and the tests should be runnable with any testrunner, so which one you use is a minor issue. Instead the big problem with getting rid of zope.testing is that many tests use other specific features like the doctest renormalizers etc. These could possibly be extracted into separate modules so you can use them with other testrunners. But zope.testing is a mess that builds on two other messes, namely unittest (yes, it's a bloody mess, the API makes no sense) and doctest (which is a horror of a mess). unittest and doctest are horrid enough for us to not to make it worse with zope.testing in addition. The path forward there would simply be to stop depending on zope.testing for all ZTK modules. That's probably a bit of work, but I think it's worth it. So, with that in mind I today went on to zc.buildout, trying to port it to Python 3 by ripping out any usage of zope.testing. Also, the standard development mode for zc.buildout is that you run a dev.py script, that uses zc.buildout to build a buildout for zc.buildout. Obviously, this circularity is the same as I encountered in setuptools, and it's evil. But with zc.buildout it's easy to get rid off at least as I could use setuptools testrunner instead. But then I realize the zc.buildout needs zc.recipe.egg. And zc.recipe.egg depends on zc.buildout. Hohum, another circular depedency. == Suggestion 2 == Move the egg recipe into zc.buildout itself, and make zc.recipe.egg a dummy package that just points at zc.buildout.recipe.egg (or whatever we decide to call it). That solves that problem. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Lennart Regebro wrote:
<This is to a large part an angry rant. Sorry for that.>
In my effort to port zope.component to Python 3 I constantly encounter one problem that is very hard to solve: Circular dependencies. It's hard to port module A, when running the tests require module B to be ported, and module B depends on module A. Or, as with setuptools, it actually uses itself, so it has to work before you can test if it works. (Sorry, that's just insane).
So far the main circularity was that everything depended on zope.testing as a testrunner, zc.buildout for making the development environment, and zope.testing obviously depended on zope.interface etc. I solved that by also adding support for setuptools/distributes testrunner and using that instead. That fixed zope.interface, zope.event and zope.exception. These modules still have buildout configurations if you want them, but you don't need buildout anymore. zope.interface, zope.event and zope.exception can now be developed and tested with only setuptools, you can run the tests with "python setup.py test" both under Python 2 and 3.
Yay! That is a big win -- I'd like to see us automate testing this way, so that future development doesn't erode it. Developing ZTK packages using buildout should be a convenience, not a mandate.
But then the time came to zope.testing itself. It, rather annoyingly, depends on itself. I did the same there, but almost all tests are even more annoyingly, doctests, and depends on a custom version of doctest included, etc, etc, etc. The result was hours and hours of minor fiddling to find exactly what character in a doctest that made it fail. So I've spent an inordinate amount of time on zope.testing, and I have come to a conclusion about it:
== Suggestion 1 ==
I think we should deprecate zope.testing. Completely. There has been some discussion about deprecating it for something else, but I think we should just deprecate it. Just say "Don't use zope.testing, it's pure evil". We can recommend another testrunner, and it seems nose is winning the wars in the Python world there, but I don't think we necessarily need to do that. Most testrunners will find the tests by themselves, and the tests should be runnable with any testrunner, so which one you use is a minor issue.
I'm +0 on this myself. Certainly I don't think that we should have packages hard-wire dependencies on zope.testing in order to run their own tests.
Instead the big problem with getting rid of zope.testing is that many tests use other specific features like the doctest renormalizers etc. These could possibly be extracted into separate modules so you can use them with other testrunners.
Maybe instead we should strip zope.testing down so that it includes only these bits (the ones that get imported by third-party test code), and move the testrunner out to a separate package, e.g. 'zope.testrunner'. I think Marius Gedminas already suggested such a thing: we could update zc.recipe.testrunner to use the new package, which would make existing buildouts continue to wok.
But zope.testing is a mess that builds on two other messes, namely unittest (yes, it's a bloody mess, the API makes no sense) and doctest (which is a horror of a mess). unittest and doctest are horrid enough for us to not to make it worse with zope.testing in addition.
The path forward there would simply be to stop depending on zope.testing for all ZTK modules. That's probably a bit of work, but I think it's worth it.
I believe that the ZTK's charter is to have its packages be widely reusable, so making them testable without a custom testrunner seems like a great idea to me.
So, with that in mind I today went on to zc.buildout, trying to port it to Python 3 by ripping out any usage of zope.testing. Also, the standard development mode for zc.buildout is that you run a dev.py script, that uses zc.buildout to build a buildout for zc.buildout. Obviously, this circularity is the same as I encountered in setuptools, and it's evil. But with zc.buildout it's easy to get rid off at least as I could use setuptools testrunner instead. But then I realize the zc.buildout needs zc.recipe.egg. And zc.recipe.egg depends on zc.buildout. Hohum, another circular depedency.
Yup. I don't even know why zc.recipe.egg is broken out from buildout at all.
== Suggestion 2 ==
Move the egg recipe into zc.buildout itself, and make zc.recipe.egg a dummy package that just points at zc.buildout.recipe.egg (or whatever we decide to call it). That solves that problem.
+1 from me. Graphs with cycles are pure evil. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvIn/oACgkQ+gerLs4ltQ4B0ACfZCum/xeqimodDnJ84AlC9M51 G7kAoJ8p+oBYCkkDOzT7E4VCCfsQmRrq =ekI6 -----END PGP SIGNATURE-----
On Apr 16, 2010, at 1:35 PM, Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Lennart Regebro wrote:
...
So, with that in mind I today went on to zc.buildout, trying to port it to Python 3 by ripping out any usage of zope.testing. Also, the standard development mode for zc.buildout is that you run a dev.py script, that uses zc.buildout to build a buildout for zc.buildout. Obviously, this circularity is the same as I encountered in setuptools, and it's evil. But with zc.buildout it's easy to get rid off at least as I could use setuptools testrunner instead. But then I realize the zc.buildout needs zc.recipe.egg. And zc.recipe.egg depends on zc.buildout. Hohum, another circular depedency.
Yup. I don't even know why zc.recipe.egg is broken out from buildout at all.
When I was working on my "support system Python" branches of zc.buildout, I guessed it was because, if it is separate, then using zc.recipe.egg (which follows the standard ``recipe = `` idiom in the .cfg files) uses the same machinery that all the other recipes use. Yes, they are the same package, essentially, which is why zc.recipe.egg is part of the svn checkout of zc.buildout. I followed the same pattern in my branches for the z3c.recipe.scripts recipe. I certainly don't feel strongly about it, but it didn't seem outrageous to me.
== Suggestion 2 ==
Move the egg recipe into zc.buildout itself, and make zc.recipe.egg a dummy package that just points at zc.buildout.recipe.egg (or whatever we decide to call it). That solves that problem.
+1 from me. Graphs with cycles are pure evil.
I sure which my branches would land first, if they are ever to land. Any changes to buildout are a pain. Big changes are a big pain. Gary
On Fri, Apr 16, 2010 at 19:48, Gary Poster <gary.poster@gmail.com> wrote:
When I was working on my "support system Python" branches of zc.buildout, I guessed it was because, if it is separate, then using zc.recipe.egg (which follows the standard ``recipe = `` idiom in the .cfg files) uses the same machinery that all the other recipes use.
Yes, they are the same package, essentially, which is why zc.recipe.egg is part of the svn checkout of zc.buildout.
I followed the same pattern in my branches for the z3c.recipe.scripts recipe.
I certainly don't feel strongly about it, but it didn't seem outrageous to me.
One problem with it is that it's tricky to work on two modules at once, unless you have buildout... And since I in my branch removed the buildouts usage of itself, I don't. So that means that zc.buildout doens't have zc.recipe.egg, unless I install it, which I can't, becuase I haven't ported it yet, which I can't do untill I posrt zc. buildout which... yeah, you get it. Outrageous? No. Bad? Yes. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64
Tres Seaver a écrit :
So far the main circularity was that everything depended on zope.testing as a testrunner, zc.buildout for making the development environment, and zope.testing obviously depended on zope.interface etc. I solved that by also adding support for setuptools/distributes testrunner and using that instead. That fixed zope.interface, zope.event and zope.exception. These modules still have buildout configurations if you want them, but you don't need buildout anymore. zope.interface, zope.event and zope.exception can now be developed and tested with only setuptools, you can run the tests with "python setup.py test" both under Python 2 and 3.
Yay! That is a big win -- I'd like to see us automate testing this way, so that future development doesn't erode it. Developing ZTK packages using buildout should be a convenience, not a mandate.
Depending on setuptools for tests in another evil thing. We should not assume a *setup* tool to be a testrunner, and we should not depend on the behaviour of setuptools to write tests. Setuptools is already doing too many things and Tarek is taking care (with distutils2 and distribute) to cleany separate the functionalities, such as installing, distributing, etc. Testing is not related to configuring nor installing, and "python setup.py test" is no more meaningful than "python setup.py makethecoffee". Christophe
On Sun, Apr 18, 2010 at 10:37, Christophe Combelles <ccomb@free.fr> wrote:
Depending on setuptools for tests in another evil thing. We should not assume a *setup* tool to be a testrunner, and we should not depend on the behaviour of setuptools to write tests.
I'm not sure what you mean there, setuptools doesn't behave in any particular way when running tests.
Setuptools is already doing too many things and Tarek is taking care (with distutils2 and distribute) to cleany separate the functionalities, such as installing, distributing, etc.
Yes, but that work is not done yet. And one thing that's needed to be able to run the tests under Python 3 is building it. So it's hard to separate the building and the running of testsm as a Python 3 capable testrunner would need to run setup.py build first anyway. Which means you depends on setuptools/distribute whichever way you turn.
Testing is not related to configuring nor installing, and "python setup.py test" is no more meaningful than "python setup.py makethecoffee".
It might seem so, but it is unfortunately not true when considering Python 3. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
Lennart Regebro a écrit :
On Sun, Apr 18, 2010 at 10:37, Christophe Combelles <ccomb@free.fr> wrote:
Depending on setuptools for tests in another evil thing. We should not assume a *setup* tool to be a testrunner, and we should not depend on the behaviour of setuptools to write tests.
I'm not sure what you mean there, setuptools doesn't behave in any particular way when running tests.
ok, not a lot, there is currently one single particular behaviour, which is adding tests from the result of the "additional_tests" function in a module. That probably won't hurt.
Setuptools is already doing too many things and Tarek is taking care (with distutils2 and distribute) to cleany separate the functionalities, such as installing, distributing, etc.
Yes, but that work is not done yet. And one thing that's needed to be able to run the tests under Python 3 is building it. So it's hard to separate the building and the running of testsm as a Python 3 capable testrunner would need to run setup.py build first anyway. Which means you depends on setuptools/distribute whichever way you turn.
Testing is not related to configuring nor installing, and "python setup.py test" is no more meaningful than "python setup.py makethecoffee".
It might seem so, but it is unfortunately not true when considering Python 3.
On Sun, Apr 18, 2010 at 11:35, Christophe Combelles <ccomb@free.fr> wrote:
ok, not a lot, there is currently one single particular behaviour, which is adding tests from the result of the "additional_tests" function in a module. That probably won't hurt.
Yeah, setuptools test discovery isn't very good so you need to be explicit. Many modules also only have one test_suite, and then you can point directly to that one. -- Lennart Regebro: http://regebro.wordpress.com/ Python 3 Porting: http://python-incompatibility.googlecode.com/ +33 661 58 14 64
On 4/18/10 10:37 , Christophe Combelles wrote:
Tres Seaver a écrit :
So far the main circularity was that everything depended on zope.testing as a testrunner, zc.buildout for making the development environment, and zope.testing obviously depended on zope.interface etc. I solved that by also adding support for setuptools/distributes testrunner and using that instead. That fixed zope.interface, zope.event and zope.exception. These modules still have buildout configurations if you want them, but you don't need buildout anymore. zope.interface, zope.event and zope.exception can now be developed and tested with only setuptools, you can run the tests with "python setup.py test" both under Python 2 and 3.
Yay! That is a big win -- I'd like to see us automate testing this way, so that future development doesn't erode it. Developing ZTK packages using buildout should be a convenience, not a mandate.
Depending on setuptools for tests in another evil thing. We should not assume a *setup* tool to be a testrunner, and we should not depend on the behaviour of setuptools to write tests.
All zope.* packages already depend on setuptools to setup the namespace, so I don't see a problem with using setuptools to run tests. The dependency is already there. Wichert.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Christophe Combelles wrote:
Tres Seaver a écrit :
So far the main circularity was that everything depended on zope.testing as a testrunner, zc.buildout for making the development environment, and zope.testing obviously depended on zope.interface etc. I solved that by also adding support for setuptools/distributes testrunner and using that instead. That fixed zope.interface, zope.event and zope.exception. These modules still have buildout configurations if you want them, but you don't need buildout anymore. zope.interface, zope.event and zope.exception can now be developed and tested with only setuptools, you can run the tests with "python setup.py test" both under Python 2 and 3. Yay! That is a big win -- I'd like to see us automate testing this way, so that future development doesn't erode it. Developing ZTK packages using buildout should be a convenience, not a mandate.
Depending on setuptools for tests in another evil thing. We should not assume a *setup* tool to be a testrunner, and we should not depend on the behaviour of setuptools to write tests. Setuptools is already doing too many things and Tarek is taking care (with distutils2 and distribute) to cleany separate the functionalities, such as installing, distributing, etc.
Testing is not related to configuring nor installing, and "python setup.py test" is no more meaningful than "python setup.py makethecoffee".
The developers of the software are not the only folks who run its tests. In particular, "downstream" consumers need to run them, too, Keeping the burden of running them as low as possible helps make sure that folks who are considering using the project can run its tests easily, as well as tools like 'cheesecake', If we want to encourage "casual" contributors to test the code (e.g., users who finnd a bug, or help out during a bug day) we need to make it easy and documented how to do so. The "hacking" page I wrote for zope.event is aimed at such contributors, and describes both ways to run the tests: http://palladion.com/static/zope.event-docs/hacking.html Finally, keeeping the required infrastructure for running the ZTK tests minima is essential to the topic of this thread, which is helping Lennart's efforts porting to Zope3. Right now, their reliance on zc.buildout + zope.testing for running the tests is the major blocker to porting the other ZTK packages to Python 3. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iEYEARECAAYFAkvK+WgACgkQ+gerLs4ltQ7AFACeOG/i5SIU1G3pp9dpzeBFFl6C oTsAoNTyMqg+Oa1kNbG9EuVl2yqFV4iW =jUS3 -----END PGP SIGNATURE-----
On Fri, Apr 16, 2010 at 6:05 PM, Lennart Regebro <regebro@gmail.com> wrote: ...
== Suggestion 1 ==
I think we should deprecate zope.testing. Completely. There has been some discussion about deprecating it for something else, but I think we should just deprecate it. Just say "Don't use zope.testing, it's pure evil". We can recommend another testrunner, and it seems nose is winning the wars in the Python world there, but I don't think we necessarily need to do that. Most testrunners will find the tests by themselves, and the tests should be runnable with any testrunner, so which one you use is a minor issue.
As the author of one of those other testrunners, I can tell you that if you do this you'll find that your number one biggest problem is getting layers to work.
But zope.testing is a mess that builds on two other messes, namely unittest (yes, it's a bloody mess, the API makes no sense) and doctest (which is a horror of a mess).
unittest's API makes sense and it's not a mess. It's got problems, but they are defects and not bone-deep. Further, the current maintainer of unittest, Michael Foord, has been doing some excellent work in addressing those defects. The testing-in-python mailing list[1] is probably the best place to take up the issue if you wish to do so. jml [1] http://lists.idyll.org/listinfo/testing-in-python
On Fri, Apr 16, 2010 at 19:53, Jonathan Lange <jml@mumak.net> wrote:
As the author of one of those other testrunners, I can tell you that if you do this you'll find that your number one biggest problem is getting layers to work.
Ah, right, layers are in zope.testing too. To actually get widespread movement from zope.testing we would have to make some other layer support. Hm...
unittest's API makes sense and it's not a mess.
No, the API to run tests doesn't make any sense, but we can take that discussion some other day. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
Hi Lennart & co, On 17 April 2010 02:38, Lennart Regebro <regebro@gmail.com> wrote:
On Fri, Apr 16, 2010 at 19:53, Jonathan Lange <jml@mumak.net> wrote:
As the author of one of those other testrunners, I can tell you that if you do this you'll find that your number one biggest problem is getting layers to work.
Ah, right, layers are in zope.testing too. To actually get widespread movement from zope.testing we would have to make some other layer support. Hm...
As you may know, I've been working on plone.testing. This is mainly to make it easier for people working on Plone packages to write "good" tests (and a lot of it is just about patterns, rather than code), but in fact it doesn't depend on Plone (and only soft-depends on Zope 2). I'm certainly hoping to use it for my "plain Zope 3/Toolkit" packages in the future. plone.testing makes very heavy use of layers. I think layers are a great feature, when done properly, and I haven't seen any better approach. The setUpClass/setUpModule stuff in unittest2 is nice, but doesn't really solve the same problem. For integration testing with something as complex as Zope 2 (or, arguably, the ZODB, various bits of the ZTK, and so on) layers allow "us" the framework authors to make life much easier for those people who are not experts in the framework. Anyway, a few high level observations: - In neither plone.testing (apart from its own tests), nor in code that uses it, would I imagine actually depending on zope.testing via an import. We use unittest(2) and doctest from the standard library. - We do depend on a zope.testing-compatible test runner, in that we expect layers to work. In reality, we depend on zc.recipe.testrunner, though I would *love* to be able to do 'python setup.py test' as well (and have that execute tests with layers). I have no idea how that works or whether it'd be possible. - The way zope.testing promotes writing layers is actually pretty evil. It overloads the 'class' keyword, essentially, making you spell "base layers" as base classes. This has a few problems: - If your "base layer" has a testTearDown method, say, and your layer doesn't, the base class version will actually inherit into the child layer. zope.testing will then run the same code twice, once for the base layer and once for the child layer. - You can't use OOP inheritance to re-use layer conveniences. - People get quite confused. :) - You can't have two copies of a layer - all layers are module-level global singletons. This becomes somewhat important when layers manage and expose resources (like database connections). Anyway, based on Ross Patterson's work in collective.testcaselayer, I think we have something better in plone.testing's layer module. General info and patterns: http://svn.plone.org/svn/plone/plone.testing/trunk/README.txt Layer module: http://svn.plone.org/svn/plone/plone.testing/trunk/src/plone/testing/layer.p... Layer doctests: http://svn.plone.org/svn/plone/plone.testing/trunk/src/plone/testing/layer.t... If I could have my cake and eat it too, I'd like: - A test runner that supports layers - A test runner that supports the new fixture methods in unittest2/Python 2.7 - setUpClass and setUpModule - A test runner that properly reports skipped tests (pretty easy) - A test runner that supports the above begin run from 'setup.py test' - A test runner installable via a buildout part just like zc.recipe.testrunner We could quite possibly factor layer.py out of plone.testing and push it into something else if that was desirable. It's self-contained and has no dependencies. Martin
On Sat, Apr 17, 2010 at 06:34, Martin Aspeli <optilude+lists@gmail.com> wrote:
Hi Lennart & co,
On 17 April 2010 02:38, Lennart Regebro <regebro@gmail.com> wrote:
On Fri, Apr 16, 2010 at 19:53, Jonathan Lange <jml@mumak.net> wrote:
As the author of one of those other testrunners, I can tell you that if you do this you'll find that your number one biggest problem is getting layers to work.
Ah, right, layers are in zope.testing too. To actually get widespread movement from zope.testing we would have to make some other layer support. Hm...
As you may know, I've been working on plone.testing. This is mainly to make it easier for people working on Plone packages to write "good" tests (and a lot of it is just about patterns, rather than code), but in fact it doesn't depend on Plone (and only soft-depends on Zope 2). I'm certainly hoping to use it for my "plain Zope 3/Toolkit" packages in the future.
plone.testing makes very heavy use of layers. I think layers are a great feature, when done properly, and I haven't seen any better approach. The setUpClass/setUpModule stuff in unittest2 is nice, but doesn't really solve the same problem. For integration testing with something as complex as Zope 2 (or, arguably, the ZODB, various bits of the ZTK, and so on) layers allow "us" the framework authors to make life much easier for those people who are not experts in the framework.
Anyway, a few high level observations:
- In neither plone.testing (apart from its own tests), nor in code that uses it, would I imagine actually depending on zope.testing via an import. We use unittest(2) and doctest from the standard library.
- We do depend on a zope.testing-compatible test runner, in that we expect layers to work. In reality, we depend on zc.recipe.testrunner, though I would *love* to be able to do 'python setup.py test' as well (and have that execute tests with layers). I have no idea how that works or whether it'd be possible.
- The way zope.testing promotes writing layers is actually pretty evil. It overloads the 'class' keyword, essentially, making you spell "base layers" as base classes. This has a few problems:
- If your "base layer" has a testTearDown method, say, and your layer doesn't, the base class version will actually inherit into the child layer. zope.testing will then run the same code twice, once for the base layer and once for the child layer.
- You can't use OOP inheritance to re-use layer conveniences.
- People get quite confused. :)
- You can't have two copies of a layer - all layers are module-level global singletons. This becomes somewhat important when layers manage and expose resources (like database connections).
Anyway, based on Ross Patterson's work in collective.testcaselayer, I think we have something better in plone.testing's layer module.
Aha, interesting.
General info and patterns: http://svn.plone.org/svn/plone/plone.testing/trunk/README.txt Layer module: http://svn.plone.org/svn/plone/plone.testing/trunk/src/plone/testing/layer.p... Layer doctests: http://svn.plone.org/svn/plone/plone.testing/trunk/src/plone/testing/layer.t...
If I could have my cake and eat it too, I'd like:
- A test runner that supports layers - A test runner that supports the new fixture methods in unittest2/Python 2.7 - setUpClass and setUpModule - A test runner that properly reports skipped tests (pretty easy) - A test runner that supports the above begin run from 'setup.py test' - A test runner installable via a buildout part just like zc.recipe.testrunner
We could quite possibly factor layer.py out of plone.testing and push it into something else if that was desirable. It's self-contained and has no dependencies.
All this sounds good and not to complicated compared to extracting the testrunner itself. :) Which in turn is less complicated than the work you've already done. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64
Hi, On 04/17/2010 06:34 AM, Martin Aspeli wrote:
Hi Lennart& co,
On 17 April 2010 02:38, Lennart Regebro<regebro@gmail.com> wrote:
On Fri, Apr 16, 2010 at 19:53, Jonathan Lange<jml@mumak.net> wrote:
As the author of one of those other testrunners, I can tell you that if you do this you'll find that your number one biggest problem is getting layers to work.
Ah, right, layers are in zope.testing too. To actually get widespread movement from zope.testing we would have to make some other layer support. Hm...
As you may know, I've been working on plone.testing. This is mainly to make it easier for people working on Plone packages to write "good" tests (and a lot of it is just about patterns, rather than code), but in fact it doesn't depend on Plone (and only soft-depends on Zope 2). I'm certainly hoping to use it for my "plain Zope 3/Toolkit" packages in the future.
plone.testing makes very heavy use of layers. I think layers are a great feature, when done properly, and I haven't seen any better approach. The setUpClass/setUpModule stuff in unittest2 is nice, but doesn't really solve the same problem. For integration testing with something as complex as Zope 2 (or, arguably, the ZODB, various bits of the ZTK, and so on) layers allow "us" the framework authors to make life much easier for those people who are not experts in the framework.
Anyway, a few high level observations:
- In neither plone.testing (apart from its own tests), nor in code that uses it, would I imagine actually depending on zope.testing via an import. We use unittest(2) and doctest from the standard library.
- We do depend on a zope.testing-compatible test runner, in that we expect layers to work. In reality, we depend on zc.recipe.testrunner, though I would *love* to be able to do 'python setup.py test' as well (and have that execute tests with layers). I have no idea how that works or whether it'd be possible.
- The way zope.testing promotes writing layers is actually pretty evil. It overloads the 'class' keyword, essentially, making you spell "base layers" as base classes. This has a few problems:
- If your "base layer" has a testTearDown method, say, and your layer doesn't, the base class version will actually inherit into the child layer. zope.testing will then run the same code twice, once for the base layer and once for the child layer.
- You can't use OOP inheritance to re-use layer conveniences.
- People get quite confused. :)
- You can't have two copies of a layer - all layers are module-level global singletons. This becomes somewhat important when layers manage and expose resources (like database connections).
Minor note: zope.testing *promotes* layers the wrong way and zope.app.testing definitely implements them the wrong way. However, the testrunner itself deals with them in a way that allows you to use them correctly. I think the situation itself could be remedied if the actual layers that are around would be rewritten and a few annoyances (handling of missing methods) straightened out. -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Tue, Apr 20, 2010 at 12:09 PM, Christian Theune <ct@gocept.com> wrote:
Minor note: zope.testing *promotes* layers the wrong way and zope.app.testing definitely implements them the wrong way.
That's prety vague. Could you say specifically in what ways zope.testing promotes layers the wrong way and zope.app.testing uses them the wrong way? Jim -- Jim Fulton
On 04/20/2010 08:44 PM, Jim Fulton wrote:
On Tue, Apr 20, 2010 at 12:09 PM, Christian Theune<ct@gocept.com> wrote:
Minor note: zope.testing *promotes* layers the wrong way and zope.app.testing definitely implements them the wrong way.
That's prety vague. Could you say specifically in what ways zope.testing promotes layers the wrong way and
zope.testing uses the attribute '__bases__' to store the information what the base layers are. __*__ are supposedly Python internal attributes. Specifically __bases__ is known to be used to store information which base classes a class has. Looking at this I (and others too) get directed towards: aha, so layers are classes and use inheritance to signal what base layers are. Which is exactly not what is happening.
zope.app.testing uses them the wrong way?
Actually it doesn't. I confused this with Zope 2's Testing: There's Testing/ZopeTestCase/layer.py which defines a class with classmethods and in a similar fashion there is Products.PloneTestCase that defines classes, derives them and thus kind of piggybacks on the class inheritance mechanism to establish __bases__ paired with static methods but without actually inheriting methods. We struggled through some hairy details that I fail to remember when we worked on gocept.selenium last year which tries to establish a generic layer that can be combined with others. Looking at the layer code in gocept.selenium right now it feels relatively clean, although the exact choice of attribute names IMHO is confusing for anyone trying to understand whats going on and why. Christian -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Tue, Apr 20, 2010 at 2:58 PM, Christian Theune <ct@gocept.com> wrote:
zope.testing uses the attribute '__bases__' to store the information what the base layers are. __*__ are supposedly Python internal attributes. Specifically __bases__ is known to be used to store information which base classes a class has.
This sort of misdirection has, unfortunately, a long history in Zope 3 (and the various things that's become). Witness __name__. (There are others, but most aren't *actually* used by Python implementations.) Removing existing __*__ name ("underware") usage is probably untenable, but I hope we can avoid extending our foolishness. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
On 04/20/2010 09:04 PM, Fred Drake wrote:
On Tue, Apr 20, 2010 at 2:58 PM, Christian Theune<ct@gocept.com> wrote:
zope.testing uses the attribute '__bases__' to store the information what the base layers are. __*__ are supposedly Python internal attributes. Specifically __bases__ is known to be used to store information which base classes a class has.
This sort of misdirection has, unfortunately, a long history in Zope 3 (and the various things that's become). Witness __name__. (There are others, but most aren't *actually* used by Python implementations.)
Removing existing __*__ name ("underware") usage is probably untenable, but I hope we can avoid extending our foolishness.
Yeah, I guess so: my impression of the OO structure after understanding what's going on was quite satisfied. It's just very misleading. :) -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Tue, Apr 20, 2010 at 3:04 PM, Fred Drake <fdrake@gmail.com> wrote:
On Tue, Apr 20, 2010 at 2:58 PM, Christian Theune <ct@gocept.com> wrote:
zope.testing uses the attribute '__bases__' to store the information what the base layers are. __*__ are supposedly Python internal attributes. Specifically __bases__ is known to be used to store information which base classes a class has.
This sort of misdirection has, unfortunately, a long history in Zope 3 (and the various things that's become). Witness __name__. (There are others, but most aren't *actually* used by Python implementations.)
Removing existing __*__ name ("underware") usage is probably untenable, but I hope we can avoid extending our foolishness.
OK, let's stop new uses of __*__ names. We won't provide initializers for classes, or implement operations either. The use of protocols like __name__ and __bases__ is intended to conform to common usage in Python. Let's invent new names that are specific to our own frameworks. I can live with saying we shouldn't invent new __*__ names, even though I consider that a meta protocol. I get annoyed at criticism for following standard protocols. Jim -- Jim Fulton
On 04/20/2010 09:24 PM, Jim Fulton wrote:
On Tue, Apr 20, 2010 at 3:04 PM, Fred Drake<fdrake@gmail.com> wrote:
On Tue, Apr 20, 2010 at 2:58 PM, Christian Theune<ct@gocept.com> wrote:
zope.testing uses the attribute '__bases__' to store the information what the base layers are. __*__ are supposedly Python internal attributes. Specifically __bases__ is known to be used to store information which base classes a class has.
This sort of misdirection has, unfortunately, a long history in Zope 3 (and the various things that's become). Witness __name__. (There are others, but most aren't *actually* used by Python implementations.)
Removing existing __*__ name ("underware") usage is probably untenable, but I hope we can avoid extending our foolishness.
OK, let's stop new uses of __*__ names. We won't provide initializers for classes, or implement operations either.
The use of protocols like __name__ and __bases__ is intended to conform to common usage in Python.
Let's invent new names that are specific to our own frameworks.
I can live with saying we shouldn't invent new __*__ names, even though I consider that a meta protocol. I get annoyed at criticism for following standard protocols.
I don't know (or at least can't remember) all of the history of the discussion about that, but I wouldn't argue about following standard protocols. I think the issue is with that it's not standard protocol the way we use it - at least I can't find our use of __bases__ documented in Python's documentation[1] about __bases__ and thus have a hard time saying we're following standard protocols. Christian [1] .. http://docs.python.org/library/stdtypes.html?highlight=__bases__#class.__bas... -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Tue, Apr 20, 2010 at 3:47 PM, Christian Theune <ct@gocept.com> wrote:
I don't know (or at least can't remember) all of the history of the discussion about that, but I wouldn't argue about following standard protocols.
It *is* arguable that __name__ is a standard protocol. It's also not clear that our using it as we do is really the same thing. (And I don't think it's interesting to discuss whether we do the right thing or not.)
I think the issue is with that it's not standard protocol the way we use it - at least I can't find our use of __bases__ documented in Python's documentation[1] about __bases__ and thus have a hard time saying we're following standard protocols.
Our uses of __bases__ and __parent__ don't match Python, and there's a general BDFL proclamation that underware are for Python implementations (IIRC). While we can argue that our use is reasonable, the fact that there's reasonable dissent suggests something different would have been a better choice. -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
On 04/20/2010 10:05 PM, Fred Drake wrote:
On Tue, Apr 20, 2010 at 3:47 PM, Christian Theune<ct@gocept.com> wrote:
I don't know (or at least can't remember) all of the history of the discussion about that, but I wouldn't argue about following standard protocols.
It *is* arguable that __name__ is a standard protocol. It's also not clear that our using it as we do is really the same thing. (And I don't think it's interesting to discuss whether we do the right thing or not.)
I think the issue is with that it's not standard protocol the way we use it - at least I can't find our use of __bases__ documented in Python's documentation[1] about __bases__ and thus have a hard time saying we're following standard protocols.
Our uses of __bases__ and __parent__ don't match Python, and there's a general BDFL proclamation that underware are for Python implementations (IIRC). While we can argue that our use is reasonable, the fact that there's reasonable dissent suggests something different would have been a better choice.
I think the two of us agreeing, right? -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Tue, Apr 20, 2010 at 4:09 PM, Christian Theune <ct@gocept.com> wrote:
I think the two of us agreeing, right?
You & I are. I think Jim's not (which is fine, of course). -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
On Tue, Apr 20, 2010 at 4:05 PM, Fred Drake <fdrake@gmail.com> wrote: ...
I think the issue is with that it's not standard protocol the way we use it - at least I can't find our use of __bases__ documented in Python's documentation[1] about __bases__ and thus have a hard time saying we're following standard protocols.
Our uses of __bases__ and __parent__ don't match Python,
We disagree wrt __bases__.
and there's a general BDFL proclamation that underware are for Python implementations (IIRC).
That proclamation changed over time. It was much weaker in the past. It was strengthened with "applications should not expect to define additional names using this convention. The set of names of this class defined by Python may be extended in future versions." in Python 2.3. In fact, Guido was aware of and didn't object to my use of __*__names.
While we can argue that our use is reasonable, the fact that there's reasonable dissent suggests something different would have been a better choice.
The fact that there is dissent from a choice doesn't mean that it is wrong. I'm not saying that my use of __*__s was "right" in any absolute sense. I get that there are differences of opinion. To say that zope.testing "promotes" layers the wrong way, simply because it used the name __bases__, which doesn't even go against the BDFL's pronouncement on the use of __*__s is misleading at best. Jim -- Jim Fulton
On 04/20/2010 10:48 PM, Jim Fulton wrote:
On Tue, Apr 20, 2010 at 4:05 PM, Fred Drake<fdrake@gmail.com> wrote: ...
I think the issue is with that it's not standard protocol the way we use it - at least I can't find our use of __bases__ documented in Python's documentation[1] about __bases__ and thus have a hard time saying we're following standard protocols.
Our uses of __bases__ and __parent__ don't match Python,
We disagree wrt __bases__.
and there's a general BDFL proclamation that underware are for Python implementations (IIRC).
That proclamation changed over time. It was much weaker in the past. It was strengthened with
"applications should not expect to define additional names using this convention. The set of names of this class defined by Python may be extended in future versions."
in Python 2.3.
In fact, Guido was aware of and didn't object to my use of __*__names.
While we can argue that our use is reasonable, the fact that there's reasonable dissent suggests something different would have been a better choice.
The fact that there is dissent from a choice doesn't mean that it is wrong.
I'm not saying that my use of __*__s was "right" in any absolute sense. I get that there are differences of opinion.
To say that zope.testing "promotes" layers the wrong way, simply because it used the name __bases__, which doesn't even go against the BDFL's pronouncement on the use of __*__s is misleading at best.
Something in this discussion thread is borked. I sense tension. I sense that the argument doesn't have direction and I don't feel invited to share my thoughts. I'd like to figure out why that is because I want this to happen less often. (Right now this caused me to spend 30 minutes not writing a technical answer which makes me sad because I think in real life this issue could have been debated much more quickly and constructively.) For now I'll go to bed, maybe looking at it tomorrow will help. -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
Hi Christian, On 21 April 2010 02:58, Christian Theune <ct@gocept.com> wrote:
On 04/20/2010 08:44 PM, Jim Fulton wrote:
On Tue, Apr 20, 2010 at 12:09 PM, Christian Theune<ct@gocept.com> wrote:
Minor note: zope.testing *promotes* layers the wrong way and zope.app.testing definitely implements them the wrong way.
That's prety vague. Could you say specifically in what ways zope.testing promotes layers the wrong way and
zope.testing uses the attribute '__bases__' to store the information what the base layers are. __*__ are supposedly Python internal attributes. Specifically __bases__ is known to be used to store information which base classes a class has.
Looking at this I (and others too) get directed towards: aha, so layers are classes and use inheritance to signal what base layers are. Which is exactly not what is happening.
In fact, it's a little worse than that. Consider this pair of layers: class Base: @classmethod def setUp(cls): print "Setting up base" @classmethod def tearDown(cls): print "Tearing down base" class Child(Base): @classmethod def setUp(cls): print "Setting up child" Note that there's no tearDown on the child (perhaps it doesn't need one). What actually happens in this case is that the test runner still finds a tearDown on Child, it's just that it's inherited from Base. So in effect, Base's tearDown is called twice. This also happens with things like testSetUp() and testTearDown(). If the base defines them and a child doesn't, they're called twice. The other problem is that it's hard to also use inheritance in the OOP sense to re-use layer logic. Also, if the layer manages any state, it has to be set as a class variable (on cls), which is effectively global. If you want to re-use a layer but isolate the resources its creates from those created by existing layers, you have to re-implement the layer. These insights by Ross Patterson led to collective.testcaselayer, which was lightly refactored into the layer module of the nascent plone.testing. See: http://svn.plone.org/svn/plone/plone.testing/trunk/src/plone/testing/layer.p... http://svn.plone.org/svn/plone/plone.testing/trunk/src/plone/testing/layer.t... http://svn.plone.org/svn/plone/plone.testing/trunk/README.txt This module also contains an implementation of a resource manager that allows layers to define shared resources in a stack that lets child layers shadow those resources (i.e. provide a changed fixture). We use this for things like ZODB connections and Zope 2 app roots. It's explained best in the README, and tested in layer.txt. Having used this pattern for a while, I'm pretty sure it's an improvement on the layers-are-classes thing, which in addition to the problems above, has caused a fair amount of confusion.
> zope.app.testing uses them the wrong way?
Actually it doesn't. I confused this with Zope 2's Testing:
There's Testing/ZopeTestCase/layer.py which defines a class with classmethods and in a similar fashion there is Products.PloneTestCase that defines classes, derives them and thus kind of piggybacks on the class inheritance mechanism to establish __bases__ paired with static methods but without actually inheriting methods.
FTR, the ZopeTestCase mess is basically what plone.testing.z2 tries to fix (insofar as it's possible). The PloneTestCase mess will hopefully be fixed by a plone.app.testing building on plone.testing.
We struggled through some hairy details that I fail to remember when we worked on gocept.selenium last year which tries to establish a generic layer that can be combined with others.
You're not the only one. ;-) Martin
participants (10)
-
Christian Theune -
Christophe Combelles -
Fred Drake -
Gary Poster -
Jim Fulton -
Jonathan Lange -
Lennart Regebro -
Martin Aspeli -
Tres Seaver -
Wichert Akkerman