Re: [Zope-dev] Zope unit testing framework?
Martijn Faassen <faassen@vet.uu.nl>
Hi there,
Recently I've been reading some books [1] which put emphasis on the idea of unit testing. At the Python conference Joel Shprentz held a short talk about doing unit tests in Python [2]
Basically a unit tests are some simple test functions that are associated with a unit of code (a python module). The tests exercise the code in the module, and either succeed or fail (there may also be an error due to an uncaught exception). Each time you change something in the module you run the associated tests. The idea is that this makes debugging easier, catching bugs more quickly and giving better diagnostics. Doing unit tests in Python isn't hard to do.
I'm having more trouble thinking of a good way to do unit tests in Zope, though. Partially this is because Zope has various parts (the management interface for folders, ZClasses, product development), which each may need a different testing strategy. Partially this is because Zope currently mixes model and view together (though there are thoughts to do something about this [3]) -- unit tests are about testing code (methods and such), not user interfaces. Another problem is that much Zope code is not easy to test in isolation from Zope. Too much of the code depends on various Zope facilities (base classes, object database integration).
Has anybody any ideas on how to do unit testing within the Zope framework? I'd like to try adopting somekind of unit test strategy for my Zope development.
I got addicted to unit testing after reading John Lakos' excellent book, _Large Scale C++ Software Design_. He emphasises not only unit tests, but a more elaborate strategy, "design for testability:" not only does each module come with its own tests, but the system knows about dependencies between modules, and guarantees to test the depended-on modules before their dependents (so as to avoid spurious errors). In the Zope world, I think this has to translate to somthing like: 1. Each Python module should have a set of associated tests, which would by preference live inside the "if __name__ == '__main__'" block at the bottom of the module (an alternative would be to use Tim Peters' DocTest stuff, which embeds the unit tests in docstrings). One would then have a script, like the one which compiles all the Python modules during installation, to run these modules (perhaps using some fancy dependency analysis based on the import statements?) 2. For testing ZClasses, I am afraid that some kind of ZClient-based test will be required. Each test would: * Generate a suitably-unique folder name. * Construct a folder of that name off the root of the ZODB. * Build all scaffolding objects inside the new folder. * Run the tests, capturing the results and comparing them against expectations. * Finally, blow away the sandbox folder. Perhaps, as you point out, some of the effort to disentangle the core Python model from the HTML view will make such testing easier to lay down in the Zope core itself. Given a suitable framework, product developers could even allow third parties to contribute tests highlighting bugs they have found.
[1] Martin Fowler, ed., Refactoring: Improving the Design of Existing Code, Addison Wesley Longman, 1999
This is the best book I have read on the actual practice of "Extreme Programming" -- I recommend it highly.
Kent Beck, Extreme Programming Explained, Addison Wesley Longman, 1999
This book is a somewhat higher-level view of XP, and an "apologetic" for it (in the old, technical sense). Still quite useful to read, but not as much meat for the working programmer to digest.
[2] http://www.python.org/workshops/2000-01/proceedings/posters/shprentz/shprent... http://c2.com/cgi/wiki?PythonUnit
I have put up the code I currently use for unit testing Python modules at: http://www.zope.org/Members/tseaver/PyUnit
[3] http://www.zope.org/Resources/Mozilla/ZWiki/ZopeMVC http://www.zope.org/Resources/Mozilla/ZWiki/ModelViewController
-- ========================================================= Tres Seaver tseaver@palladion.com 713-523-6582 Palladion Software http://www.palladion.com
Tres Seaver wrote:
I got addicted to unit testing after reading John Lakos' excellent book, _Large Scale C++ Software Design_. He emphasises not only unit tests, but a more elaborate strategy, "design for testability:" not only does each module come with its own tests, but the system knows about dependencies between modules, and guarantees to test the depended-on modules before their dependents (so as to avoid spurious errors).
In fact I also read that book a while back, and it was the first to set me on of good testing strategies. I don't think Lakos discusses tests that automatically check the outcome though, as the UnitTest frameworks are doing. The advantage there is that you don't have to hand-verify the test output, and that you can test often (so you'll see quickly when something goes wrong). Lakos's book is very insightful though. And then I discovered Python and since then most of my hacking in C++ has been on a hold..
In the Zope world, I think this has to translate to somthing like:
1. Each Python module should have a set of associated tests, which would by preference live inside the "if __name__ == '__main__'" block at the bottom of the module (an alternative would be to use Tim Peters' DocTest stuff, which embeds the unit tests in docstrings). One would then have a script, like the one which compiles all the Python modules during installation, to run these modules (perhaps using some fancy dependency analysis based on the import statements?)
How would this work for things which use the Zope framework heavily, though? It's perfectly possible to do associate somekind of unit testing with Python modules, but the integration with the Zope framework seems to require having to use the framework in order to do the tests. That could slow the tests down a lot or make them too hard to perform, and the philosophy of these tests is that they should run quickly and should be easy to run, so you'll run them often.
2. For testing ZClasses, I am afraid that some kind of ZClient-based test will be required. Each test would:
* Generate a suitably-unique folder name.
* Construct a folder of that name off the root of the ZODB.
Or perhaps in some special 'scratchpad area'.
* Build all scaffolding objects inside the new folder.
* Run the tests, capturing the results and comparing them against expectations.
* Finally, blow away the sandbox folder.
Perhaps, as you point out, some of the effort to disentangle the core Python model from the HTML view will make such testing easier to lay down in the Zope core itself. Given a suitable framework, product developers could even allow third parties to contribute tests highlighting bugs they have found.
[1] Martin Fowler, ed., Refactoring: Improving the Design of Existing Code, Addison Wesley Longman, 1999
This is the best book I have read on the actual practice of "Extreme Programming" -- I recommend it highly.
Likewise. Like "Design Patterns", it talks about things about which you already know but until then had no good way to talk about, and then adds some useful things you *didn't* already know.
Kent Beck, Extreme Programming Explained, Addison Wesley Longman, 1999
This book is a somewhat higher-level view of XP, and an "apologetic" for it (in the old, technical sense). Still quite useful to read, but not as much meat for the working programmer to digest.
I agree with your assessment here too.
[2] http://www.python.org/workshops/2000-01/proceedings/posters/shprentz/shprent... http://c2.com/cgi/wiki?PythonUnit
I have put up the code I currently use for unit testing Python modules at:
Cool! Regards, Martijn
How would this work for things which use the Zope framework heavily, though? It's perfectly possible to do associate somekind of unit testing with Python modules, but the integration with the Zope framework seems to require having to use the framework in order to do the tests. That could slow the tests down a lot or make them too hard to perform, and the philosophy of these tests is that they should run quickly and should be easy to run, so you'll run them often.
Seems like it should be reasonable to write test scripts that use the framework for most applications. import Zope, ZPublisher ZPublisher.Zope(mytest) ... etc. This would be a bit of a time hit to load the framework, but shouldn't be prohibitive, especially on a fast machine. --jfarr
Martijn Faassen wrote:
Tres Seaver wrote:
I got addicted to unit testing after reading John Lakos' excellent book, _Large Scale C++ Software Design_. He emphasises not only unit tests, but a more elaborate strategy, "design for testability:" not only does each module come with its own tests, but the system knows about dependencies between modules, and guarantees to test the depended-on modules before their dependents (so as to avoid spurious errors).
In fact I also read that book a while back, and it was the first to set me on of good testing strategies. I don't think Lakos discusses tests that automatically check the outcome though, as the UnitTest frameworks are doing. The advantage there is that you don't have to hand-verify the test output, and that you can test often (so you'll see quickly when something goes wrong).
Umm, I'd have to go back and look, but I think he specifies automatic failure detection, modeled on the "power on self test" of IC's. The way I did "design for testing" with C++ was to create/capture desired output, put it under version control, and then use the diff command line tool to automate, as part of a "make test" target. New tests required appending new output. A typical module would consist of four elements: * Foo.h # interface * Foo.cpp # implementation * Foo.test.cpp # Test driver main() * Foo.req # required output The "make test" sub-target for Foo looks like (tested):: test: Foo.pass clean: rm -f *.o *.test *.pass *.errors Foo.o : Foo.h Foo.cpp ${CC} -c Foo.cpp Foo.test.o : Foo.h Foo.test.cpp ${CC} -c Foo.test.cpp Foo.test : Foo.o Foo.test.o ${CC} -o $@ Foo.o Foo.test.o -lstdc++ Foo.pass : Foo.test Foo.req ./Foo.test | diff Foo.req - 2>&1 > Foo.errors touch $@
Lakos's book is very insightful though. And then I discovered Python and since then most of my hacking in C++ has been on a hold..
In the Zope world, I think this has to translate to somthing like:
1. Each Python module should have a set of associated tests, which would by preference live inside the "if __name__ == '__main__'" block at the bottom of the module (an alternative would be to use Tim Peters' DocTest stuff, which embeds the unit tests in docstrings). One would then have a script, like the one which compiles all the Python modules during installation, to run these modules (perhaps using some fancy dependency analysis based on the import statements?)
How would this work for things which use the Zope framework heavily, though? It's perfectly possible to do associate somekind of unit testing with Python modules, but the integration with the Zope framework seems to require having to use the framework in order to do the tests. That could slow the tests down a lot or make them too hard to perform, and the philosophy of these tests is that they should run quickly and should be easy to run, so you'll run them often.
Here is how I am approaching this with a small Product I am working on now: * The "core" business logic is Zope-agnostic (doesn't import *anything* Zopish); I test these modules using PyUnit (using a shell script, for the nonce). * The __init__ module for the Product imports the "core" modules and derives "Zopish" subclasses from them, mixing in things like OFS.SimpleItem and ObjectManager. I don't test these classes at all, since they literally have no distinct methods (I think the only member is 'meta_type'). It also declares the factory methods and management interface, and registers the "Zopish" classes in its 'initialize()'. * An alternate is to have the __init__ module register the "core" clases as base classes; then create another "view" product which holds ZClasses derived from them (I am beginning to think that this is a higher-productivity technique, since tweaking the DTML is so much simpler). The ZClasses have to be tested via ZClient and diff, as in my original #2 below.
2. For testing ZClasses, I am afraid that some kind of ZClient-based test will be required. Each test would:
* Generate a suitably-unique folder name.
* Construct a folder of that name off the root of the ZODB.
Or perhaps in some special 'scratchpad area'.
That works too. The "JUnit" tests all try to guarantee that no one test run leaves anything lying around which might interfere with a subsequent test, which is why I was randomizing the folder name -- even if the cleanup step barfed, you would still never carry over artifacts. -- ========================================================= Tres Seaver tseaver@palladion.com 713-523-6582 Palladion Software http://www.palladion.com
participants (3)
-
Jonothan Farr -
Martijn Faassen -
Tres Seaver