[Grok-dev] z3c.testsetup and layers

[Jan-Wijbrand Kolman]

Hi,

I'm trying to grasp testing layers in general and functional testing
layers in particular - all from a z3c.testsetup perspective. Since it is
not all that clear in my head still, let met try to write things down
for others to comment on.

I also have some questions and suggestions for z3c.testsetup in particular.


The summary of test layers:
===========================

* Test layers is a feature of zope.testing.

* Test layers can be used to group tests that have a shared test setup
and teardown (the setup and teardown happen once for all the tests in 
grouped in that layer).

* Test layers are classes and thus test layers can inherit from each
other.

* The zope.testing.testrunner will make sure that setup functions of
more generic layers are called before that of the specific layers and
the other way around for the teardown functions.

* The setup and teardown functions are to be defined as classmethods(!) 
on the test layers.

* See also:

http://apidoc.zope.org/++apidoc++/Code/zope/testing/testrunner-layers-api.txt/index.html


So far so good.


The summary of functional test layers:
======================================

* Functional test layers is a feature of zope.app.testing (notice the 
.app in there!).

* Test layers can be used to group tests that have a shared test setup
and teardown in the context of a shared component registry that has been 
initialized according to a *.zcml file that can be specified for 
functional test layers (the setup and teardown and the component 
registry initialization happen once for all the tests in grouped in that 
layer).

* Functional test layers are classes, however, for some reason (probably 
historical), they're still old style classes.

* Still, since they're classes functional test layers can inherit from 
each other.

* The setup and teardown methods are just that, methods (not class 
methods!).

* The testrunner __cannot__ on its own make sure that the setup methods 
of the more generic layers are called before that to the specific 
layers, nor the other way around for the teardown methods.

* You have to make sure all setups and teardown methods in the test 
layer inheritance hierarchy are called in the correct order::

   class MyFunctionalLayer(zope.app.testing.functional.ZCMLLayer):

       def setUp(self):
           super(MyFunctionalLayer, self).setUp()
           # and now my stuff

       def tearDown(self):
           # first my stuff
           super(MyFunctionalLayer, self).tearDown()

* This difference in layer behaviour is annoyingly confusing.


So far so good (albeit a bit annoying).


The summary of using (functional)test layers in z3c.testsetup:
==============================================================

There're three "directives" that z3c.testsetup recognizes for setting up 
test layers:

1) :layer: <dotted.name.of.layer.def>

2) :zcml-layer: <ZCML_filename>

3) :functional-zcml-layer: <ZCML_filename>

Note that these layer-ish directives __only__ work in docttests!


Ad 1):
------

This is the most "generically" useful directive. By specifying a dotted 
name to the layer, the test is grouped in that layer.

It can point to either a "normal" layer class (i.e. of the "non 
functional" kind). Alternatively it can point to a layer instance if it 
is to be a functional layer. Compare:

layers.py:

   class MyLayer(object):
     pass


   class MyFunctionalLayer(ZCMLLayer):
     pass

   my_functional_layer = MyFunctionalLayer(
     '/path/to/testing.zcml', __name__, 'My Functional Layer')


test_non_functional.txt:

   :layer: layers.MyLayer


test_functional.txt:

   :layer: layers.my_functional_layer

Note that if the layer definition is an instance of ZCMLLayer (directly 
or indirectly), the tests are suited up in a 
zope.app.testing.functional.FunctioalDocFileSuite

Ad 2 and 3):
------------

The functionality of both :zcml-layer: and :functional-zcml-layer: can 
be accomplished by using :layer:

However, in case you just want to use a common *.zcml file for a group 
of tests and not need to provide a common setup and teardown you can use 
these directives.

The directives' differences are subtle - and not yet completely clear to 
me. In case of :functional-zmcl-layer:

* There're more convenience functions available in the test's globals. 
(what extra functions are these?)

* Somehow the setUp and tearDown functions (what setUp and tearDown 
functions?) are wrapped in something (in what and why?).

* Anything more?


So far so... well... alright. The layer-ish directives were a source of 
confusion for me and maybe for others too. I have couple of suggestions:


Suggestions
===========

* The :zcml-layer: and :functional-zcml-layer: directives have a 
slightly confusing name. They really only point to a zcml file, not to a 
layer definition like :layer: does.

I can imagine that :zcml-file: and :functional-zcml-file: would've been 
less confusing names.

* The difference between :functional-zcml-layer: and :zcml-layer: is 
really not all that obvious (at least to me). I wonder if we cannot just 
have just one, with the behaviour of :functional-zcml-layer:?

Alternatively, the z3c.testsetup documentation could provides some 
directions for choosing one over the other.

* z3c.testsetup should support the :layer: directives in unittests too.

* The functional layers and "normal" layers should probably behave 
similar. This is not something z3c.testsetup can fix. I'll post a 
message about this on zope-dev.


I hope my summary is correct and the suggestions make sense. I hope it 
is somewhat complete as well as useful for others. I'd really like to 
get feedback though.


regards,
jw