[Grok-dev] Re: How to hook up custom layers with z3c.testsetup?

[Philipp von Weitershausen]

Uli Fouquet wrote:
> I think you might be confused by the marker name 'Test-Layer'. I can
> understand that. It should better be named 'Test-Type' or similar.
> 
>> Also, I've not managed to make out the difference between the 'python'  
>> and the 'unit' setting for this parameter as described by the README.
> 
> There are currently three test types supported: (1) unit doctests
> ('unit'), (2) functional doctests ('functional') and (3) non-doctests
> ('python'). The latters are registered by `unittest.defaultTestLoader`
> and must take care themselves for setup, any functional layers etc. The
> z3c.testsetup package currently only cares for the layer setup of the
> second type and must confess, that I've never setup a layer for a normal
> unit test yet.
> 
> The marker 'unit|functional|python' therefore not only decides about
> applying or not applying a layer, but also about different test loaders,
> different setUp/tearDown-methods, etc.

So tests of type 'unit' will get an automatic setUp/tearDown? What do 
these do? Do they call zope.testing.cleanup.cleanUp()?

>> I read the docs but I was much confused about all the assumptions the  
>> package makes. I think even a medium-sized application will quickly  
>> outgrow these assumptions.
> 
> That might be right. For me, however, it became handy in several
> projects and I think for others too.

I never doubted that :).

> Anyway, when your testing setup
> becomes too complex, you're certainly better off setting up the tests
> yourself and in the good old-fashioned way.

I must strongly disagree here, because we "trick" people into thinking 
that test setup is very easy. Then they follow down the road that we 
paved with z3c.testsetup, only to tell them when it outgrows their 
needs, they need to do things *completely differently*. I would rather 
not break the linearity of the learning curve there. In fact, Grok is 
all about making the learning curve linear...

> The package is not a
> replacement for any other testing environment but only a 'frontend' that
> sets up some defaults. Lots of 'assumptions' (I'd call it 'defaults')
> therefore are needed. Without it the package would not make much sense.

I agree that defaults are a good idea and I'm not proposing to change 
much with the defaults. I'm proposing to go further than what it can 
already do, so we won't break people's assumptions (in this case Ivo's 
rather valid assumption that integration tests can't be too different 
from functional tests).

> For the layer stuff, however, I'd agree that this is not solved
> satisfactorily yet. It is quite difficult to setup different layers with
> the package. Your proposal below sounds good in that respect.
> 
>>  For instance, guessing that
>>
>>    :Test-Layer: functional
>>
>> *implicitly* loads ftesting.zcml didn't even cross my mind for a  
>> second, I thought it was somehow hooked up to FunctionalLayer.
> 
> Don't know what you mean here. I might be wrong but when setting up a
> functional doctest, there is always some ZCML file involved, isn't it?

Ys.

> The default behaviour of the zope.app.testing package, if I remember
> correctly, is to look for such a file somewhere in the depths of
> `parts/test/` or so.

The default behaviour of the zope.app.testing package was to look in 
INSTANCE/etc/ftesting.zcml. This was so utterly problematic that nobody 
uses it anymore. If you look at the code in zope.app.testing.functional, 
you'll have concentrate very hard, not because it's complicated but to 
prevent vomiting ;).

We now explicitly define layers, take a look at virtually every Zope 
package out there that has ftests, including Grok. I don't think we 
should go back to making the same mistake again and implicitly load 
files based on naming conventions and magic places. After all, there 
*is* a FunctionalLayer definition in a standard grokproject. Isn't it 
used? I think it should be used and simply mapped to the 'functional' 
test type. Or better, the layer should simply be referred to from the 
test, like I suggested::

   :Test-Layer: myapp.tests.FunctionalLayer

> Compared to this the new default is more reasonable, I think.
> 
>> I think it could be improved in a very simple way: In doctests, you  
>> simply refer to the dotted name of the layer, e.g.:
>>
>>    :Test-Layer: myapp.tests.IntegrationLayer
>>
>> z3c.testsetup just finds the doctests, aggregates them, applies the  
>> layer they specify and returns them to the test runner. By default,  
>> there'd be a FunctionalLayer in grokproject (like there's already)  
>> which is being referred to from the minimal test in app.py.
>>
>> WIth this, people can easily create new layers (just create another  
>> ZCML file, another ZCMLLayer definition in tests.py) and easily hook  
>> them up to their doctests.
>>
>> What do you think?
> 
> Nice idea!
> 
> I agree with the idea in general and personally do not like the tag name
> 'Test-Layer' to declare a test _type_ at all. 
> 
> The problem with it is, that the name is now in use and also its
> semantics. We would therefore run some people into terrible trouble when
> we change it. What we need here might be a 'migration' policy. What
> about the following?
> 
> In the beginning let the test types be declared also by another keyword 
> 
>   - :Test-Type: unit|functional|python

Why do we still need this? If we support arbitrary layers, these should 
become superfluous. In fact, following the rule that there should be 
only way to do things, this should go away.

The 'fucntional' type would be absorbed by using the FunctionalLayer 
definition, 'unit' would be a special layer that has the setup and 
teardown that you mention above and 'python' would be, um, nothing, I 
suppose?

> The keyword :Test-Layer: might be supported for a while, but should
> issue deprecation warnings when used.
> 
> For the real layer setup I could imagine some (optional) additional
> markers (declarations), like::
> 
>   - :Test-Layername: NameOfLayer
> 
>   - :Test-LayerZCML: filename.zcml  (looked up in package root)
> 
>   - :Test-Layerdef: dotted.name.of.definition

Too much choice for my taste. I think we really only need the last one. 
A ZCML file can be turned into a layer with a one-liner in tests.py. And 
thanks to grokproject's default FunctionalLayer, people can easily see 
how to do this. Ivo figured *that* out very easily, and if not, he 
could've always taken my book where exactly this is documented, or taken 
any other Zope package. I don't think we should reinvent well-documented 
conventions from Zope.

The only really difficult problem for Ivo was hooking up his layer to 
the test. Hence my suggestion to simply specify the dotted layer name in 
the test file.

> So we could move the layer-related parameters step by step from the test
> setup file to the real test files, where they might belong. After some
> releases then 'Test-Layer' could become a synonym for 'Test-Layerdef'
> and the latter might vanish completely some day.

Well, following my urge for simplification, I suggest we allow 
Test-Layer to specify dotted names of layers, and for BBB, we still 
allow 'unit', 'python' and 'functional' while phasing them out.