[Grok-dev] Re: How to hook up custom layers with z3c.testsetup?

[Uli Fouquet]

Hi there,

Philipp von Weitershausen wrote:
> Uli Fouquet wrote:
> > 
> > The marker 'unit|functional|python' therefore not only decides about
> > applying or not applying a layer, but also about different test loaders,
> > different setUp/tearDown-methods, etc.
> 
> So tests of type 'unit' will get an automatic setUp/tearDown? What do 
> these do? Do they call zope.testing.cleanup.cleanUp()?

That's exactly what they do currently.

> > Anyway, when your testing setup
> > becomes too complex, you're certainly better off setting up the tests
> > yourself and in the good old-fashioned way.
> 
> I must strongly disagree here, because we "trick" people into thinking 
> that test setup is very easy. Then they follow down the road that we 
> paved with z3c.testsetup, only to tell them when it outgrows their 
> needs, they need to do things *completely differently*. I would rather 
> not break the linearity of the learning curve there. In fact, Grok is 
> all about making the learning curve linear...

Hm, difficult to disagree here :-) In the beginning z3c.testsetup was
planned for simple setups only. I didn't realize that it might be usable
for very complex setups too. But overall it should.

> I agree that defaults are a good idea and I'm not proposing to change 
> much with the defaults. I'm proposing to go further than what it can 
> already do, so we won't break people's assumptions (in this case Ivo's 
> rather valid assumption that integration tests can't be too different 
> from functional tests).

Fine. I'll be the last to oppose improvements :-)

> The default behaviour of the zope.app.testing package was to look in 
> INSTANCE/etc/ftesting.zcml. This was so utterly problematic that nobody 
> uses it anymore. If you look at the code in zope.app.testing.functional, 
> you'll have concentrate very hard, not because it's complicated but to 
> prevent vomiting ;).

:-)

> We now explicitly define layers, take a look at virtually every Zope 
> package out there that has ftests, including Grok. I don't think we 
> should go back to making the same mistake again and implicitly load 
> files based on naming conventions and magic places. After all, there 
> *is* a FunctionalLayer definition in a standard grokproject. Isn't it 
> used?

Right, it is not used by default and I agree that this is bad. Before
releasing the new grokproject, that should be fixed somehow, for example
by really using it.

>  I think it should be used and simply mapped to the 'functional' 
> test type. Or better, the layer should simply be referred to from the 
> test, like I suggested::
> 
>    :Test-Layer: myapp.tests.FunctionalLayer

The first approach, mapping myapp.tests.FunctionalLayer, would IMHO only
shift the problem of automatically finding ftesting.zcml to
automatically finding the FunctionalLayer definition. Furthermore it
would continue the problem of different layers for different test files.

The second approach, referring to the layer definition from the test
file, sounds good to me. There is, however, one problem with it (see
below).

> >> I think it could be improved in a very simple way: In doctests, you  
> >> simply refer to the dotted name of the layer, e.g.:
> >>
> >>    :Test-Layer: myapp.tests.IntegrationLayer
> >>
> >> z3c.testsetup just finds the doctests, aggregates them, applies the  
> >> layer they specify and returns them to the test runner. By default,  
> >> there'd be a FunctionalLayer in grokproject (like there's already)  
> >> which is being referred to from the minimal test in app.py.
> >>
> >> WIth this, people can easily create new layers (just create another  
> >> ZCML file, another ZCMLLayer definition in tests.py) and easily hook  
> >> them up to their doctests.
> >>
> >> What do you think?
> > 
> > Nice idea!
> > 
> > I agree with the idea in general and personally do not like the tag name
> > 'Test-Layer' to declare a test _type_ at all. 
> > 
> > The problem with it is, that the name is now in use and also its
> > semantics. We would therefore run some people into terrible trouble when
> > we change it. What we need here might be a 'migration' policy. What
> > about the following?
> > 
> > In the beginning let the test types be declared also by another keyword 
> > 
> >   - :Test-Type: unit|functional|python
> 
> Why do we still need this? If we support arbitrary layers, these should 
> become superfluous. In fact, following the rule that there should be 
> only way to do things, this should go away.
> 
> The 'fucntional' type would be absorbed by using the FunctionalLayer 
> definition, 'unit' would be a special layer that has the setup and 
> teardown that you mention above and 'python' would be, um, nothing, I 
> suppose?

Right, 'python' means no setup for this file. But it means also: load
tests from this file.

Just to be sure: if I understand you correctly, you would propose setups
like this::

  # tests.py
  import myapp
  from zope.app.testing.functional import ZCMLLayer
  from zope.testing import cleanup
  from grok.testing import register_all_tests

  class UnitDocTestLayer(object):
     def setUp(self, *args, **kw):
       pass
     def tearDown(self, *args, **kw):
       cleanup.cleanUp()

  ftesting_zcml = os.path.join(
    os.path.dirname(myapp.__file__), 'ftesting.zcml')

  FunctionalLayer = ZCMLLayer(ftesting_zcml, __name__, 'FunctionalLayer')
  UnitLayer = UnitDocTestLayer()
  
  test_suite = register_all_tests('myapp')
  # EOF

where the single doctest files refer to the above defined layers:

  # simple_unitdoctest.txt
  :Test-Layer: myapp.tests.UnitLayer

and::

  # functional_doctest.txt
  :Test-Layer: myapp.tests.FunctionalLayer
    >>> getRootFolder()
    <...>

To be honest, such a tests.py as above scares me a bit, although it
might be autogenerated by grokproject.

Another problem we run into here is: the marker not only says what layer
to use, but it also says that the current file is a test file at all. It
separates test files from non-test files.

For 'python' tests there would be no layer and so there would be no
marker. We could possibly provide a dummy-layer for that purpose, but
that might confuse people and sounds wrong to me. 

The questions then remains, how to mark 'python' tests. They can be
found by the testrunner, of course, based on the filename. But this
would mean to tell people: if you have this kind of test, mark it by one
of those strings. If you have another kind of test, then use one of the
following filenames. One time you mark it by a certain string in the
test file, another time by configuring the testrunner. Having all kinds
of tests marked in a similar manner is nice IMO and (beside the layer
problem) easy to understand.

> > The keyword :Test-Layer: might be supported for a while, but should
> > issue deprecation warnings when used.
> > 
> > For the real layer setup I could imagine some (optional) additional
> > markers (declarations), like::
> > 
> >   - :Test-Layername: NameOfLayer
> > 
> >   - :Test-LayerZCML: filename.zcml  (looked up in package root)
> > 
> >   - :Test-Layerdef: dotted.name.of.definition
> 
> Too much choice for my taste. I think we really only need the last one. 

Agreed. Less might be more here.

> A ZCML file can be turned into a layer with a one-liner in tests.py. And 
> thanks to grokproject's default FunctionalLayer, people can easily see 
> how to do this. Ivo figured *that* out very easily, and if not, he 
> could've always taken my book where exactly this is documented, or taken 
> any other Zope package. I don't think we should reinvent well-documented 
> conventions from Zope.

That's true for grokprojects. But what about other packages written from
scratch? I really don't like to break well-documented and introduced
practices but if people are not forced to setup their own layers (while
they still have the possibility to do so, if they wish), this might be
nice. Beside this it is clear to me, that Ivos problem came from a
crappy layer handling in the current package.

> The only really difficult problem for Ivo was hooking up his layer to 
> the test. Hence my suggestion to simply specify the dotted layer name in 
> the test file.

I agree completely. Layers should be declarable in the test files.
Because of the above mentioned problem I only would propose to mark that
layer declaration with a different tag and possibly change the name of
the marker strings.

> > So we could move the layer-related parameters step by step from the test
> > setup file to the real test files, where they might belong. After some
> > releases then 'Test-Layer' could become a synonym for 'Test-Layerdef'
> > and the latter might vanish completely some day.
> 
> Well, following my urge for simplification, I suggest we allow 
> Test-Layer to specify dotted names of layers, and for BBB, we still 
> allow 'unit', 'python' and 'functional' while phasing them out.

How should we distinguish those types of tests then? I might have missed
an important point here. If I understood you right, there is no need for
such differentiation, but it is not enirely clear to me, why this is the
case. Note, that currently tests of the different types are setup using
different methods:

 'unit' leads to a call to zope.testing.doctest.DocFileSuite

 'functional' leads to a call to 
       zope.app.testing.functional.FunctionalDocFileSuite

 'python' leads to a call to unittest.defaultTestLoader

The marker string therefore cares also for different test loaders (which
might be done better). Furthermore it separates testfiles from
non-testfiles. I do not see, how all this could be handled with a layer
definition.

Beside this the :Test-Layerdef: marker seems to be a good idea anyway,
so I will start implementing this :-)

Oh, and thanks for all your valuable input! That helped me much to
understand the layer concept better :-)

Best regards,

-- 
Uli

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 191 bytes
Desc: Dies ist ein digital signierter Nachrichtenteil
Url : http://mail.zope.org/pipermail/grok-dev/attachments/20080624/17109aa3/attachment.bin