[Grok-dev] grokproject configures z3c.testsetup instead of grok.testing

[Kevin Teague]

> grok.testing or z3c.testsetup?
> ------------------------------
>
> The main point behind having 'grok.testing.register_all_tests' was to
> have an easier testsetup environment available with stock Grok. Import
> grok and forget about complicately named package imports. I think this
> is still a valid consideration.

+0.5

I like the idea of aliases within the grok namespace to make imports
simpler. But conversely, using imports from the grok namespace tends
to
convey a deeper integration, it takes a newbie a while to realize,
"Oh, Grok is just recommending z3c.testsetup for testing, it's not
doing anything magical to z3c.testsetup to make it work."

> Setting up Layers
> -----------------
>
> Layers are a powerful tool of the Zope testing machinery. They were not
> very well supported with z3c.testsetup < 0.3. You could only define one
> ZCML file or ZCML based layer which was used only for functional tests,
> but all thereof. Furthermore for z3c.testsetup 'layer' alway meant
> 'ZCML-layer' that is setup using zope.app.testing.functional although
> there are also non-ZCML layers and ZCML layers that do not need a fully
> fledged ZODB running in background (okay, the latter is rare).
>

Right. I've been looking at the 0.2.1 source release and "layers"
seemed
confusing since it looked like could only have one layer. Actually
allowing for multiple layers in 0.3 makes a lot more sense.

What about setting up layers with a relational database or an LDAP
instance? A How-To explaining that would be very nice :)

(or maybe I will write one if I figure it out ...)

>
> This raises the question: do we really need a default layer? From
> looking through zope.app.testing I learned, that it can be quite
> cumbersome to look in different locations for 'ftesting.zcml' and the
> results might be unexpected. Sometimes it is like pure guessing.
>
> From what I learned from developer feedback, I'd now tend to opt for
> skipping the default layer and instead raise an error if one is needed
> but not supplied.
>

Doesn't z3c.testsetup.doctesting.FunctionalDocTestSetup already look
for
the packages ftesting.zcml file to use for a default layer? Or am I
misunderstanding how layers are setup?


> In `grokproject` we could provide a locally defined ZCML-layer (as we
> already do) which would be included in generated projects and (this time
> _explicitly_) be used by local (functional) doctests.
>
> I admit that this could mean more setup code but people could see how to
> setup their own layer from the example code created (and the code would
> be really used).
>

+1

Yes, I think it would make the test setup more obvious if the
boilerplate
generated by grokproject used an explicit style.

> And as we are changing everything we could also switch to the new
> testsetup syntax, so that you could setup a functional test like this::
>
>   My Sample Tests
>   ===============
>
>   :doctest:
>   :functional-zcml-layer: ftesting.zcml
>
>      >>> getRootFolder()
>      <zope.app.folder.folder.Folder object at 0x...>
>
> With this registration you could use Kevins ultra short setup code::
>
>    # tests.py
>    import grok
>    test_suite = grok.testing.register_all_tests('sampleapp')
>
> or you could setup your functional tests like this::
>
>   My Sample Tests
>   ===============
>
>   :doctest:
>   :layer: sampleapp.tests.FunctionalLayer
>   :setup: sampleapp.tests.setup
>   :teardown: sampleapp.tests.teardown
>
>      >>> getRootFolder()
>      <zope.app.folder.folder.Folder object at 0x...>
>
> which would require the complete layer setup beside the
> `FunctionalLayer` definition in tests.py::
>
>   # tests.py
>   import os.path
>   import grok
>   import sampleapp
>   from zope.app.testing.functional import (ZCMLLayer,
>                                            FunctionalTestSetup)
>
>   globs = z3c.testsetup.functional.doctesting.FunctionalDocTestSetup.globs
>
>   def setup(test):
>       FunctionalTestSetup().setUp()
>       test.globs = globs
>       return
>
>   def teardown(test):
>       return FunctionalTestSetup().tearDown()
>
>   ftesting_zcml = os.path.join(
>      os.path.dirname(sampleapp.__file__), 'ftesting.zcml')
>
>   FunctionalLayer = ZCMLLayer(ftesting_zcml, __name__,
>                               'FunctionalLayer',
>                               allow_teardown=True)
>
>   test_suite = grok.testing.register_all_tests('sampleapp')
>
> A bit too much for the beginning, if you ask me (and not really needed
> for the tests done). So I would opt for having a default tests.py like
> the short one above and maybe we should add a link to the (then
> finished) grok testing documentation for people that want to create more
> complicated stuff.
>
> What do you think?
>

Well, I've been trying to sort out all of the packages at play
(z3c.testsetup, zope.app.testing, zope.testing) and it's making my
head spin ...

... but I think what's confusing me is the Layers. The old-style
syntax:

   :test-layer: python
   :test-layer: unit
   :test-layer: functional

Is confusing because the difference between 'python' and 'unit' tests
wasn't the layer (both layers are empty), but how those tests are
collected.

The new syntax:

   :doctest:
   :functional-zcml-layer: ftesting.zcml

   :doctest:
   :layer: sampleapp.tests.FunctionalLayer
   :setup: sampleapp.tests.setup
   :teardown: sampleapp.tests.teardown

   :unittest:

Is a little simpler in that :unittest: and :doctest: are just
informing the
test getters to collect tests from that file. But the layers is still
confusing. I'd rather see the layer names used:

    :doctest:
    :layer: FunctionalLayer

    :doctest:
    :layer: MyCustomFunctionalLayerName

    :unittest:

Where if the layer is not specified, then one isn't used. But for
layer
setup: specifying the ZCML file, the setUp() and tearDown() methods,
etc.
should all happen in the Layer(s) definition. Then in the test cases
you
simply specify which layer it's part of ...

... but again, I'm not sure this is possible? And if it is, I think
it'd
take quite a bit of refactoring in z3c.testsetup.

But it seems to make sense to me: a layer defines a complete set of
configuration for setup/teardown.

The confusing bit is that the
z3c.testing.testgetter.FunctionalDocTestGetter
uses the z3c.testing.doctesting.FunctionalDocTestSetup. This setup
then
defines it's own layer (with defaults for
layer_name='FunctionalLayer',
zcml_config='ftesting.zcml'). The ZCML Layer then has a setup with
configures
the FunctionalTestSetup. But FunctionalDocTestSetup doesn't delegate
to
the ZCMLLayer setup, but instead specifies in it's own setUp() to use
FunctionalTestSetup. Whew!

Purely in pseudo-code, this could support something like this in
a Grok-based package's test.py module:

import grok
import z3c.testsetup.layers.ZCMLAndZODBDemoStorage # this does not
exist!

class SampleAppFunctionalLayer
(z3c.testsetup.layers.ZCMLAndZODBDemoStorage):
    layer_name = 'Sample App Functional Layer'

    def setUp(self):
        # configures ZCML + ZODB DemoStorage
        super(SampleAppFunctionalLayer, self).setUp()
        # additional setup goes here
        pass

    def tearDown(self):
        # tears down ZODB DemoStorage
        super(SampleAppFunctionalLayer, self).tearDown()
        # additional tear down goes here
        pass

test_suite = grok.testing.register_all_tests(
    'sampleapp',
    layers = [SampleAppFunctionalLayer(allow_teardown=True), ]
)