[Zope-dev] Zope test layers, pytest, and test isolation

[Uli Fouquet]

Hi there,

Some of you might have noticed that some time ago the first version of
`zope.pytest` was released:

  http://pypi.python.org/pypi/zope.pytest

It's a try to make usage of `pytest` more comfortable in Zope-based
environments. `zope.pytest` is mainly based on Martijns, Jan-Wijbrands,
and Christian Klingers efforts. I put in some bits too.

Right now we have a problem with pytest integration when it comes to ZCA
setups: among other things `zope.pytest` offers a pytest-funcarg__
compatible function to automate ZCML based fixtures, i.e. it
automatically adds setup and teardown functionality before/after test
functions if requested by the test writer like this::

  import my.project
  from zope.pytest import configure
  from zope.component import queryUtility

  def pytest_funcarg__config(request):
      return configure(request, my.project, 'ftesting.zcml')

  def test_myutil_registered(config):
      util = queryUtility(
          my.project.interfaces.ISomeIface,
          name = 'myutil',
          default = None)
      assert util is not None

Here for test functions requiring an argument named ``config`` it is
guaranteed that before the test function is run a ZCA initialization is
performed based on the ZCML configuration given in the ``ftesting.zcml``
of `my.project`. This configuration is cached for the whole test session
(the complete test run including all tests).

If we assume that the used `ftesting.zcml` provides a named utility
named ``myutil`` then the call to ``configure`` in the
funcarg__-function should perform that registration and the
'myutil_registered' test should succeed.

All that works very well -- in principle. The problem is: the
`configure` function calls the registry setup before any real test is
run and performs a teardown only after the last of all tests was run.
And that leads to side-effects.

In other words: all tests share the same global ZCA registrations and
changes to the registrations in one test will affect other tests run
thereafter. We have a lack of test isolation.

Example::

  import my.project
  from zope.pytest import configure
  from zope.component import queryUtility, provideUtility
  from zope.interface import Interface

  def pytest_funcarg__config(request):
      return configure(request, my.project, 'ftesting.zcml')

  def test_1(config):
      util = object()
      provideUtility(util, provides=Interface, name='a_util')

  def test_2(config):
      util = queryUtility(Interface, name='a_util', default=None)
      assert util is None

  def test_3():
      util = queryUtility(Interface, name='a_util', default=None)
      assert util is None


Here the second test will fail. Even worse, also the third test will
fail, which does not require a ZCA setup at all.

That's because the global registry is setup before all tests and not
torn down before the last test was run.

This behaviour is partly intentional: we do not want to setup/teardown
the ZCA before/after single test functions as this would increase time
consumed by test runs dramatically. Instead we want to reduce the number
of ZCA setups/teardown as far as possible.

We _could_ do a per testfunction setup/teardown by simply using a
different test scope in `configure`. `pytest` offers three kinds of
scope for that purpose: `session`, `module`, and `function`. `session`
is the current default. Using `function` as scope we would get complete
test isolation but pretty long lasting test runs. That could stop people
from writing tests. Something we would like to avoid by all means.

We could instead use `module` as default scope, so that all test
functions inside a module would share one ZCA configuration (and for
different test modules the setup would be performed from scratch). This
might be a reasonable compromise. Test isolation of single tests inside
a test module would of course be broken but one could say: if you want a
fresh registry, just put your tests inside a new test module.

With Zope test layers there might be a similar problem (each layer can
create a global ZCA configuration that then will be shared amongst all
applied test methods) but I think people are used to it and can cope
with that specific test isolation breakage.

A big advantage of test layers over `pytest` testing scopes might be
that you can spread your tests associated to a certain layer over many
files/modules/packages as you like and the setup/teardown will
nevertheless only be performed once for each layer (well, normally at
least). If you have one layer and that's enough for you, you will only
have one ZCA-setup/teardown in the whole testrun. That's fast and
certainly fits many real-world use-cases.

Compared to Zope test layers I came to the conclusion that there is not
much like this concept already in `pytest` and the behaviour of test
layers can not easily be faked. `pytest` provides only the three
mentioned scopes as kind of 'natural' layers. Spreading fixtures over
many modules (as layers easily do) might contradict with the basic
design goals of pytest and I am pretty sure that Holge Krekel wouldn't
like it.

Overall, we're now looking for a satisfying solution in terms of runtime
and isolation. Some of the questions that arise:

Would it make sense to bring the Zope layer concept into `pytest`?

Are there already possibilities to mimic testlayer-like behaviour in
`pytest` which we simply overlooked?

Are there cheap/fast ways to cache/restore registry setups we hadn't had
on the screen? Really fast setups/cache-restores could make even
function-wise registrations a considerable thing.

Would it simply be okay to use the 'module' scope for registration
setups?

Or do you have completely different ideas how to solve that issue?

Any comments are really appreciated!

Best regards,

-- 
Uli

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Dies ist ein digital signierter Nachrichtenteil
Url : http://mail.zope.org/pipermail/zope-dev/attachments/20110324/9b01cad9/attachment.bin