[Zope-dev] Test fixture concepts (was: Zope test layers, pytest, and test isolation)

[Uli Fouquet]

Hi there,

I first would like to thank everybody very much for the interesting and
elaborated answers to my last questions! I learned a lot about
registries and think that for now `plone.testing` (which in fact we
hadn't had on the screen) will help to solve some of the most urgent
issues we have with zope.pytest currently. So, thank you very much!

A sidenote: IMO it would be nice to have proper deletion of
registrations in the long run as well and I am willing to help as good
as I can with that, even if that would mean much work. The
'just-setup-your-layers-more-carefully'-solution is a bit tricky in
testing-frameworks without layers ;-)

Wolfgang Schnerring wrote:

[snip]

> > Are there already possibilities to mimic testlayer-like behaviour in
> > `pytest` which we simply overlooked?
> 
> An honest counter question: Why not use zope.testrunner? What
> advantages does py.test offer?

Due to my limited experience with py.test I'm certainly not in a
position to answer that comprehensively. I personally use
zope.testrunner and layers happily in most of my projects. zope.pytest
is currently certainly more a try to see what's possible with other
approaches than zope.testrunner and some features of py.test are really
promising IMO.

The (limited) experiences with py.test, however, were awesome. Some
points that are quite cool IMHO:

- Easy finding of tests: just write some ``test_function`` in a 
  ``test_module`` and it will be found and executed. That also makes
  py.test tests more readable and maybe more intuitive.

- Lots of setup code (unrelated to fixtures) can simply be skipped. No
  need to do the ``testsuite = <complex-testcase-collecting>`` over and
  over again. Maybe the main point of py.test.

- py.test is more widespread in the Python community (that's my 
  impression; I can't proof it)

- Support of unittest/unittest2: you can write standard lib setups 
  (defining TestCases; no need to also write testsuite-setup stuff) and
  they will be found/executed. zope.testrunner for instance does not 
  support the new `setUpClass`/`tearDownClass` concept of unittest2 
  (yes, you would use layers in that case; but it might be nice if 
  zope.testrunner  would support also class-wide fixtures in 
  unittest2-style; people from other worlds might expect that to work).

- `assert` works like you would expect it to work in tests. No need
  to use `self.assertEqual()`` and friends (but you can if you prefer).

- It was very easy to run py.test from the Sphinx doctest builder
  which made it possible to test documentation on a quite complex level.
  That might be possible with zope.testrunner as well (but I never tried
  that; wait - I tried once and it worked, but was more complex to 
  setup IIRC). I see that this is a very special usecase.

That said I'd like to stress that I would not say that py.test is
superior to zope.testrunner in general. I only enjoyed the first
experiences with py.test very much and could imagine that it would be a
pleasure for others too. Other things, however, meant less fun.

Main drawbacks I see on py.test side are:

- Lack of layer support (yet). Maybe we can do something about that in
  `zope.pytest` based on `plone.testing.layer`.

- Limited doctest support. It is quite difficult (AFAIK) to define 
  fixtures for doctests or to even set the usual doctest options 
  (``ELLIPSIS``, ``NORMALIZE_WHITESPACE``, ...) at setup time. Doctests 
  are simply collected and executed and not much finetuning is possible.

My very own concern about the latter point is: if this cannot be solved
satisfactorily (in terms of readability and ease of use at least),
py.test will not become a really hot candidate in the testing framework
game on the Zope side at all. There are too many doctests used already
and whether you like them or not: every testing framework used should
offer at least minimal support for doctest fixtures and doctest options.
But I might just have missed these features in py.test and they are
provided already. 

For now I think that there is absolutely no need to think about a
general move to py.test for the ztk.

I guess Martijn or others with more py.test experience could say more
concerning the pros and cons of py.test from daily work use.

[snip]

Best regards,

-- 
Uli

-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Dies ist ein digital signierter Nachrichtenteil
Url : http://mail.zope.org/pipermail/zope-dev/attachments/20110327/b856ac1d/attachment.bin