[Grok-dev] grok.testing

[Uli Fouquet]

Dear grokkers, 

in 
	svn://svn.zope.org/repos/main/Sandbox/ulif/grok-with-testing/

there is a first, very plain and limited implementation of a Grok with
testing support as it was suggested by Martijn.

In the 'admin' subdirectory you can find an example of how it works yet.

How it works:
-------------

The basic procedure to register tests with it is as follows:

In your Grok application/component you define testlocations like this::

  import grok
  class SillyTest(grok.testing.FunctionalDocTest):
    grok.testing.file('README.txt')

The ``grok.testing.file`` directive is a must currently, for it is the
only way to register your testfile (currently).

Furthermore, you need a file ``ftests.py`` in your application root to
let the testrunner find your registered tests. It might look like this::

  import grok
  import unittest

  def test_suite():
      import grok.admin
      # We must grok our package to get the test registrations...
      grok.grok('grok.admin')
      return grok.testing.test_suite()


  if __name__ == '__main__':
      unittest.main(defaultTest='test_suite')

Now you should be able to fill ``README.txt`` with doctests as you like,
run bin/test and get the results. Non-existant files raise a GrokError
during booting.


Remarks:
--------

The ``ftests.py`` is currently needed to hook in the tests registered in
the package into the testrunner. Didn't find a smart way yet to do it in
another way, but after all the size of code needed to get doctests
running this way can be shrinked considerably.

FunctionalDocTests allow flexible test setup for special purposes. For
example, by default all tests registered with grok.testing classes (yes,
currently there is only one testing class: FunctionalDocTest) are run in
a separate layer, which is created in the base classes' ``setUp()``
method. You can easily overwrite such things in your own implementation
of ``FunctionalDocTest``, as long as you provide a ``test_suite()``
method.

To sum it up, using grok.testing you get some cheap automatic testing
without loosing the flexibility to do quite exotic testing
scenarios. :-)

Limitations:
------------

 - ftests.py is required. Otherwise the tests registered are not 
   executed.
 - Currently only functional doctests are supported. No plain unit 
   testing!
 - Only doctests in separate files (using the grok.testing.file 
   directive) are yet supported (not a big thing to change that).
 - Didn't test, how doctests in subdirectories can be declared.
 - No tests of testing yet. I would appreciate any hints how testing of 
   testing is done best.

Open Questions:
---------------

Beside the fact, that the quality of implementation is most probably
poor, there are now some options:

 - should there be (by convention) a default test/ftest directory 
   (like 'static/') where testing looks for tests (iff it exists)?

 - should a conventional testfinder mechanism be enabled? This means:

   If in a module you got a class SampleApp(grok.Application) and 
   another class implementing a FunctionalDocTest, which is called 
   ``SampleAppTest``,  should it automatically parse SampleApp for 
   docstrings with tests?

   You then could write::

     import grok
     class SampleApp(grok.Application):
       """
         >>> 1==1
         True
       """
       pass

     class SampleAppTest(grok.testing.FunctionalDocTest):
       pass

     and the doctest in SampleApp would be executed.

Well, there are tons of other things to consider, but I just wanted to
keep you up-to-date in the ongoing 'how-should-we-handle-testing'
debate.

Suggestions, comments, etc. are as usually very welcome.

Cheers,

-- 
Uli