[Grok-dev] Re: where do I put doctests for my models?

[Martijn Faassen]

Brandon Craig Rhodes wrote:
> Brandon Craig Rhodes <brandon at rhodesmill.org> writes:
> 
>> I have written a Grok Model and now need to know where to put its
>> doctest so that it gets run when I type "./bin/test". ...
> 
> I have done some more work on my question; PvW's book was helpful.
> The answer to my question seems to be that Grok does not currently
> give any help to the developer who wants testing - that there exists
> almost no "convention", but almost entirely "configuration".

Yes, this is unfortunately the case. It'd be nice if we could have some 
useful conventions to help finding tests, instead of this annoying 
test_suite() thing one has to do all the times. Of course there are also 
different classes of tests: unit tests (doctest version and unit test 
version) which promise to do their own setup before testing, and 
functional tests (FunctionalDocTest and unit test, and browser 
functional doctests) so our automation will have to be a bit intelligent.

> Could I submit a tutorial showing the current state-of-the-art in
> building tests for a Grok app, so that other beginners have somewhere
> to start from?

Yes, please. I have been chatting with Luciano a bit about this so he 
may give you some sample code. Let me paste some simple sample code in 
here too:

For unit tests:

import unittest

from zope.testing import doctest

def test_suite():
     optionflags = (
         doctest.ELLIPSIS
         | doctest.REPORT_NDIFF
         | doctest.NORMALIZE_WHITESPACE
         )

     return unittest.TestSuite([
         doctest.DocFileSuite(
             'README.txt', optionflags=optionflags)
         ])


For functional doctests (with testbrowser available):

import unittest

import zope.testbrowser.browser
import zope.testbrowser.testing

from zope.app.testing.functional import FunctionalDocFileSuite

def test_suite():
     globs = {}
     globs['Browser'] = zope.testbrowser.testing.Browser

     interaction = FunctionalDocFileSuite(
         'integration.txt',
         globs = globs,
         )
     return unittest.TestSuite([interaction])


> To get a doctest to run in my Grok buildout when I type "./bin/test",
> the following seems necessary:
> 
>  - I must create a directory called "tests" in my app.
>  - I must create an __init__.py in that directory.

You can skip this step if you are going to use doctests (like above) - 
you can create a 'tests.py' and a 'ftests.py'.

>  - I must create files with names like test*.py in the directory.

>  - Those files must have functions or classes in them named test_*.

I don't think this bit is necessary. You need to register the unit test 
classes, or the doctests, inside of test_suite(). I think the magic is 
naming it test_suite() so the test runner can find it.

>  - Those functions or classes must "from doctest import DocFileSuite"
>    and then list, filename by filename, all of my doctest files; or
>    I have to write my own recursive-descent routine (the grok tests
>    themselves seem to do this) to search for tests in my app.

No, no, you use the test runner for this (bin/test). If you register 
them in your test_suite() it should work (as above).

Here's how you create a test runner for your app in buildout.cfg:

[test]
recipe = zc.recipe.testrunner
eggs = myegg
defaults = ['--tests-pattern', '^f?tests$', '-v']

You need to place 'test' in the parts listing in the [buildout] section.

> I humbly suggest that all of the above is a massive example of
> configuration over convention - the programmer has to perform about
> six steps, each of which must be done perfectly correctly, before the
> "./bin/test" command becomes willing to run his tests.

While it's not as bad as you sketch out, you're entirely right that it's 
too much boilerplate to get going. I hope we can find ways to eliminate 
the test_suite() boilerplace and have some grok base classes. 
Unfortunately it probably means we will need to adjust Zope's test 
collector to build in support for this, as we do want to use the rest of 
that infrastructure.

> Would it be possible for the "test" command in a Grok buildout to
> automatically search for tests, instead of making the developer write
> his own "test_app.py" module?  I would love to be able to drop dozens
> of ".doctest" files in my "tests" directory inside my app, and have
> them just magically get run when I type "./bin/test".

I think fully automated searching for tests won't work, in that we have 
a bunch of different kind of tests. I was thinking about doing something 
like this:

class MyTest(grok.testing.FunctionalDoctest):
     grok.testing.file('README.txt')

replacing the test_suite() bits.

> I am willing to help code this, but am not sure what should change -
> the code of "./bin/test" itself, or something else?

It's going to be the zope.testing package that we need to investigate. 
It's likely we need to adjust the test collection phase to provide a 
hook-point for Grok, so martian can be used to collect the tests.

You can get this here:

svn co svn+ssh://svn.zope.org/repos/main/zope.testing/trunk/

Thanks very much for looking into this! This is one of the things that 
has been bugging me for a long time about testing with Grok.

Regards,

Martijn