[Grok-dev] Re: RFC: Grokking tests

[Martijn Faassen]

Hey,

I've dug through the code a bit more, and I have some comments:

* I'm trying to understand how tests are being grokked. Basically you've 
automated the old way we collected tests by using grok's module grokker. 
  Our testing strategy is Grok is very special as we have modules that 
have test code as well as code to grok. This therefore looks more like 
internal code to help grok self-test than code that is also applicable 
to the outside world. Can you explain how external users use this? Or 
was the goal to automate grok's internal testing strategy?

* Instead I'd expect people saying stuff like this:

class MyTest(grok.Doctest):
    grok.file('mydoctests.txt')

    .. support for setUp, tearDown and globals here ... :)

and this being picked up by the grokker. That's an extra place to 
register the tests for grokking, true, but it would replace the need for 
test_suite. unittest.TestCase subclasses could be grokked straight away, 
even. Instead, you still require people to write a test_suite() function 
and call grok_tests() manually. It'd be nice if we could do away with 
this entirely. I realize that makes it harder to use the test runner's 
existing scanning procedure.

* I feel uncomfortable that the grokker is just trying to make a module 
into a doctest and registers them whenever possible.

* we need to refactor things so we can have several, separate grokking 
phases. I have the impression testing.py right now has to do too much. 
(grok_tree in particular)

* is the grok.doctest directive being used anywhere? I am grepping but 
can't seem to find any case where it's being used. What am I missing?

Anyway, the fundamental issue I have is that this isn't what I had in 
mind when the word 'test grokker' was mentioned. I was imagining a 
system as sketched out above, where you have a grok.Doctest base class 
to register file-based doctests. We can pick up unittest.TestCase 
automatically. This way people can just focus on writing tests and grok 
will pick them up with minimum hassle. Maybe we still require people put 
them in modules or packages 'tests' and 'ftests' as Grok's convention 
(and Zope 3's, in fact).

Instead grok_tests is now a scanning procedure that can be used to scan 
a package for grok-style tests. This cuts away some code, automates a 
messy piece of logic we had for test registration, and reuses some more 
of grok internally.

I think it's still worthwhile to do this, but I'd like to be very 
careful we don't publish this anywhere as the way grok tells you to test 
your code. I'm not sure therefore whether this should be called 
'grok.testing', as this implies reusable code.

Of course whether we can do something I sketched out depends on whether 
we can convince Zope's testing infrastructure to actually find these 
things as tests without us having to write anything called test_suite() 
anywhere. In fact it's kind of funny, as Zope 3's test collector comes 
closest in activity to what grok does already. :)

Regards,

Martijn