[Grok-dev] Re: RFC: Grokking tests

[Philipp von Weitershausen]

Martijn Faassen wrote:
> * I'm trying to understand how tests are being grokked. Basically you've 
> automated the old way we collected tests by using grok's module grokker. 
>  Our testing strategy is Grok is very special as we have modules that 
> have test code as well as code to grok. This therefore looks more like 
> internal code to help grok self-test than code that is also applicable 
> to the outside world. Can you explain how external users use this? Or 
> was the goal to automate grok's internal testing strategy?

Hmm, you're right. I was trying to write tests for megrok.five which are 
a lot like Grok's tests. I suppose I was indeed tryinig to automate 
Grok's internal testing. Perhaps this is overkill for that...

> * Instead I'd expect people saying stuff like this:
> 
> class MyTest(grok.Doctest):
>    grok.file('mydoctests.txt')
> 
>    .. support for setUp, tearDown and globals here ... :)
> 
> and this being picked up by the grokker. That's an extra place to 
> register the tests for grokking, true, but it would replace the need for 
> test_suite. unittest.TestCase subclasses could be grokked straight away, 
> even. Instead, you still require people to write a test_suite() function 
> and call grok_tests() manually. It'd be nice if we could do away with 
> this entirely. I realize that makes it harder to use the test runner's 
> existing scanning procedure.

Yeah, I hadn't even thought further at this point, I was just trying to 
solve my problem. As far as those test classes go, how would you tell 
the testrunner to find them?

> * I feel uncomfortable that the grokker is just trying to make a module 
> into a doctest and registers them whenever possible.

Hmm, yeah.

> * is the grok.doctest directive being used anywhere? I am grepping but 
> can't seem to find any case where it's being used. What am I missing?

No, that's the unfinished bit. I need to write tests for it.

> Anyway, the fundamental issue I have is that this isn't what I had in 
> mind when the word 'test grokker' was mentioned. I was imagining a 
> system as sketched out above, where you have a grok.Doctest base class 
> to register file-based doctests. We can pick up unittest.TestCase 
> automatically. This way people can just focus on writing tests and grok 
> will pick them up with minimum hassle. Maybe we still require people put 
> them in modules or packages 'tests' and 'ftests' as Grok's convention 
> (and Zope 3's, in fact).

(that convention is going away, btw)

Yeah, I was perhaps giving out false hope when I said I worked on 
grokking tests... :/

> Instead grok_tests is now a scanning procedure that can be used to scan 
> a package for grok-style tests. This cuts away some code, automates a 
> messy piece of logic we had for test registration, and reuses some more 
> of grok internally.
> 
> I think it's still worthwhile to do this, but I'd like to be very 
> careful we don't publish this anywhere as the way grok tells you to test 
> your code. I'm not sure therefore whether this should be called 
> 'grok.testing', as this implies reusable code.

I agree, though grok.doctest() is supposed to be reusable (it lets you 
set doctest parameter for the docstrings in the current module).

> Of course whether we can do something I sketched out depends on whether 
> we can convince Zope's testing infrastructure to actually find these 
> things as tests without us having to write anything called test_suite() 
> anywhere. In fact it's kind of funny, as Zope 3's test collector comes 
> closest in activity to what grok does already. :)

Heh, yeah, I thought the same thing when I wrote this. zope.testing 
already searches the stuff, why can't it simply find this automatically...

Perhaps we can indeed hijack the test runner, though I'm not sure how 
it's supposed to find our tests w/o a test_suite(). I think we need at 
least one test_suite() to start the grokking (like we do now).


-- 
http://worldcookery.com -- Professional Zope documentation and training