[Grok-dev] zope.testing autodiscovery (was: need advice on testing)

[Wolfgang Schnerring]

Hello,

I don't know if this is the right place to talk about this, but since the
conversation started here...

* Christian Theune <ct at gocept.com>:
> Better test discovery and getting rid of the need of test_suite() is another
> refactoring/feature I want to perform on zope.testing.

I'd definitely like to see this; having to explicitly list test cases is grating
on my nerves. I've tweaked find_suites() a little to handle the easy but common
case (which saves me an awful lot of un-DRY typing, nonentheless), namely
modules containing unittest.TestCases, see the attached patch.

Is this a step in the direction you want to go? If yes, what do I need to do so
this can be committed? I'd also be willing to work on something to discover
doctests from .txt files, but I don't have a clear picture what functionality is
wanted there. Thoughts?

Wolfgang

-------------- next part --------------
--- a/trunk/src/zope/testing/testrunner/find.py
+++ b/trunk/src/zope/testing/testrunner/find.py
@@ -88,7 +88,15 @@ def find_suites(options):
                         )
                 else:
                     try:
-                        suite = getattr(module, options.suite_name)()
+                        if hasattr(module, options.suite_name):
+                            suite = getattr(module, options.suite_name)()
+                        else:
+                            suite = unittest.defaultTestLoader.loadTestsFromModule(module)
+                            if suite.countTestCases() == 0:
+                                raise TypeError(
+                                    "Module %s does not define any tests"
+                                    % module_name)
+
                         if isinstance(suite, unittest.TestSuite):
                             check_suite(suite, module_name)
                         else:
--- /dev/null
+++ b/trunk/src/zope/testing/testrunner/testrunner-discovery.txt
@@ -0,0 +1,48 @@
+Automatically discovering tests
+===============================
+
+You can explicitly specify which tests to run by providing a unittest.TestSuite
+object (or a callable that returns one) in the test modules (the name of the
+object or function can be configured with the --suite-name parameter, it
+defaults to 'test_suite'). If no such object is present, testrunner will use
+all classes in the module that inherit unittest.TestCase as tests:
+
+    >>> import os, sys
+    >>> directory_with_tests = os.path.join(this_directory, 'testrunner-ex')
+
+    >>> from zope.testing import testrunner
+
+    >>> defaults = [
+    ...     '--path', directory_with_tests,
+    ...     '--tests-pattern', '^sampletestsf?$',
+    ...     ]
+    >>> sys.argv = ['test',
+    ...             '--tests-pattern', '^sampletests_discover$',
+    ...     ]
+    >>> testrunner.run(defaults)
+    Running zope.testing.testrunner.layer.UnitTests tests:
+      Set up zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
+      Ran 1 tests with 0 failures and 0 errors in N.NNN seconds.
+    Tearing down left over layers:
+      Tear down zope.testing.testrunner.layer.UnitTests in N.NNN seconds.
+    False
+
+If the module neither provides a TestSuite nor has discoverable tests,
+testrunner will exit with an error to prevent acidentally missing test cases:
+
+    >>> sys.argv = ['test',
+    ...             '--tests-pattern', '^sampletests_discover_notests$',
+    ...     ]
+    >>> testrunner.run(defaults)
+    Test-module import failures:
+    <BLANKLINE>
+    Module: sample1.sampletests_discover_notests
+    <BLANKLINE>
+    TypeError: Module sample1.sampletests_discover_notests does not define any tests
+    <BLANKLINE>
+    <BLANKLINE>
+    <BLANKLINE>
+    Test-modules with import problems:
+      sample1.sampletests_discover_notests
+    Total: 0 tests, 0 failures, 0 errors in 0.000 seconds.
+    True
--- a/trunk/src/zope/testing/testrunner/testrunner-errors.txt
+++ b/trunk/src/zope/testing/testrunner/testrunner-errors.txt
@@ -724,15 +724,10 @@ Then run the tests:
     <BLANKLINE>
     Module: sample2.sample22.sampletests_i
     <BLANKLINE>
-    AttributeError: 'module' object has no attribute 'test_suite'
-    <BLANKLINE>
-    <BLANKLINE>
-    Module: sample2.sample23.sampletests_i
-    <BLANKLINE>
     Traceback (most recent call last):
-      File "testrunner-ex/sample2/sample23/sampletests_i.py", line 18, in ?
+      File "testrunner-ex/sample2/sample22/sampletests_i.py", line 18, in ?
         class Test(unittest.TestCase):
-      File "testrunner-ex/sample2/sample23/sampletests_i.py", line 23, in Test
+      File "testrunner-ex/sample2/sample22/sampletests_i.py", line 23, in Test
         raise TypeError('eek')
     TypeError: eek
     <BLANKLINE>
@@ -773,7 +768,6 @@ Then run the tests:
       sample2.sampletests_i
       sample2.sample21.sampletests_i
       sample2.sample22.sampletests_i
-      sample2.sample23.sampletests_i
     Total: 213 tests, 0 failures, 0 errors in N.NNN seconds.
     True
 
--- /dev/null
+++ b/trunk/src/zope/testing/testrunner/testrunner-ex/sample1/sampletests_discover.py
@@ -0,0 +1,5 @@
+import unittest
+
+class TestA(unittest.TestCase):
+    def test_truth(self):
+        self.assert_(True)
--- /dev/null
+++ b/trunk/src/zope/testing/testrunner/testrunner-ex/sample1/sampletests_discover_notests.py
@@ -0,0 +1,2 @@
+def test_function_that_would_never_be_run():
+    self.assert_(True)