[Zope-Coders] Help with Unit Testing: safe imports

[Tres Seaver]

On Mon, 2002-08-19 at 11:58, Jeremy Hylton wrote:
> >>>>> "TS" == Tres Seaver <tseaver@zope.com> writes:
> 
>   TS> On Mon, 2002-08-19 at 10:01, Guido van Rossum wrote:
>   >> > Python's own 'unittest.py' fails silently if the test module
>   >> > bails with import errors.
>   >> >
>   >> > Successful import of the module-under-test should *not* be
>   >> > assumed by the test module..
>   >>
>   >> I don't understand this.  Can you describe a scenario in more
>   >> detail?
> 
>   TS> Constructing a suite of tests for an entire package is tricky:
>   TS> what is the framework supposed to do if it encounters a module
>   TS> which looks like a test (has the correct name, imports unittest,
>   TS> etc.) but which raises an ImportError at module scope?  
> 
> Then the tool running the tests should report this as a failure.  I
> think the the problem, dear Tres, lies not in our tests, but in our
> tools.  <wink>.
> 
>   TS> The tests which *should* have been run are unavailable, due to
>   TS> the partial import, so the reporting machinery can't log them as
>   TS> failed tests.  The execution environment *may* try to report the
>   TS> problem, but the "standard" ones do not do so.  For instance,
>   TS> with module-scope imports, the stock unittestgui.py will show a
>   TS> "green bar", with a reduced number of passing tests.
> 
> I'd report this as a bug on unittestgui.py.
> 
>   TS> The idiom I propose (import the module-under-test only within
>   TS> the context of 'testFoo' method) has netiher disadvantage:
>   TS> import errors are caught and reported as test failures, without
>   TS> any need for additional reporting machinery.
> 
> But it makes writing the tests more tedious because they have to
> include lots of repetitive boilerplate.  If the boilerplate were truly
> necessary that would be okay.  I live with Py_INCREF() and Py_DECREF()
> after all.  But we can just fix the tools.

Note that the stock 'unittest.py' swallows ImportErrors:

    def loadTestsFromName(self, name, module=None):
        """Return a suite of all tests cases given a string specifier.

        The name may resolve either to a module, a test case class, a
        test method within a test case class, or a callable object which
        returns a TestCase or TestSuite instance.

        The method optionally resolves the names relative to a given
module.
        """
        parts = string.split(name, '.')
        if module is None:
            if not parts:
                raise ValueError, "incomplete test name: %s" % name
            else:
                parts_copy = parts[:]
                while parts_copy:
                    try:
                        module = __import__(string.join(parts_copy,'.'))
                        break
                    except ImportError:
                        del parts_copy[-1]
                        if not parts_copy: raise
                parts = parts[1:]
        obj = module
        for part in parts:
            obj = getattr(obj, part)

Note that this does *not* yield appropriate results:

----------------- begin sample ---------------

$ ll
total 16
drwxr-xr-x    3 tseaver  tseaver      4096 Aug 19 13:44 ./
drwxrwxrwt   15 root     root         8192 Aug 19 13:44 ../
drwxr-xr-x    3 tseaver  tseaver      4096 Aug 19 13:36 foo/
lrwxrwxrwx    1 tseaver  tseaver        30 Aug 19 13:39 unittest.py ->
/usr/lib/python2.1/unittest.py

$ find .
.
./foo
./foo/bar
./foo/bar/baz
./foo/bar/baz/__init__.py
./foo/bar/baz/tests
./foo/bar/baz/tests/__init__.py
./foo/bar/baz/tests/testBaz.py
./foo/bar/baz/Baz.py
./foo/bar/__init__.py
./foo/__init__.py
./unittest.py

$ cat foo/bar/baz/Baz.py
import Qux  # Note:  this won't import!

class Baz:
    pass

$ cat foo/bar/baz/tests/testBaz.py
import unittest

from foo.bar.baz.Baz import Baz

class BazTests( unittest.TestCase ):

    def test_something( self ):
        pass

$ PYTHONPATH=`pwd` python2.1 unittest.py foo.bar.baz.tests.testBaz
Traceback (most recent call last):
  File "unittest.py", line 712, in ?
    main(module=None)
  File "unittest.py", line 663, in __init__
    self.parseArgs(argv)
  File "unittest.py", line 690, in parseArgs
    self.createTests()
  File "unittest.py", line 696, in createTests
    self.module)
  File "unittest.py", line 458, in loadTestsFromNames
    suites.append(self.loadTestsFromName(name, module))
  File "unittest.py", line 433, in loadTestsFromName
    obj = getattr(obj, part)
AttributeError: 'foo.bar.baz.tests' module has no attribute 'testBaz'

----------------- end sample -----------------

The ImportError (in Baz.py) is swallowed, leaving testBaz.py in
a "partially imported" mess.

Tres.
-- 
===============================================================
Tres Seaver                                tseaver@zope.com
Zope Corporation      "Zope Dealers"       http://www.zope.com