[Zope-dev] zope.testrunner and nose count doctests differently

[Raphael Ritz]

On 11/3/11 12:13 PM, Chris Withers wrote:
> On 03/11/2011 11:05, Jim Fulton wrote:
>> On Thu, Nov 3, 2011 at 5:14 AM, Chris Withers<chris at simplistix.co.uk>   wrote:
>>> Hi Michael,
>>>
>>> On 03/11/2011 09:12, Michael Howitz wrote:
>>>> Run both test runners with the option -vv to see which tests are run.
>>>> (I did this for your code and the list of tests seems to be equal.)
>>
>> It would be interesting for the rest of us to know the length of that list.
>
> $ bin/test -m testfixtures.tests.test_docs -vv
> Running tests at level 1
> Running zope.testrunner.layer.UnitTests tests:
>     Set up zope.testrunner.layer.UnitTests in 0.000 seconds.
>     Running:
>    testfixtures/docs/api.txt
>    testfixtures/docs/changes.txt
>    testfixtures/docs/comparing.txt
>    testfixtures/docs/components.txt
>    testfixtures/docs/datetime.txt
>    testfixtures/docs/description.txt
>    testfixtures/docs/development.txt
>    testfixtures/docs/exceptions.txt
>    testfixtures/docs/files.txt
>    testfixtures/docs/index.txt
>    testfixtures/docs/installation.txt
>    testfixtures/docs/license.txt
>    testfixtures/docs/logging.txt
>    testfixtures/docs/mocking.txt
>    testfixtures/docs/streams.txt
>    testfixtures/docs/utilities.txt


Note that the above are 16 files ...

>     Ran 316 tests with 0 failures and 0 errors in 0.150 seconds.

... containing what zope.testrunner considers 316 individual tests.

> Tearing down left over layers:
>     Tear down zope.testrunner.layer.UnitTests in 0.000 seconds.
>
> $ bin/nosetests testfixtures/tests/test_docs.py -vv
> nose.config: INFO: Ignoring files matching ['^\\.', '^_', '^setup\\.py$']
> testfixtures_nose/docs/api.txt ... ok
> testfixtures_nose/docs/changes.txt ... ok
> testfixtures_nose/docs/comparing.txt ... ok
> testfixtures_nose/docs/components.txt ... ok
> testfixtures_nose/docs/datetime.txt ... ok
> testfixtures_nose/docs/description.txt ... ok
> testfixtures_nose/docs/development.txt ... ok
> testfixtures_nose/docs/exceptions.txt ... ok
> testfixtures_nose/docs/files.txt ... ok
> testfixtures_nose/docs/index.txt ... ok
> testfixtures_nose/docs/installation.txt ... ok
> testfixtures_nose/docs/license.txt ... ok
> testfixtures_nose/docs/logging.txt ... ok
> testfixtures_nose/docs/mocking.txt ... ok
> testfixtures_nose/docs/streams.txt ... ok
> testfixtures_nose/docs/utilities.txt ... ok

These are the same 16 files (I assume) that nose considers one
test each irrespective of their content. The individual tests
in there are run but not counted separately.

Is it this what's confusing you? It certainly confused me
when I noticed this for the first time. I'd even consider
this a bug in node's reporting but never dared to file a
ticket.

Raphael

>
> ----------------------------------------------------------------------
> Ran 16 tests in 0.141s
>
>> My guess is that when you're using the zope testruner, you're somehow
>> picking up zope.testing.doctest, which counts each doctest example as
>> a  test.
>
> The code uses Manuel, under both nose and zope.testrunner:
>
> from doctest import REPORT_NDIFF,ELLIPSIS
> from glob import glob
> from manuel import doctest,codeblock,capture
> from manuel.testing import TestSuite
> from os.path import dirname,join,pardir
>
> def test_suite():
>       m =  doctest.Manuel(optionflags=REPORT_NDIFF|ELLIPSIS)
>       m += codeblock.Manuel()
>       m += capture.Manuel()
>       return TestSuite(
>           m,
>           *glob(join(dirname(__file__),pardir,pardir,'docs','*.txt'))
>           )
>
> cheers,
>
> Chris
>