[Zope-dev] Test Runner (was Re: Dependencies and future of zope 3)

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Jim Fulton wrote:
> On Sep 3, 2008, at 8:37 PM, Tres Seaver wrote:
>> The change under discussion about testing is whether or not to remove
>> the test requirements as part of 'install_requires' in setup.py.  Some
>> folks have objecctions to using the  setuptools-provided  
>> 'tests_require'
>> field, although I think the argument there is weak:  packages which
>> spell 'tests_require' and 'test_suite' in their setup can be tested
>> trivially with 'python setup.py test', which seems a win to me.
> 
> I agree that this is a win.  It is not a big win IMO, as I prefer a  
> separate test runner.
> 
> It would be far more useful to me if the data were captured as part of  
> egg info.
> 
>> I'm not volunteering to write it, but it strikes me as odd that folks
>> haven't morphed zc.buildout (and / or associated recipes) to use this
>> field.  I *did* write a setuptools add on which saved the
>> 'tests_require' info into the EGG_INFO directory:  that package  
>> could be
>> used to capture the metadata during installation of packages, for
>> consumption by a testrunner later.
> 
> That's very good.  It would be much much much better if setuptools  
> would do this by default. Now that setuptools' development is opening  
> up, perhaps we can do this.

'egginfo' uses the established "setuptools plugin" API:  I don't have
commit access, but I would be fine with having it merged.  In the
meantime, including it within 'zc.buildout' environments would make the
information widely available.

> The setuptools add-on doesn't help when you don't have source releases  
> and many interesting packages get distributed as eggs on Windows.

'egginfo' includes the test info in bdist_egg zipfiles if it is present
on the system where the egg is built.

>> I discovered today I think the time is ripe for a "blank buffer"  
>> rewrite
>> of the testrunner:  it is so full of "twisty passages" that my
>> confidence in its own internal correctness is seriously depleted.  It
>> has too many features,
> 
> I doubt it.  I find the vast majority of it's features very useful. I  
> agree that the test discovery is ripe for a revamp.

Can you truly say that reading the code doesn't make your eyes cross?
Here are a couple I noted while debugging broken discovery yesterday:

 - Extensive use of generators, in spite of the fact that we always
   consume them all.  It feels like a "here's a new language feature,
   let's use it" choice, and makes debugging the code harder.  I doubt
   very much that the decision was driven by an actual constraint (the
   only one I can imagine is RAM consumption).

 - '--path' vs '--test-path' vs. '--package-path' interact in
   unpredictable ways, based on ordering.


>> and too many incompatible ways to run it:  I
>> would love a simpler version, whose discovery logic used egg metadata
>> instead of (mal)heuristics (e.g., ordering of '--test-path' and
>> '--package-path' arguments can make some tests unfindable).
> 
> I'm very open to revamping test discovery.
> 
> I'm opposed to a rewrite.  I sure as heck wouldn't be willing to spend  
> time on a rewrite. I wouldn't be opposed to replacing out test runner  
> with a bunch of nose plugins that added missing features.  I have  
> almost zero interest in using setuptools test runner during development.

I typically use 'setup.py test' in two cases:

 - When evaluating a new package, I create a virtualenv and run
   'setup.py develop' and 'setup.py test' using the virtualenv's
   python.  Packages whose tests run clealy in a sandbox get a higher
   "score" than those which don't:  for one thing, it guarantees that
   their dependencies are accurate.

 - When building packages which I hope will be used outsize of Zopeland,
   I want to make the tests accessible to folks who don't use
   zc.buildout or zope.testing.

I tend to skip running 'setup.py test' only in projects where I am
writing custom software, and where the project is already committed to
another testrunner.


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFIv/4u+gerLs4ltQ4RAp+8AJ9iTagUjGOdkgYGIA7xGqeyFu6X/wCcD74d
6LZrrCg9OsDb8+FYHG2rVmM=
=d2kX
-----END PGP SIGNATURE-----