[Zope-dev] RFC: adding support for testing from bare setuptools

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Asbestos
- --------

First, note that I am not proposing to replace the current testing
machinery used by Zope developers working on a given package (typically,
buildout plus recipes which configure zope.testing.testrunner as a
standalone script).  Nor am I proposing that we modify that machinery to
take advantage of setuptools' test-oriented metadata (that would be a
separate proposal).


Proposal
- --------

Instead, what I *am* proposing is adding metadata which allows consumers
of such packages to verify that the package is downloaded / installed
correctly.  In particular, I want for the following scenario to work
seamlessly::

 $ /path/to/virtualenv --no-site-packges sandbox
 ...
 $ cd sandbox
 $ tar xzf zope.foo.bar-3.5.6.tar.gz
 $ cd zope.foo.bar-3.5.6           # [1]

 $ ../bin/python setup.py test     # [2]
 # Runs egg_info, installs regular and testing dependencies, and
 # runs all unit (non-layer) tests

 $ ../bin/python setup.py test    # [3]
 # Runs egg_info, installs regular and t
 # runs *all* tests


Theory of Operation
- -------------------

[1] Ideally, this would be some variation on 'easy_install' which got
the package into the environment in such a way that its 'setup.py' could
be run.  Fetching and unpacking the tarball manually works today, however ;)

[2] 'python setup.py test' is the "standard" way to test a
setuptools-based package, and would be expected to work by most
egg-savvy Python developers.  setuptools provides the 'test' command,
using a built-in testrunner which mostly Just Works(TM).  However, it
doesn't support the 'layer' feature provided by 'zope.testrunner', and
therefore many Zopeish packages emit failures when run this way.

Without monkeypatching setuptools, we can't replace the testrunner it
uses, but we *can* supply a different test loader using an option in
setup.py[4].  The attached module provides such a loader, which filters
out the doomed tests, leaving only those which can run cleanly without
layers (e.g., the unit tests).

[3] In the second method, we use the setuptools entry point mechanism to
register a custom 'test' command[5].  This is a wee bit fragile, as
running 'setup.py test' without first generating the egg_info will use
the "standard" testrunner (but only the first time).  If such fragility
is an issue, we can register a separate command, e.g. 'ftest';  in
either case, we have to document that you need to run 'setup.py
egg_info' one time before running the custom command.  If the ideal
installation outlined in [1] above worked, that would solve the problem,
too, because the egg-info would be generated by easy_install.

This techniquie drives the zope.testing.testrunner via an entry point,
which is cool, but it cannot expose all of testrunner's features.
In particular, the configuration knobs which the testrunner expects to
get on the command-line clash with distutils' own command-line syntax
(yes, I've tried to make it work).  The command I have provided just
runs all the tests, unit and functional, which seems to be reasonable
assurance for the package conumer that the package installed cleanly.


Implementation
- --------------

The intent is to add the attached module as
'zope.testing.testrunner.eggsupport', and then to modify our packages
(opportunistically, or through a 'geddon) as follows:

 - Add 'setup_requires = ["eggtestinfo"]', so that the testing metadata
   is captured in EGG-INFO.  Note that this information is not needed
   when running the tests from a source distribution, but should ease
   a TBD mode for testing binary distributions.

 - Add 'tests_require = ["zope.testing >= 3.7"]' (or whatever the next
   released version including this feature would be), to make the
   loader available.

 - Add 'test_loader = "zope.testing.testrunner.eggsupport:SkipLayers"'
   which enables 'setup.py test' to use the custom loader.

Once the new 'zope.testing' package is present, then the 'ftest' command
will also be available, but still needs to be called out via an entry
point::

   entry_points="""
   [distutils.commands]
   ftest = zope.testing.testrunner.eggsupport:ftest
   """

The second attachment is a diff showing the changes to a typical Zope2
product's setup.py to support this proposal.



References
- ----------

[4] http://peak.telecommunity.com/DevCenter/setuptools#test-loader

[5] http://peak.telecommunity.com/DevCenter/setuptools#adding-commands


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.6 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iD8DBQFI0TXh+gerLs4ltQ4RAuEGAJ411zdUI+GjQZEkPtf2uB1zgX3KeQCgvjlA
rHFCcXsSaPpY9AklWGqlmqk=
=PbVE
-----END PGP SIGNATURE-----
-------------- next part --------------
A non-text attachment was scrubbed...
Name: eggsupport.py
Type: application/x-httpd-cgi
Size: 3702 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080917/f98cc48a/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: GS_eggsupport_changes.patch
Type: text/x-patch
Size: 1111 bytes
Desc: not available
Url : http://mail.zope.org/pipermail/zope-dev/attachments/20080917/f98cc48a/attachment-0001.bin