-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Asbestos - -------- First, note that I am not proposing to replace the current testing machinery used by Zope developers working on a given package (typically, buildout plus recipes which configure zope.testing.testrunner as a standalone script). Nor am I proposing that we modify that machinery to take advantage of setuptools' test-oriented metadata (that would be a separate proposal). Proposal - -------- Instead, what I *am* proposing is adding metadata which allows consumers of such packages to verify that the package is downloaded / installed correctly. In particular, I want for the following scenario to work seamlessly:: $ /path/to/virtualenv --no-site-packges sandbox ... $ cd sandbox $ tar xzf zope.foo.bar-3.5.6.tar.gz $ cd zope.foo.bar-3.5.6 # [1] $ ../bin/python setup.py test # [2] # Runs egg_info, installs regular and testing dependencies, and # runs all unit (non-layer) tests $ ../bin/python setup.py test # [3] # Runs egg_info, installs regular and t # runs *all* tests Theory of Operation - ------------------- [1] Ideally, this would be some variation on 'easy_install' which got the package into the environment in such a way that its 'setup.py' could be run. Fetching and unpacking the tarball manually works today, however ;) [2] 'python setup.py test' is the "standard" way to test a setuptools-based package, and would be expected to work by most egg-savvy Python developers. setuptools provides the 'test' command, using a built-in testrunner which mostly Just Works(TM). However, it doesn't support the 'layer' feature provided by 'zope.testrunner', and therefore many Zopeish packages emit failures when run this way. Without monkeypatching setuptools, we can't replace the testrunner it uses, but we *can* supply a different test loader using an option in setup.py[4]. The attached module provides such a loader, which filters out the doomed tests, leaving only those which can run cleanly without layers (e.g., the unit tests). [3] In the second method, we use the setuptools entry point mechanism to register a custom 'test' command[5]. This is a wee bit fragile, as running 'setup.py test' without first generating the egg_info will use the "standard" testrunner (but only the first time). If such fragility is an issue, we can register a separate command, e.g. 'ftest'; in either case, we have to document that you need to run 'setup.py egg_info' one time before running the custom command. If the ideal installation outlined in [1] above worked, that would solve the problem, too, because the egg-info would be generated by easy_install. This techniquie drives the zope.testing.testrunner via an entry point, which is cool, but it cannot expose all of testrunner's features. In particular, the configuration knobs which the testrunner expects to get on the command-line clash with distutils' own command-line syntax (yes, I've tried to make it work). The command I have provided just runs all the tests, unit and functional, which seems to be reasonable assurance for the package conumer that the package installed cleanly. Implementation - -------------- The intent is to add the attached module as 'zope.testing.testrunner.eggsupport', and then to modify our packages (opportunistically, or through a 'geddon) as follows: - Add 'setup_requires = ["eggtestinfo"]', so that the testing metadata is captured in EGG-INFO. Note that this information is not needed when running the tests from a source distribution, but should ease a TBD mode for testing binary distributions. - Add 'tests_require = ["zope.testing >= 3.7"]' (or whatever the next released version including this feature would be), to make the loader available. - Add 'test_loader = "zope.testing.testrunner.eggsupport:SkipLayers"' which enables 'setup.py test' to use the custom loader. Once the new 'zope.testing' package is present, then the 'ftest' command will also be available, but still needs to be called out via an entry point:: entry_points=""" [distutils.commands] ftest = zope.testing.testrunner.eggsupport:ftest """ The second attachment is a diff showing the changes to a typical Zope2 product's setup.py to support this proposal. References - ---------- [4] http://peak.telecommunity.com/DevCenter/setuptools#test-loader [5] http://peak.telecommunity.com/DevCenter/setuptools#adding-commands Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.6 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org iD8DBQFI0TXh+gerLs4ltQ4RAuEGAJ411zdUI+GjQZEkPtf2uB1zgX3KeQCgvjlA rHFCcXsSaPpY9AklWGqlmqk= =PbVE -----END PGP SIGNATURE-----