[Zope-dev] Automating tests of the ZTK / zopeapp package sets

[Tres Seaver]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

After successfully configuring the Hudson[1] continuous integration
server yesterday to test the various repoze.* packages[2], I thought I
would experiment with using it to drive the compatibility tests for ZTK
and zopeapp.  Here is what I did:

In a shell on my server:

 $ wget http://hudson-ci.org/latest/hudson.war
 $ java -jar hudson.war --httpPort=19999
 ....

In the browser, visit 'http://laguna.palladion.com:19999, and configure
the server:

- - Enable security
  - allow signup
  - using Matrix
  - set up 'tseaver' as having all permissions
  - set up 'anonymous' as having 'Read' permission overall and for jobs

- - Register as 'tseaver'
  - Turn off signup (no, I don't want you guys running arbitrary
    shell scripts on my server ;).

- - Create a job, 'ztk-trunk-py26':

  http://laguna.palladion.com:19999/job/ztk-trunk-py26/

  - Use SVN, pointed at
   'svn://svn.zope.org/repos/main/zopetoolkit/trunk'

    - Set to poll SVN every 15 minutes.

    - Enable using 'svn up'

  - Added build step as a shell script::

    cd trunk
    /opt/Python-2.6.4/bin/python bootstrap.py
    bin/buildout
    bin/test-ztk
    bin/test-zopeapp

  - Ran a build manually.  The first one barfed due to a typo
    in my script, but the second one ran, taking 21 minutes.

The test runs successfully to completion, but doesn't produce statistics
in a form that Hudson understands.  The following changes to
'zope.testing.testrunner' would make the output more informative:

 - Add a flag like the '--with-xunit' flag to the Nose testrunner, which
   dumped results into a JUnit-compatible XML file[3].

 - Add a flag like the '--with-xcompat' flag to the Nose testrunner
   (when run with the 'nosexcover' plugin), producing a Cobertura-
   compabtible XML file[4].

At that point, the build process would be nicely automated, with
browsable results including coverage, pretty graphs, etc.  See the
'repoze' jobs for examples of those outputs.

Questions for discussion:

- - I find the prettier interface and easier setup than buildbot worth
  running a 3rd-party Java app (rather than a 3rd party Python app).
  Would that be acceptable among the folks running our automated tests?

- - Is it worth adding the XML support for test results and coverage to
  'zope.testing.testrunner'?

- - Would people be willing to run Hudson "permanently" on enough hosts
  to make this a reasonable replacement for buildbot (we need Windows,
  too)?

- - Hudson seems general purpose enough to allow automating other code-
  related tasks, e.g., production of dependency graphs.  Maybe we could
  even automate building Windows installers?

I will try to leave the ZTK Hudson server up for a little bit, to allow
folks to see what it can deliver already.

[1] http://hudson-ci.org/

[2] http://hudson.repoze.org/

[3] http://old.nabble.com/schema-for-junit-xml-output-td22193385.html

[4] http://cobertura.sourceforge.net/xml/coverage-01.dtd


Tres.
- --
===================================================================
Tres Seaver          +1 540-429-0999          tseaver at palladion.com
Palladion Software   "Excellence by Design"    http://palladion.com
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAkuQIcAACgkQ+gerLs4ltQ61pACfXsIdiRGY2LQ+u8d/qpG2okpT
eKQAn0gjegHqlvu7w3RV01IruUx7S5QM
=OmZN
-----END PGP SIGNATURE-----