[Zope-dev] Automating tests of the ZTK / zopeapp package sets

[Marius Gedminas]

On Thu, Mar 04, 2010 at 04:10:29PM -0500, Tres Seaver wrote:
> After successfully configuring the Hudson[1] continuous integration
> server yesterday to test the various repoze.* packages[2], I thought I
> would experiment with using it to drive the compatibility tests for ZTK
> and zopeapp.  Here is what I did:
> 
> In a shell on my server:
> 
>  $ wget http://hudson-ci.org/latest/hudson.war
>  $ java -jar hudson.war --httpPort=19999
>  ....

As a sysadmin I preferred

  echo deb http://hudson-ci.org/debian binary/ > /etc/apt/sources.list.d/hudson.list
  sudo apt-get update && sudo apt-get install hudson

  # Visit http://server:8080/ or edit /etc/default/hudson to change the
  # port number

This makes it run in a separate Unix user account, start automatically
on system boot, and it lets me get updates easily.

> In the browser, visit 'http://laguna.palladion.com:19999, and configure
> the server:
> 
> - Enable security
>   - allow signup
>   - using Matrix
>   - set up 'tseaver' as having all permissions
>   - set up 'anonymous' as having 'Read' permission overall and for jobs
> 
> - Register as 'tseaver'
>   - Turn off signup (no, I don't want you guys running arbitrary
>     shell scripts on my server ;).

We have commit rights to the SVN repository.  We already can run
arbitrary shell scripts on your server. ;)

Of course the whole audit trail thing might make it less attractive ;)

> - Create a job, 'ztk-trunk-py26':
...
>   - Added build step as a shell script::
> 
>     cd trunk
>     /opt/Python-2.6.4/bin/python bootstrap.py
>     bin/buildout
>     bin/test-ztk
>     bin/test-zopeapp
> 
>   - Ran a build manually.  The first one barfed due to a typo
>     in my script, but the second one ran, taking 21 minutes.

And the third and subsequent ones took 11 minutes.  The initial buildout
run is not fast.

> The test runs successfully to completion, but doesn't produce statistics
> in a form that Hudson understands.  The following changes to
> 'zope.testing.testrunner' would make the output more informative:
> 
>  - Add a flag like the '--with-xunit' flag to the Nose testrunner, which
>    dumped results into a JUnit-compatible XML file[3].

I'd like that.

>  - Add a flag like the '--with-xcompat' flag to the Nose testrunner
>    (when run with the 'nosexcover' plugin), producing a Cobertura-
>    compabtible XML file[4].

I believe you can do that with Ned Batchelder's coverage.py if you
include 'coverage' in your eggs list in buildout.cfg and run the tests
as

   bin/coverage run bin/test [options as usual]
   bin/coverage xml

Downside: Hudson runs the steps as a shell script with 'set -e' enabled,
which means that the first command that fails stops the script.  You
won't get coverage results if your tests fail.  (Also, if ztk tests
fail, zopeapp tests won't even be run).

The syntax for combining coverage files produced by several test runs is
left as an exercise for the reader:
http://nedbatchelder.com/code/coverage/cmd.html

> At that point, the build process would be nicely automated, with
> browsable results including coverage, pretty graphs, etc.  See the
> 'repoze' jobs for examples of those outputs.

Nice.

Incidentally, I recommend the "green balls" plugin that I see you're
already using for hudson.repoze.org.   Blue doesn't really signal "job
was successful" to me.

> Questions for discussion:
> 
> - I find the prettier interface and easier setup than buildbot worth
>   running a 3rd-party Java app (rather than a 3rd party Python app).
>   Would that be acceptable among the folks running our automated tests?

A year ago I'd have said "there will be no PHP or Java on my servers".
A month ago I installed Hudson because Buildbot finally got to me.  I'm
still holding the fort against PHP, though ;)

> - Is it worth adding the XML support for test results and coverage to
>   'zope.testing.testrunner'?

I think so, at least for the test results.

> - Would people be willing to run Hudson "permanently" on enough hosts
>   to make this a reasonable replacement for buildbot (we need Windows,
>   too)?

I'd have to overcome my paranoia of letting somebody else define what
shell commands get run on my servers, but it seems enough people are
already providing Linux build slaves.

> - Hudson seems general purpose enough to allow automating other code-
>   related tasks, e.g., production of dependency graphs.  Maybe we could
>   even automate building Windows installers?

It would be *awesome* if people making releases of core packages like
zope.interface didn't need to build Windows binary eggs by themselves.

> I will try to leave the ZTK Hudson server up for a little bit, to allow
> folks to see what it can deliver already.
> 
> [1] http://hudson-ci.org/
> 
> [2] http://hudson.repoze.org/
> 
> [3] http://old.nabble.com/schema-for-junit-xml-output-td22193385.html
> 
> [4] http://cobertura.sourceforge.net/xml/coverage-01.dtd

Regards,
Marius Gedminas
-- 
http://pov.lt/ -- Zope 3 consulting and development
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 197 bytes
Desc: Digital signature
Url : http://mail.zope.org/pipermail/zope-dev/attachments/20100305/9b342377/attachment.bin