Hi, I also looked at Hudson a few days ago and found it to be awesome. :) On 03/04/2010 10:10 PM, Tres Seaver wrote:

After successfully configuring the Hudson[1] continuous integration server yesterday to test the various repoze.* packages[2], I thought I would experiment with using it to drive the compatibility tests for ZTK and zopeapp. Here is what I did:

In a shell on my server:

$ wget http://hudson-ci.org/latest/hudson.war $ java -jar hudson.war --httpPort=19999

That's what blew me off my chair. :)

....

In the browser, visit 'http://laguna.palladion.com:19999, and configure the server:

- Enable security - allow signup - using Matrix - set up 'tseaver' as having all permissions - set up 'anonymous' as having 'Read' permission overall and for jobs

- Register as 'tseaver' - Turn off signup (no, I don't want you guys running arbitrary shell scripts on my server ;).

- Create a job, 'ztk-trunk-py26':

See below

http://laguna.palladion.com:19999/job/ztk-trunk-py26/

- Use SVN, pointed at 'svn://svn.zope.org/repos/main/zopetoolkit/trunk'

- Set to poll SVN every 15 minutes.

- Enable using 'svn up'

- Added build step as a shell script::

cd trunk /opt/Python-2.6.4/bin/python bootstrap.py bin/buildout bin/test-ztk bin/test-zopeapp

- Ran a build manually. The first one barfed due to a typo in my script, but the second one ran, taking 21 minutes.

Right. That was my experience as well. Also a good thing is that you can now take those as templates, or - if you like - generate those XML files into the filesystem for new builds and ping the server to reload those.

The test runs successfully to completion, but doesn't produce statistics in a form that Hudson understands. The following changes to 'zope.testing.testrunner' would make the output more informative:

- Add a flag like the '--with-xunit' flag to the Nose testrunner, which dumped results into a JUnit-compatible XML file[3].

- Add a flag like the '--with-xcompat' flag to the Nose testrunner (when run with the 'nosexcover' plugin), producing a Cobertura- compabtible XML file[4].

Right. However Hudson does understand failures/successes at least. I made a job for the ZODB where the individual steps (checkout, bootstrap/buildout, test) where separated steps so that made the UI integration a little bit better.

At that point, the build process would be nicely automated, with browsable results including coverage, pretty graphs, etc. See the 'repoze' jobs for examples of those outputs.

Here's another awesome part: You can configure matrix build jobs which would allow us to define a single job for i.e. ZODB and then say "we want this tun run in the combinations of Unix/Windows, 32/64, Python 2.4/2.5/2.6, trunk/branchx /branchy" and it goes off automatically trying to find build nodes that can provide all combinations. It took my about 10 minutes to get that going including the installment of the Windows builder.

Questions for discussion:

- I find the prettier interface and easier setup than buildbot worth running a 3rd-party Java app (rather than a 3rd party Python app). Would that be acceptable among the folks running our automated tests?

It would be for me. Right now I'll continue to clean up what we have with buildbot. I really don't want to put anybody off. Getting more coverage (however) and making it more visible and regular is my primary goal. I don't think we have to through anything out. I'll be happy to provide even more tests using other build systems in parallel.

- Is it worth adding the XML support for test results and coverage to 'zope.testing.testrunner'?

Maybe. I'm happy hacking on the testrunner. :)

- Would people be willing to run Hudson "permanently" on enough hosts to make this a reasonable replacement for buildbot (we need Windows, too)?

I'm definitely willing to run it for 32/64 bit Linux machines. I'm short on Windows licenses but Hudson seems easy to integrate securely distributed over multiples sites and easy enough to set up with the JNRE/service installer.

- Hudson seems general purpose enough to allow automating other code- related tasks, e.g., production of dependency graphs. Maybe we could even automate building Windows installers?

How awesome that would be. :)

I will try to leave the ZTK Hudson server up for a little bit, to allow folks to see what it can deliver already.

[1] http://hudson-ci.org/

[2] http://hudson.repoze.org/

[3] http://old.nabble.com/schema-for-junit-xml-output-td22193385.html

[4] http://cobertura.sourceforge.net/xml/coverage-01.dtd

Thanks a lot for the effort. I'm really excited about Hudson and I'm willing to help out here. Christian -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development

On Thu, Mar 04, 2010 at 04:10:29PM -0500, Tres Seaver wrote:

After successfully configuring the Hudson[1] continuous integration server yesterday to test the various repoze.* packages[2], I thought I would experiment with using it to drive the compatibility tests for ZTK and zopeapp. Here is what I did:

In a shell on my server:

$ wget http://hudson-ci.org/latest/hudson.war $ java -jar hudson.war --httpPort=19999 ....

As a sysadmin I preferred echo deb http://hudson-ci.org/debian binary/ > /etc/apt/sources.list.d/hudson.list sudo apt-get update && sudo apt-get install hudson # Visit http://server:8080/ or edit /etc/default/hudson to change the # port number This makes it run in a separate Unix user account, start automatically on system boot, and it lets me get updates easily.

In the browser, visit 'http://laguna.palladion.com:19999, and configure the server:

- Enable security - allow signup - using Matrix - set up 'tseaver' as having all permissions - set up 'anonymous' as having 'Read' permission overall and for jobs

- Register as 'tseaver' - Turn off signup (no, I don't want you guys running arbitrary shell scripts on my server ;).

We have commit rights to the SVN repository. We already can run arbitrary shell scripts on your server. ;) Of course the whole audit trail thing might make it less attractive ;)

- Create a job, 'ztk-trunk-py26': ... - Added build step as a shell script::

cd trunk /opt/Python-2.6.4/bin/python bootstrap.py bin/buildout bin/test-ztk bin/test-zopeapp

- Ran a build manually. The first one barfed due to a typo in my script, but the second one ran, taking 21 minutes.

And the third and subsequent ones took 11 minutes. The initial buildout run is not fast.

The test runs successfully to completion, but doesn't produce statistics in a form that Hudson understands. The following changes to 'zope.testing.testrunner' would make the output more informative:

- Add a flag like the '--with-xunit' flag to the Nose testrunner, which dumped results into a JUnit-compatible XML file[3].

I'd like that.

- Add a flag like the '--with-xcompat' flag to the Nose testrunner (when run with the 'nosexcover' plugin), producing a Cobertura- compabtible XML file[4].

I believe you can do that with Ned Batchelder's coverage.py if you include 'coverage' in your eggs list in buildout.cfg and run the tests as bin/coverage run bin/test [options as usual] bin/coverage xml Downside: Hudson runs the steps as a shell script with 'set -e' enabled, which means that the first command that fails stops the script. You won't get coverage results if your tests fail. (Also, if ztk tests fail, zopeapp tests won't even be run). The syntax for combining coverage files produced by several test runs is left as an exercise for the reader: http://nedbatchelder.com/code/coverage/cmd.html

At that point, the build process would be nicely automated, with browsable results including coverage, pretty graphs, etc. See the 'repoze' jobs for examples of those outputs.

Nice. Incidentally, I recommend the "green balls" plugin that I see you're already using for hudson.repoze.org. Blue doesn't really signal "job was successful" to me.

Questions for discussion:

- I find the prettier interface and easier setup than buildbot worth running a 3rd-party Java app (rather than a 3rd party Python app). Would that be acceptable among the folks running our automated tests?

A year ago I'd have said "there will be no PHP or Java on my servers". A month ago I installed Hudson because Buildbot finally got to me. I'm still holding the fort against PHP, though ;)

- Is it worth adding the XML support for test results and coverage to 'zope.testing.testrunner'?

I think so, at least for the test results.

- Would people be willing to run Hudson "permanently" on enough hosts to make this a reasonable replacement for buildbot (we need Windows, too)?

I'd have to overcome my paranoia of letting somebody else define what shell commands get run on my servers, but it seems enough people are already providing Linux build slaves.

- Hudson seems general purpose enough to allow automating other code- related tasks, e.g., production of dependency graphs. Maybe we could even automate building Windows installers?

It would be *awesome* if people making releases of core packages like zope.interface didn't need to build Windows binary eggs by themselves.

I will try to leave the ZTK Hudson server up for a little bit, to allow folks to see what it can deliver already.

[1] http://hudson-ci.org/

[2] http://hudson.repoze.org/

[3] http://old.nabble.com/schema-for-junit-xml-output-td22193385.html

[4] http://cobertura.sourceforge.net/xml/coverage-01.dtd

Regards, Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development