buildbot news: sending notifications and current broken builds
Hi, I've watched my buildbot send out notifications for a while and pondered how to make them available to more people. I decided to send them to this list on a 'problem' level. The 'problem' level sends a message when a builder changes from a good state to a bad state but doesn't send a new message on subsequent failing builds. For me this has been reasonably quiet over the last two weeks although about 60 builders are broken. The risk of sending mail here is that a massive build failure can cause about 200 messages being send at once. If anybody knows a better approach with buildbot, please let me know, I'd be happy to implement it. For everyone else: please have a look at the list of broken builds: http://zopebuildbot.whq.gocept.com/cruise If a project that you feel responsible for is broken, please check out whats wrong and maybe fix it. If the build environment isn't suitable, I'll be happy to help. This page should show a green bar! Christian -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development
--On 6. Juni 2008 08:23:40 +0200 Christian Theune <ct@gocept.com> wrote:
Hi,
I've watched my buildbot send out notifications for a while and pondered how to make them available to more people. I decided to send them to this list on a 'problem' level.
The 'problem' level sends a message when a builder changes from a good state to a bad state but doesn't send a new message on subsequent failing builds.
For me this has been reasonably quiet over the last two weeks although about 60 builders are broken.
The risk of sending mail here is that a massive build failure can cause about 200 messages being send at once. If anybody knows a better approach with buildbot, please let me know, I'd be happy to implement it.
For everyone else: please have a look at the list of broken builds: http://zopebuildbot.whq.gocept.com/cruise
A minor usability suggestion: please provide links at the top of the page for every top-level namespace that would setup the filter for this namespace. Andreas
Hello, Not this one, but I added some features on the branch: svn+ssh://svn.zope.org/repos/main/gocept.bsquare/branches/pcardune-setup Friday, June 6, 2008, 8:35:57 AM, you wrote: AJ> A minor usability suggestion: AJ> please provide links at the top of the page for every top-level namespace AJ> that would setup the filter for this namespace. AJ> Andreas -- Best regards, Adam GROSZER mailto:agroszer@gmail.com -- Quote of the day: There's nothing so pathetic as a forgetful liar. - Unknown
On Fri, Jun 06, 2008 at 08:35:57AM +0200, Andreas Jung wrote:
A minor usability suggestion:
please provide links at the top of the page for every top-level namespace that would setup the filter for this namespace.
Done. I'm only displaying namespaces that have at least two packages in them, though. -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development
Hey Christian. Thanks for this. It would be nice to have a link from this page to the failure output (to address "works for me" issues, as well as for a quick triage). Gary
On Jun 6, 2008, at 7:44 AM, Gary Poster wrote:
Hey Christian. Thanks for this. It would be nice to have a link from this page to the failure output (to address "works for me" issues, as well as for a quick triage).
...but I found 'em by digging around on the site, so, mostly nevermind.
On Fri, Jun 06, 2008 at 07:50:30AM -0400, Gary Poster wrote:
On Jun 6, 2008, at 7:44 AM, Gary Poster wrote:
Hey Christian. Thanks for this. It would be nice to have a link from this page to the failure output (to address "works for me" issues, as well as for a quick triage).
...but I found 'em by digging around on the site, so, mostly nevermind.
The report entries for each build are linked to the build overview page now. -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Fri, Jun 6, 2008 at 2:23 AM, Christian Theune <ct@gocept.com> wrote:
I've watched my buildbot send out notifications for a while and pondered how to make them available to more people.
We really need something like this. Thanks. A suggestion: I'd like to have the tests run with --all. -- Benji York Senior Software Engineer Zope Corporation
On Jun 6, 2008, at 8:37 AM, Benji York wrote:
On Fri, Jun 6, 2008 at 2:23 AM, Christian Theune <ct@gocept.com> wrote:
I've watched my buildbot send out notifications for a while and pondered how to make them available to more people.
We really need something like this. Thanks.
A suggestion: I'd like to have the tests run with --all.
Heh, I was going to try and fix a time-out failure (zc.blist) by moving the intentionally long tests to --all...maybe there would be some way to say "don't run me with --all"? Otherwise don't love this idea, though I understand its point.
On Fri, Jun 6, 2008 at 9:34 AM, Gary Poster <gary@zope.com> wrote:
On Jun 6, 2008, at 8:37 AM, Benji York wrote:
A suggestion: I'd like to have the tests run with --all.
Heh, I was going to try and fix a time-out failure (zc.blist) by moving the intentionally long tests to --all...maybe there would be some way to say "don't run me with --all"? Otherwise don't love this idea, though I understand its point.
I avoided suggesting this earlier, but perhaps we need to define a level to run the tests at. Then buildbot could run all tests at level X and below and you could put the truly long-running tests above X. -- Benji York Senior Software Engineer Zope Corporation
On Fri, Jun 6, 2008 at 9:44 AM, Benji York <benji@zope.com> wrote:
I avoided suggesting this earlier, but perhaps we need to define a level to run the tests at. Then buildbot could run all tests at level X and below and you could put the truly long-running tests above X.
Another approach would be to have the buildbot run a bin/buildbot-test instead of bin/test, if present. That let's the package suggest what makes sense for running under buildbot without having to get people to agree on how "levels" are used (a more complicated issue). -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "Chaos is the score upon which reality is written." --Henry Miller
On Jun 6, 2008, at 9:48 AM, Fred Drake wrote:
On Fri, Jun 6, 2008 at 9:44 AM, Benji York <benji@zope.com> wrote:
I avoided suggesting this earlier, but perhaps we need to define a level to run the tests at. Then buildbot could run all tests at level X and below and you could put the truly long-running tests above X.
Another approach would be to have the buildbot run a bin/buildbot-test instead of bin/test, if present. That let's the package suggest what makes sense for running under buildbot without having to get people to agree on how "levels" are used (a more complicated issue).
Huh, that sounds like it might be easy-ish for Christian, putting the responsibility where it arguably should be, on the package maintainer. +1, if Christian says it's easy for him.
On Fri, Jun 06, 2008 at 09:56:38AM -0400, Gary Poster wrote:
On Jun 6, 2008, at 9:48 AM, Fred Drake wrote:
On Fri, Jun 6, 2008 at 9:44 AM, Benji York <benji@zope.com> wrote:
I avoided suggesting this earlier, but perhaps we need to define a level to run the tests at. Then buildbot could run all tests at level X and below and you could put the truly long-running tests above X.
Another approach would be to have the buildbot run a bin/buildbot-test instead of bin/test, if present. That let's the package suggest what makes sense for running under buildbot without having to get people to agree on how "levels" are used (a more complicated issue).
Huh, that sounds like it might be easy-ish for Christian, putting the responsibility where it arguably should be, on the package maintainer. +1, if Christian says it's easy for him.
Sounds easy to do. Putting that on my list. -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Fri, Jun 06, 2008 at 09:34:30AM -0400, Gary Poster wrote:
On Jun 6, 2008, at 8:37 AM, Benji York wrote:
A suggestion: I'd like to have the tests run with --all.
Heh, I was going to try and fix a time-out failure (zc.blist) by moving the intentionally long tests to --all...maybe there would be some way to say "don't run me with --all"? Otherwise don't love this idea, though I understand its point.
There are two ways to avoid buildbot timeouts: * pass the -v option to the test runner (when buildout sees dots appearing it resets its timeout counter, and the dots themselves do not obscure errors in the log) * increase the buildbot timeout (pass timeout=60*60 to the step that runs the tests) If even buildbot, who has the patience of a computer, is not running tests that take a long time, will anyone else ever run them? Marius Gedminas -- Many people enjoy Perl, many enjoy Python, some enjoy /bin/tcsh. The latter population should however, needless to say, be put into working camps. -- viktor on Slashdot
On Tue, Jun 10, 2008 at 12:35:17AM +0300, Marius Gedminas wrote:
On Fri, Jun 06, 2008 at 09:34:30AM -0400, Gary Poster wrote:
On Jun 6, 2008, at 8:37 AM, Benji York wrote:
A suggestion: I'd like to have the tests run with --all.
Heh, I was going to try and fix a time-out failure (zc.blist) by moving the intentionally long tests to --all...maybe there would be some way to say "don't run me with --all"? Otherwise don't love this idea, though I understand its point.
There are two ways to avoid buildbot timeouts:
* pass the -v option to the test runner (when buildout sees dots appearing it resets its timeout counter, and the dots themselves do not obscure errors in the log) * increase the buildbot timeout (pass timeout=60*60 to the step that runs the tests)
I opted for using -v for now. We'll see how that goes.
If even buildbot, who has the patience of a computer, is not running tests that take a long time, will anyone else ever run them?
Exactly. Christian -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Fri, Jun 06, 2008 at 08:37:48AM -0400, Benji York wrote:
On Fri, Jun 6, 2008 at 2:23 AM, Christian Theune <ct@gocept.com> wrote:
I've watched my buildbot send out notifications for a while and pondered how to make them available to more people.
We really need something like this. Thanks.
A suggestion: I'd like to have the tests run with --all.
I've added --all for now, we'll see what comes out of the discussion of identifying a specific level for each package that the buildbot should run. -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development
H! On Fri, Jun 06, 2008 at 08:23:40AM +0200, Christian Theune wrote:
I've watched my buildbot send out notifications for a while and pondered how to make them available to more people. I decided to send them to this list on a 'problem' level.
+1
The 'problem' level sends a message when a builder changes from a good state to a bad state but doesn't send a new message on subsequent failing builds.
This implies that buildbot keeps the state somewhere. Then it could also remember the message ID of the failure message and later, when the builder changes from a bad state to a good state, send another message, announcing the end of the failure, with an In-Reply-To header containing the previous message ID, for proper threading. That would be lovely.
For everyone else: please have a look at the list of broken builds: http://zopebuildbot.whq.gocept.com/cruise
Nice. Marius Gedminas -- Please do not even think about automatically normalizing file names anywhere. There is absolutely no need for introducing such nonsense, and deviating from the POSIX requirement that filenames be opaque byte strings is a Bad Idea[TM] (also known as NTFS). -- Markus Kuhn
On Mon, Jun 9, 2008 at 5:41 PM, Marius Gedminas <mgedmin@b4net.lt> wrote:
This implies that buildbot keeps the state somewhere. Then it could also remember the message ID of the failure message and later, when the builder changes from a bad state to a good state, send another message, announcing the end of the failure, with an In-Reply-To header containing the previous message ID, for proper threading.
That would be lovely.
Indeed it would. -- Benji York Senior Software Engineer Zope Corporation
On Mon, Jun 09, 2008 at 05:46:54PM -0400, Benji York wrote:
On Mon, Jun 9, 2008 at 5:41 PM, Marius Gedminas <mgedmin@b4net.lt> wrote:
This implies that buildbot keeps the state somewhere. Then it could also remember the message ID of the failure message and later, when the builder changes from a bad state to a good state, send another message, announcing the end of the failure, with an In-Reply-To header containing the previous message ID, for proper threading.
That would be lovely.
Indeed it would.
I know I'm being cheap, but I was happy that buildbot is actually doing the majority of the work on its own. I should put a ticket to the buildbot people and ask them for that feature probably. Christian -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 7 · fax +49 345 1229889 1 Zope and Plone consulting and development
On Tue, Jun 10, 2008 at 7:20 AM, Christian Theune <ct@gocept.com> wrote:
I know I'm being cheap, but I was happy that buildbot is actually doing the majority of the work on its own.
And very happy I am as well, but don't forget that enhancement requests are the reward for a job well done. ;) -- Benji York Senior Software Engineer Zope Corporation
Christian Theune a écrit :
Hi,
I've watched my buildbot send out notifications for a while and pondered how to make them available to more people. I decided to send them to this list on a 'problem' level.
The 'problem' level sends a message when a builder changes from a good state to a bad state but doesn't send a new message on subsequent failing builds.
For me this has been reasonably quiet over the last two weeks although about 60 builders are broken.
The risk of sending mail here is that a massive build failure can cause about 200 messages being send at once. If anybody knows a better approach with buildbot, please let me know, I'd be happy to implement it.
A way to avoid sending 200 failure messages at once would be to put failures in a queue and send a unique mail containing all failures, when: - the last failure in the queue was more than X minutes ago or - the first failure in the queue was at least Y hours ago Christophe
For everyone else: please have a look at the list of broken builds: http://zopebuildbot.whq.gocept.com/cruise
If a project that you feel responsible for is broken, please check out whats wrong and maybe fix it. If the build environment isn't suitable, I'll be happy to help. This page should show a green bar!
Christian
participants (8)
-
Adam GROSZER -
Andreas Jung -
Benji York -
Christian Theune -
Christophe Combelles -
Fred Drake -
Gary Poster -
Marius Gedminas