Hi Lennart, On Mon, Apr 19, 2010 at 2:42 PM, Jonathan Ballet <jonathan.ballet@securactive.net> wrote:

Hi,

On Mon, Apr 19, 2010 at 1:03 PM, Lennart Regebro <regebro@gmail.com> wrote:

...

OK, I made a branch of zope.testing that doesn't include the testrunner. That was easy peasy.

http://svn.zope.org/zope.testing/branches/regebro-notestrunner/

And I made a zope.testrunner:

http://svn.zope.org/zope.testrunner/trunk

That was pretty easy too. Except one thing: The shuffle feature shuffles things differently in zope.testrunner compared to in zope.testing.testrunner, which seems to be because it finds the layers in a different order.

I took some time to look at this yesterday, and it seems you were right: the order should have changed since the layers now have a different name. I can't speak for other people who are using the feature, but this will not affect us very much. We use the shuffle seed feature for two cases: * we want to reproduce on a developer computer a test run which has failed on the Buildbot; * while reviewing someone's code, if tests are failing on a computer but not on another, we want to provide a way for tests to fail the same way on both computers, by "pinning" the order. Both cases have a short life time (we don't need to reproduce a particular order from X months ago), so this change is OK for us. Regards, Jonathan

On Wed, Apr 21, 2010 at 13:47, Marius Gedminas <marius@gedmin.as> wrote:

Also see https://bugs.launchpad.net/zope.testing/+bug/497871: there's consensus that unit tests should always be the first layer to run, so the layer name shouldn't influence the ordering.

The layers run in the same order, but the *discovery* of the layers means they get *shuffled* in a different order, which means the result of the shuffle is different, because the random shuffle is not reseeded between each layer. I hoped that the different discovery order was just for zope.testrunner vs zope.testing.testrunner, but from Jonathans answer above it seems not. -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64

On Tue, Apr 20, 2010 at 18:11, Christian Theune <ct@gocept.com> wrote:

The intention is to make a few successive runs have the same ordering. I wouldn't complain if a switch to a different test runner version changed ordering for a seed as long as the order then remains stable.

Right, that's what I thought. And I think it actually will not change the ordering, I think it only did that for the testrunners own tests, as a result of the reorganization, but I'm not *sure*. :) -- Lennart Regebro: Python, Zope, Plone, Grok http://regebro.wordpress.com/ +33 661 58 14 64