Hi, I *thought* the test runner had a feature to run tests in randomized order, but I can't find it. Is that something that Zope 2 had and that was never ported to zope.testing? I couldn't find in the history either. /me is confused. -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
Hi Christian, On Fri, Oct 2, 2009 at 2:00 PM, Christian Theune <ct@gocept.com> wrote:
Hi,
I *thought* the test runner had a feature to run tests in randomized order, but I can't find it. Is that something that Zope 2 had and that was never ported to zope.testing?
I couldn't find in the history either.
/me is confused.
I was about to send the same mail today, but I found out your mail before. I didn't find anything relative to random tests in Zope 2 (but I'm not familiar at all with Zope 2, so I'm pretty sure I could have missed something), and I ended up writing a new feature for zope.testing's testrunner. I made a branch 'shuffle-tests', available on the Mercurial repository at http://bitbucket.org/multani/zope.testing-shuffle/ , which adds two new options: * --shuffle to randomize the order of the tests in a layer (layers are not yet randomized, I didn't look at it yet); * --shuffle-seed XXX, to initialize the RNG with a specific value, which allows to reproduce a specific randomization of tests (inspired by this thread http://lists.idyll.org/pipermail/testing-in-python/2009-April/001812.html ) It's not yet ready-to-commit: it still misses tests (tricky to write) and more documentation. However, feedbacks are welcome! Cheers, Jonathan
Hi, On 10/08/2009 03:28 PM, Jonathan Ballet wrote:
Hi Christian,
On Fri, Oct 2, 2009 at 2:00 PM, Christian Theune <ct@gocept.com> wrote:
Hi,
I *thought* the test runner had a feature to run tests in randomized order, but I can't find it. Is that something that Zope 2 had and that was never ported to zope.testing?
I couldn't find in the history either.
/me is confused.
I was about to send the same mail today, but I found out your mail before.
I didn't find anything relative to random tests in Zope 2 (but I'm not familiar at all with Zope 2, so I'm pretty sure I could have missed something), and I ended up writing a new feature for zope.testing's testrunner.
I made a branch 'shuffle-tests', available on the Mercurial repository at http://bitbucket.org/multani/zope.testing-shuffle/ , which adds two new options:
* --shuffle to randomize the order of the tests in a layer (layers are not yet randomized, I didn't look at it yet); * --shuffle-seed XXX, to initialize the RNG with a specific value, which allows to reproduce a specific randomization of tests (inspired by this thread http://lists.idyll.org/pipermail/testing-in-python/2009-April/001812.html )
Sounds right. It could even be folded into a single option: --shuffle[=XXX]
It's not yet ready-to-commit: it still misses tests (tricky to write) and more documentation.
That's what working branches are for. You do have commit rights, don't you? It's not that tricky to test: seed the RNG from outside the tests.
However, feedbacks are welcome!
Thanks for the work! Christian -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
Hey, On Fri, Oct 9, 2009 at 2:33 PM, Christian Theune <ct@gocept.com> wrote:
* --shuffle to randomize the order of the tests in a layer (layers are not yet randomized, I didn't look at it yet); * --shuffle-seed XXX, to initialize the RNG with a specific value, which allows to reproduce a specific randomization of tests (inspired by this thread http://lists.idyll.org/pipermail/testing-in-python/2009-April/001812.html )
Sounds right. It could even be folded into a single option:
--shuffle[=XXX]
I'm not sure how to do that, considering those use cases: * the user doesn't want to shuffle its tests and so, doesn't set --shuffle * the user want to shuffle its tests, without specifying an order, and use --shuffle * the user want to shuffle its tests in specific order, and use --shuffle=42 If I want this option to have an optional value, I think I must set a default value to the option. But then, how to make the distinction between the two first cases? I might miss something I don't know in option parsing, but I didn't find anything...
It's not yet ready-to-commit: it still misses tests (tricky to write) and more documentation.
That's what working branches are for. You do have commit rights, don't you?
I don't (personally) have commit rights, but my boss (sdouche) has. I'll see with him. Thanks for your advices! Jonathan
On Fri, Oct 9, 2009 at 8:51 AM, Jonathan Ballet <jonathan.ballet@securactive.net> wrote:
If I want this option to have an optional value, I think I must set a default value to the option. But then, how to make the distinction between the two first cases? I might miss something I don't know in option parsing, but I didn't find anything...
I'd try the same approach as default options in Python functions. Make the default None (which isn't a valid seed) and then you can tell the difference. -- Benji York Senior Software Engineer Zope Corporation
Hi, On Fri, Oct 9, 2009 at 3:30 PM, Benji York <benji@zope.com> wrote:
On Fri, Oct 9, 2009 at 8:51 AM, Jonathan Ballet <jonathan.ballet@securactive.net> wrote:
If I want this option to have an optional value, I think I must set a default value to the option. But then, how to make the distinction between the two first cases? I might miss something I don't know in option parsing, but I didn't find anything...
I'd try the same approach as default options in Python functions. Make the default None (which isn't a valid seed) and then you can tell the difference.
That's what I originally though, but then, how can I make the difference between 'the user has not set --shuffle' (the option's value is then None), and 'the user has set the --shuffle option, but without specifying a seed' (then the option's value is None again)? Jonathan
On Fri, Oct 09, 2009 at 02:33:02PM +0200, Christian Theune wrote:
On 10/08/2009 03:28 PM, Jonathan Ballet wrote:
I made a branch 'shuffle-tests', available on the Mercurial repository at http://bitbucket.org/multani/zope.testing-shuffle/ , which adds two new options:
* --shuffle to randomize the order of the tests in a layer (layers are not yet randomized, I didn't look at it yet); * --shuffle-seed XXX, to initialize the RNG with a specific value, which allows to reproduce a specific randomization of tests (inspired by this thread http://lists.idyll.org/pipermail/testing-in-python/2009-April/001812.html )
Sounds right. It could even be folded into a single option:
--shuffle[=XXX]
I doubt that. Ordinarily with the GNU long option style these two are equivalent: --shuffle=42 --shuffle 42 but there's no way to tell the second variant from a seed-less --shuffle followed by an non-option argument. Marius Gedminas -- http://pov.lt/ -- Zope 3 consulting and development
On Fri, Oct 9, 2009 at 14:33, Christian Theune <ct@gocept.com> wrote: Hi Christian.
That's what working branches are for.
Done : http://svn.zope.org/zope.testing/branches/sdouche-shuffle/ -- Sebastien Douche <sdouche@gmail.com> Twitter: http://bit.ly/afkrK (agile, python, open source)
Hello there, as I mentioned by Sebastien, the patch has been commited into the sdouche-shuffle branch. It now has tests and a bit of documentation too. Regarding CLI arguments, I'm not sure this could be changed like Christian proposed. If you know how to do it, I'm listening, 'cause I don't know! :) I have one more thing I would like to discuss: how should this feature behave with the --repeat feature? Currently, the shuffling is done before the repeat feature. This means that tests will *not* get reordered for each loop (just once, at the very beginning of tests launching). I know how to do it, so that for each loop, tests will get a different order (basically, I would propose to wrap the __iter__() method of the TestSuite used to launch tests, and to shuffle here): class ShuffledTestSuite(unittest.TestSuite): def __iter__(self): tests = list(super(Shuffled, self)) seed = long(time.time() * 256) shuffler = random.Random() shuffler.shuffle(tests) return iter(tests) However, using this means we would loose the seed printing (which seems very convenient to me). I'm not sure if we could get it and print it for each loop... Reviews, comments and feedbacks are welcome! Cheers, Jonathan
On 10/19/2009 11:57 AM, Jonathan Ballet wrote:
Hello there,
as I mentioned by Sebastien, the patch has been commited into the sdouche-shuffle branch. It now has tests and a bit of documentation too.
Regarding CLI arguments, I'm not sure this could be changed like Christian proposed. If you know how to do it, I'm listening, 'cause I don't know! :)
I have one more thing I would like to discuss: how should this feature behave with the --repeat feature? Currently, the shuffling is done before the repeat feature. This means that tests will *not* get reordered for each loop (just once, at the very beginning of tests launching). I know how to do it, so that for each loop, tests will get a different order (basically, I would propose to wrap the __iter__() method of the TestSuite used to launch tests, and to shuffle here):
class ShuffledTestSuite(unittest.TestSuite): def __iter__(self): tests = list(super(Shuffled, self)) seed = long(time.time() * 256) shuffler = random.Random() shuffler.shuffle(tests) return iter(tests)
However, using this means we would loose the seed printing (which seems very convenient to me). I'm not sure if we could get it and print it for each loop...
I'd say leave it as is for now. I think it's more important to be able to seed (for the sake of reproduction, so btw: do you log the seed if you generated one?) externally than to reorder during each repeated run. Also, we can add that later on in case you figure out a way how to do it. ;) Christia -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
Hello Christian, On Tue, Nov 3, 2009 at 11:07 AM, Christian Theune <ct@gocept.com> wrote:
[...] btw: do you log the seed if you generated one?)
Yep, the seed is displayed at the end of tests, so that it can be reused to reproduce a failing test. $ ./bin/test --shuffle -cvvv Running tests at level 1 Running zope.testing.testrunner.layer.UnitTests tests: Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds. ... Ran 257 tests with 0 failures and 0 errors in 21.908 seconds. Tests were shuffled using seed number 321854366611 I already find it *very* useful ... Cheers, Jonathan
Hello, On Tue, Nov 3, 2009 at 11:21 AM, Jonathan Ballet <jonathan.ballet@securactive.net> wrote:
Hello Christian,
On Tue, Nov 3, 2009 at 11:07 AM, Christian Theune <ct@gocept.com> wrote:
[...] btw: do you log the seed if you generated one?)
Yep, the seed is displayed at the end of tests, so that it can be reused to reproduce a failing test.
$ ./bin/test --shuffle -cvvv Running tests at level 1 Running zope.testing.testrunner.layer.UnitTests tests: Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds. ... Ran 257 tests with 0 failures and 0 errors in 21.908 seconds. Tests were shuffled using seed number 321854366611
Any thoughts on this feature? Did someone try it? Is there any possibilities to merge this into trunk and to release a new version with this? Cheers, Jonathan
Hi there, On 11/23/2009 10:27 AM, Jonathan Ballet wrote:
Hello,
On Tue, Nov 3, 2009 at 11:21 AM, Jonathan Ballet <jonathan.ballet@securactive.net> wrote:
Hello Christian,
On Tue, Nov 3, 2009 at 11:07 AM, Christian Theune<ct@gocept.com> wrote:
[...] btw: do you log the seed if you generated one?)
Yep, the seed is displayed at the end of tests, so that it can be reused to reproduce a failing test.
$ ./bin/test --shuffle -cvvv Running tests at level 1 Running zope.testing.testrunner.layer.UnitTests tests: Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds. ... Ran 257 tests with 0 failures and 0 errors in 21.908 seconds. Tests were shuffled using seed number 321854366611
Any thoughts on this feature? Did someone try it? Is there any possibilities to merge this into trunk and to release a new version with this?
It's still on my todo list and I'll get to it at some point ... maybe even during this week. ;) Christian -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 10/16/2009 06:59 PM, Sebastien Douche wrote: | On Fri, Oct 9, 2009 at 14:33, Christian Theune<ct@gocept.com> wrote: | | Hi Christian. | |> That's what working branches are for. | | Done : | http://svn.zope.org/zope.testing/branches/sdouche-shuffle/ Reviewed and merged. I made some minor textual changes, otherwise that code was fine. Christian - -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAksrerUACgkQdUt9X/gknwIUXwCgqnT4K11BB0H8R/1uKNOM3toz uX4An3XKwn+SDAXfnGFm77IyFlAFxJf/ =9VoN -----END PGP SIGNATURE-----
Hi, On Fri, Dec 18, 2009 at 1:51 PM, Christian Theune <ct@gocept.com> wrote:
Reviewed and merged.
I made some minor textual changes, otherwise that code was fine.
Thank you Christian! BTW, the Pypi page for zope.testing is a bit broken currently (the HTML has not been generated), can you fix this too? Regards, Jonathan
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi, On 12/18/2009 02:49 PM, Jonathan Ballet wrote: | Hi, | | On Fri, Dec 18, 2009 at 1:51 PM, Christian Theune<ct@gocept.com> wrote: |> |> Reviewed and merged. |> |> I made some minor textual changes, otherwise that code was fine. | | Thank you Christian! | BTW, the Pypi page for zope.testing is a bit broken currently (the | HTML has not been generated), can you fix this too? Gnah. I stared at this for a while and didn't find any fault. Turns out it probably was a glitch when registering that version with PyPI. I re-registered the 3.8.6 package and the page shows fine now. Christian - -- Christian Theune · ct@gocept.com gocept gmbh & co. kg · forsterstraße 29 · 06112 halle (saale) · germany http://gocept.com · tel +49 345 1229889 0 · fax +49 345 1229889 1 Zope and Plone consulting and development -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.9 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAksx7CYACgkQdUt9X/gknwJ4QACgh6dH8M4a1/3uNLAc16PGhi5n gBQAoJnZCmAFLWttyhmQt1LfkTp7z5ct =G0// -----END PGP SIGNATURE-----
participants (5)
-
Benji York -
Christian Theune -
Jonathan Ballet -
Marius Gedminas -
Sebastien Douche