Stefane Fermigier wrote:

Sidnei da Silva wrote:

...

On Sun, Aug 21, 2005 at 10:20:28AM +0200, Stefane Fermigier wrote: | I have added support for coverage analysis using coverage.py from Gareth | Rees and Ned Batchelder, as well as support for profiling using either the | profile or hotshot modules from the Python standard library, to Zope 2's | test runner test.py. | | Justification: | | - coverage.py will let you interactively focus on the modules you want to | check for coverage. | | - profile is slow to collect data but quick for analysis. | | - hotshot is fast when collecting but very slow for analysis. On the other | hand, one can feed the great KCacheGrind tool with hotshot data after some | transformation and this is the best way I have found so far for | interactively exploring profile data.

Eerm, the test runner in Zope 2 is the same from Zope 3. The version in Zope 3 X3.0 has profile support using hotshot and coverage support using the trace.py module, but the version in Zope 2 is earlier than that.

How does that differ from what you're proposing?

1. I haven't found hotshot support in either Zope 2.8.1 nor the TRUNK.

2. Has I wrote, coverage.py can be used interactively (at least, from the command line) to focus on whichever package you are working on at the moment. You can also collect data from several runs, which is useful for us because we have to test each CPS package in a different run.

3. I understand that trace.py and coverage.py have some overlap, and should probably be merged into one great and up to date coverage tool. But before it is done, I find useful to have both tools at our disposal.

4. Same for hotshot and profile. Both are useful.

I don't think Sidnei was questioning the value of these, but just pointing out that another version of the test runner already had some of this. I'll note that I'm working on a newer test runner that I hope to use in Zope 2.9 and 3.2. The new test runner is a nearly complete rewrite to provide: - A more flexible test runner that can be used for a variety of projects. The current test runner has been forked for ZODB, Zope 3, and Zope 2. That's why the Zope 3 version has features that are lacking in the Zope 2 version. - Support for "layers" of tests, so that it can handle unit tests and functional tests. - A slightly better UI. - Tests (of the test runner itself :) See: http://svn.zope.org/zope.testing/trunk/src/zope/testing/testrunner.txt?view=... http://svn.zope.org/zope.testing/trunk/src/zope/testing/testrunner.py?view=l... I'd like to include coverage and profiling support. (There is partial support now, but untested.) Wanna help? Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Stuart Bishop wrote:

Jim Fulton wrote:

...

I'll note that I'm working on a newer test runner that I hope to use in Zope 2.9 and 3.2. The new test runner is a nearly complete rewrite to provide:

- A more flexible test runner that can be used for a variety of projects. The current test runner has been forked for ZODB, Zope 3, and Zope 2. That's why the Zope 3 version has features that are lacking in the Zope 2 version.

- Support for "layers" of tests, so that it can handle unit tests and functional tests.

- A slightly better UI.

- Tests (of the test runner itself :)

See:

http://svn.zope.org/zope.testing/trunk/src/zope/testing/testrunner.txt?view=...

http://svn.zope.org/zope.testing/trunk/src/zope/testing/testrunner.py?view=l...

Hi Jim.

I've been looking over this - fixing tests seems to take up a significant amount of our time, so I might have some interesting use cases.

A large proportion of our tests use a relational database. Some of them want an empty database, some of them want just the schema created but no data, some of them want the schema created and the data. Some of them need the component architecture, and some of them don't. Some of them need one or more twisted servers running, some of them don't.

Note that we mix and match. We have 4 different types of database fixture (none, empty, schema, populated), 2 different types of database connection mechanisms (psycopgda, psycopg), 2 types of CA fixture (none, loaded), and (currently) 4 states of external daemons needed. If we were to arrange this in layers, it would take 56 different layers, and this will double every time we add a new daemon, or add more database templates (e.g. fat for lots of sample data to go with the existing thin).

As a way of supporting this better, instead of specifying a layer a test could specify the list of resources it needs:

import testresources as r

class FooTest(unittest.TestCase): resources = [r.LaunchpadDb, r.Librarian, r.Component] [...]

class BarTest(unittest.TestCase): resources = [r.EmptyDb]

class BazTest(unittest.TestCase): resources = [r.LaunchpadDb, r.Librarian]

This is pretty much how layers work. Layers can be arranged in a DAG (much like a traditional multiple-inheritence class graph). So, you can model each resource as a layer and specific combinations of resources as layers. The test runner will attempt to run the layers in an order than minimizes set-up and tear-down of layers. ...

Some other nice things could be done with the resources:

- If the setUp raises NotImplementedError (or whatever), tests using this resource are skipped (and reported as skipped). This nicely handles tests that should only be run in particular environments (Win32, Internet connection, python.net installed etc.)

That's a good idea.

- If the setUp raises another exception, all tests using this resource fail. The common case we see is 'database in use', where PostgreSQL does not let us destroy or use as a template a database that has open connections to it. Also useful for general sanity checking of the environment - no point running the tests if we know they are going to fail or have skewed results.

Good.

- A resource should have a pretest and posttest hooks. pretest is used for lightweight resource specific initialization (e.g. setUp creates a fresh database from a dump and pretest initializes the connection pool). posttest can be used to ensure tests cleaned up properly or other housekeeping (e.g. issuing a rollback). This could also apply to layers in the current environment. This eliminates tedious boilerplate from testcases.

Ah, so the layer specifies additional per-test setUp and tearDown that is used in addition to the tests's own setUp and tearDown. This sounds reasonable.

- A resource could provide useful data to the test runner. For example, if a resource says it doesn't use or lock any shared system resources, the test runner could decide to run tests in parallel.

Although a less blue sky use would be specifying a dependancy on another resource.

This is handled by layers now. Layers have __bases__ -- layers are build on other layers. That's why they are caled layers. :)

On another note, enforcing isolation of tests has been a continuous problem for us. For example, a developer registering a utility or otherwise mucking around with the global environment and forgetting to reset things in tearDown. This goes unnoticed for a while, and other tests get written that actually depend on this corruption. But at some point, the order the tests are run changes for some reason and suddenly test 500 starts failing. It turns out the global state has been screwed, and you have the fun task of tracking down which of the proceeding 499 tests screwed it. I think this is a use case for some sort of global posttest hook.

How so?

Perhaps this would be best done by allowing people to write wrappers around the one-true-testrunner?

or we could simply provide such a hook, if it's needed. I think this sort of thing is better handled with layers.

This seems to be the simplest way of allowing customization of the test runner:

def pretest(...): [...] def posttest(...): [...] if __name__ == '__main__': zope.test.testrunner.main(pretest=pretest,posttest=posttest)

Other policy could also be configured - e.g. 'Run these tests or tests using this resource first. If any failures, don't bother running any more'. Or 'Stop running tests after 1 failure'.

without a lot more thought and details, it's not at all clear that the hooks you've specialized would provide the simplest way to do this.

These sorts of policies are important for us as we run our tests in an automated environment (we can't commit to our trunk. Instead, we send a request to a daemon which runs the test suite and commits on our behalf if the tests all pass).

Hm, seems rather restrictive... I guess making the new test runner class-based would more easily allow this sort of customization.

Our full test suite currently takes 45 minutes to run and it is becoming an issue.

Hm, I can see why you'd like to parallelize things. Of course, this only helps you if you have enough hardware to benefit from the parallelization. The test runner is already prepared to run layers in separate processes if they can't be torn down. I don't think it would take much to have an option to run the layers in separate processes, or to arrange the layers into sets run as separate processes. Of course, because the new test runner makes it easy to select test subsets in various ways, you could probably arrange the parallelization with a controller script.

We need to speed them up, determine slow tests in need of pruning or optimization, short circuit test runs and reduce test suite maintenance. So I should be able to get time to help (although I need to look closer at the SchoolTool and py.test runners to see if they are closer to what we need).

k Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org

Stuart Bishop wrote:

Jim Fulton wrote:

...

...

A large proportion of our tests use a relational database. Some of them want an empty database, some of them want just the schema created but no data, some of them want the schema created and the data. Some of them need the component architecture, and some of them don't. Some of them need one or more twisted servers running, some of them don't.

Note that we mix and match. We have 4 different types of database fixture (none, empty, schema, populated), 2 different types of database connection mechanisms (psycopgda, psycopg), 2 types of CA fixture (none, loaded), and (currently) 4 states of external daemons needed. If we were to arrange this in layers, it would take 56 different layers, and this will double every time we add a new daemon, or add more database templates (e.g. fat for lots of sample data to go with the existing thin).

As a way of supporting this better, instead of specifying a layer a test could specify the list of resources it needs:

import testresources as r

class FooTest(unittest.TestCase): resources = [r.LaunchpadDb, r.Librarian, r.Component] [...]

class BarTest(unittest.TestCase): resources = [r.EmptyDb]

class BazTest(unittest.TestCase): resources = [r.LaunchpadDb, r.Librarian]

This is pretty much how layers work. Layers can be arranged in a DAG (much like a traditional multiple-inheritence class graph). So, you can model each resource as a layer and specific combinations of resources as layers. The test runner will attempt to run the layers in an order than minimizes set-up and tear-down of layers.

So my example could be modeled using layers like:

import layers as l

class FooLayer(l.LaunchpadDb, l.Librarian, l.Component): pass class FooTest(unittest.TestCase): layer = 'FooLayer' [...]

class BarLayer(l.LaunchpadDb, l.Librarian, l.Component): pass class BarTest(unitest.TestCase): layer = 'BarLayer' [...]

class BazLayer(l.LaunchpadDb, l.Librarian): pass class BazTest(unittest.TestCase): layer = 'BazLayer' [...]

In general I would need to define a layer for each test case (because the number of combinations make it impractical to explode all the possible combinations into a tree of layers, if for no other reason than naming them).

That's too bad. Perhaps layers don't fit your need then.

If I tell the test runner to run all the tests, will the LaunchpadDb, Librarian and Component layers each be initialized just once?

If all of the tests means these 3, then yes.

If I tell the test runner to run the Librarian layer tests, will all three tests be run?

No, no tests will be run. None of the tests are in the librarian layer. They are in layers build on the librarian layer.

What happens if I go and define a new test:

class LibTest(unittest.TestCase): layer = 'l.Librarian' [...]

If I run all the tests, will the Librarian setup/teardown be run once (by running the tests in the order LibTest, BazTest, FooTest, BarTest and initializing the Librarian layer before the LaunchpadDb layer)?

Yes

I expect not, as 'layer' indicates a heirarchy which isn't as useful to me as a set of resources.

I don't follow this.

If layers don't work this way, it might be possible to emulate resources somehow:

If each test *really* has a unique set of resources, then perhaps laters don't fit.

class ResourceTest(unittest.TestCase): @property def layer(self): return type(optimize_order(self.resources))

Howver, optimize_order would need to know about all the other tests so would really be the responsibility of the test runner (so it would need to be customized/overridden), and the test runner would need to support the layer attribute possibly being a class rather than a string.

Layers can be classes. In fact, I typically use classes with class methods for setUp and tearDown.

...

Ah, so the layer specifies additional per-test setUp and tearDown that is used in addition to the tests's own setUp and tearDown. This sounds reasonable.

But what to call them? setUpPerTest? The pretest and posttest names I used are a bit sucky.

<shrug> testSetUp?

...

...

On another note, enforcing isolation of tests has been a continuous problem for us. For example, a developer registering a utility or otherwise mucking around with the global environment and forgetting to reset things in tearDown. This goes unnoticed for a while, and other tests get written that actually depend on this corruption. But at some point, the order the tests are run changes for some reason and suddenly test 500 starts failing. It turns out the global state has been screwed, and you have the fun task of tracking down which of the proceeding 499 tests screwed it. I think this is a use case for some sort of global posttest hook.

How so?

In order to diagnose the problem I describe (which has happened far too often!), you would add a posttest check that is run after each test. The first test that fails due to this check is the culprit.

so your post-test thing would check to make sure there weren't any left-over bits from tests. This makes sense.

I see now though that this could be easily modeled by having a 'global' or 'base' layer in your test suite, and mandate its use by all tests in your application. Or the check could go in a more specific layer if appropriate.

I think it makes more sense to be able to provide a hook. I would be inclined to make this a command-line options. (2 actually). Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org