[Zope-dev] Re: Test runner: layers, subprocesses, and tear down

[Martin Aspeli]

Benji York wrote:
> I'm working on making the zope.testing test runner run tests in
> parallelized subprocesses.

I have some recent experience parallelising (and distributing across 
machines) test runs. This was in Java, with TestNG and Selenium, but we 
learned some interesting things. We basically cut a 45 minute test run 
to 10 minutes by distributing the tests across three machines, each 
running a full stack (Oracle, JBoss, Firefox) and Selenium Grid. I 
realise you're not trying to do anything quite as complex as that, but a 
parallel test runner ought to be extensible to support distribution 
across nodes in a grid. The main challenge there is to distribute 
deployment of the code to run, and to sync test setup so that all 
environments are identical. I suspect you'll find this out of scope to 
begin with, but I'd keep it in the back of your mind.

You will likely need some way of declaring tests that have to run in 
series. Sometimes that's just for sanity's sake, other times it's a 
requirement due to shared resources.

A nice way to do this is to make it possible to annotate tests to group 
them, and then to be able to declaratively configure some groups as 
serial. Any functional test that uses a shared external resource will 
require this. TestNG supports (as far as I recall):

  - Run all tests (methods) randomly and parallelise

  - Run groups of tests (classes or declaratively specified named 
groups) in parallel, but run tests within the groups sequentially

  - Run all tests in series (i.e. single-threaded)

We should probably use test layers as the main grouping mechanism here. 
If you could declare a layer as "can be run in parallel with other 
layers" or "tests in this layer run in series", that'd be pretty 
powerful. I'm not 100% sure how this works with layers that derive from 
one another, and where you'd have two layers with a shared base class, 
though.

Parallisation can offer huge (!) speed increases, but it can also be 
hard to debug tests. I'd be tempted to let single threaded by the 
default, safe choice, and let people opt into parallisation only when 
they know what they are doing. Most test runs are quite quick anyway.

Test result reporting can be difficult. You'll probably need to collect 
all failures with tracebacks and report at the end. For long running 
test suites, this may not be ideal, since it's helpful to get early 
warning, so if you can find a way to get test output to be atomically 
output, then that'd be nice.

Debugging stuff that happens in parallel with pdb is also tricky. It 
must be easy to turn off parallel running and to run individual tests in 
a single process for each debugging.

To make this work with Selenium grid, we ended up building some 
infrastructure to manage environments (i.e. an allocation of database, 
web server and so on), and locks on those environments. We'd spawn one 
thread for each environment and feed tests to those threads as fast as 
they could run them. Each test run then grabbed an environment on setup, 
executed, and then released the lock for another test.

Oh, and please don't get rid of any tear-down. You'll definitely need it 
one day. Letting environments go dirty is generally troublesome, and 
gets only more difficult when you may have multiple threads trying to 
use those environments at once. I don't know how you've structured this, 
but I'd consider whether one layer could be shared across multiple 
threads/subprocesses, or if it's always a one-to-one thing.

I realise this is somewhat rambling, but I hope it's useful in any case. :)

Cheers,
Martin

-- 
Author of `Professional Plone Development`, a book for developers who
want to work with Plone. See http://martinaspeli.net/plone-book