[Zope-dev] Re: Test runner: layers, subprocesses, and tear down

[Benji York]

On Fri, Jul 4, 2008 at 4:49 PM, Marius Gedminas <marius at gedmin.as> wrote:

> I tried this in a Zope 3.4 checkout I had handy on a Core 2 Duo machine
> (1.8 GHz, running 64-bit Ubuntu Hardy).  One test module could not be
> loaded, which explains the slightly lower number of tests reported:

> Here are the results:
>
>                        time                                # tests
>                        real        user        system      reported
> old test runner         3m16.033s   2m44.670s   0m2.832s    6895
> zope.testing trunk      2m27.816s   1m58.971s   0m2.196s    6890
> new test runner -j0     2m37.322s   2m5.808s    0m2.944s    6890
> new test runner -j1     2m32.249s   1m58.847s   0m2.652s    6890
> new test runner -j2     2m22.287s   3m51.214s   0m13.457s   584
> new test runner -j3     2m20.560s   3m46.990s   0m12.613s   584
> new test runner -j4     2m30.026s   3m43.198s   0m13.241s   584

I'm really curious why you didn't see more improvement.

> At the end of the experiment I discovered that I have CPU frequency
> scaling enabled.  It only scales down to 1.6 GHz and quickly jumps back
> up to 1.87 GHz.
>
> I find the speedup by switching to a modern test runner somewhat
> unexpected.  Can those 5 missing tests really account for 45 seconds?
>
> Zope 3 appears to be composed of a multitude of small tests.  If my
> numbers are correct, the advantage of using both CPU cores is almost
> completely negated by the extra bookkeeping that the test runner has to
> do.

There's no appreciable bookkeeping for the parallelization, so I don't
know where the CPU time is going.

> Visual ogling of my CPU usage applet shows that -j0/1 use only one
> CPU, while -j2 and above use only one CPU for the first test layer
> (zope.app.apidoc.testing.APIDocLayer) and then use both CPUs for the
> rest.  Bug?

Long story short: it made the changes to the code much less invasive to
do that way.

> The total number of tests is misreported when you have -jN with N > 1.

I haven't seen that symptom, but I'll try to reproduce it by running the
3.4 tests.

> "Test-module import failures" is printed several times.  test -j4
> printed that message 37 times!  test -j1 only did it once.  -j2 and -j3
> also did that a bit often (once per layer?)

Interesting.  I'll investigate.

> As far as I can understand, the granularity of the test distribution to
> CPUs is a test layer?

Right.

> If so, that's rather unfortunate for my application, which has only
> two layers (unit and functional).  Especially given the quirk that the
> first test layer is run on one CPU while the other idles.

If the need is great enough, you can always introduce an arbitrary
number of layers.  Also, once this code is working properly, I (or
someone else) might look into changing the granularity to the level of
individual tests.
-- 
Benji York
Senior Software Engineer
Zope Corporation