[Zope] Testing products: Testing all in one batch

[Paul Winkler]

Chris Withers said:
> Paul Winkler wrote:
>> Seems like the only *useful* way I can get it to run is with a loop
>> that does "bin/zopectl test" once for each subdirectory of Products
>> that I actually care about, taking care to ignore problematic
>> third-party products.  Which is OK I guess, but I'd prefer not
>> to have to skip anything.
>
> I have a python script that builds a big command line to test.py of the
> form:
>
> bin/zopectl test Products/Product1|Products/Products2|etc
>
> ...it's not pretty, but it does work, and lets you exclude geb0rken
> products like Archetypes from test runs...

Aha, that's much like what I'm doing except that yours runs in one go
and thus exposes tests that don't clean up after themselves.
Which is a Good Thing if I can just find and fix all the nasty things...

>> - Some third-party products (e.g. CMF) ship with tests that just plain
>>   fail.
>
> Really? That's not been the case for a long time. When trying to test a
> Plohn setup, I found some of the Plohn tests were causing CMF tests to
> fail if the CMF tests were run in the same batch, 'cos they no doubt
> leave lots of crap lying around :-(
>
>>   If I install CMF 1.4.8 and nothing else, and run ./bin/zopectl test,
>> I get 1 error and 1 failure.
>
> Why are you using something so ancient? ;-)

Irrelevant to this discussion, but A) it's hardly ancient,
and B) upgrading to 1.5.x has been postponed indefinitely as it breaks a
lot of old things (some mine, some third-party) and I have not had any
time to devote a day or three to resolving the incompatibilities.
Anyway, as somebody replied to me in zope-cmf list last night,
the breakage is fixed on the 1.4 branch and there will likely be
a 1.4.9 release.

>> - Some of my own Product tests run fine in isolation but break when
>>   run alongside other installed product tests.
>
> Then either your tests or the other tests are leaving turds...

Indeed. I wish it was easier to diagnose and resolve problems like this.
The problem is that tests seem to be run in alphabetical order, I
don't know of a way to force another order, and so I'm left
with adding and removing things from Products to see what minimal
set of products will trigger the error - which is a crude technique
since some products' tests will not run at all unless some other
product is present (e.g. you can't run CMFDefault tests if
CMFCore is not present).

>> - Does this jibe with others' experience?
>
> Yes.

Thanks.  I'm always hoping that things like this are just me doing
something stupid and the problem will go away with some trivial tweak
I've overlooked... sadly, no such luck.

>> - Is there a better (or best) way to run all my Product tests?
>
> If you find it, let me know! :-)

Will do :)


-- 
Paul Winkler
http://www.slinkp.com