[Zope] Testing products: Testing all in one batch

[Paul Winkler]

Has anybody set up a batch job to test all installed Products
in a zope instance?

Seems like the only *useful* way I can get it to run is with a loop that
does "bin/zopectl test" once for each subdirectory of Products
that I actually care about, taking care to ignore problematic
third-party products.  Which is OK I guess, but I'd prefer not
to have to skip anything.

I think you *should* be able to simply do "./bin/zopectl test"
and that will run everything, but with real-world products this
seems to be a huge mess that spews a lot of red herring errors:

- Some otherwise excellent third-party products have tests with
  dependencies that the product itself does not have, and which
  I do not want to install. (I will contact the author(s) privately.)

- Some third-party products (e.g. CMF) ship with tests that just plain
  fail.
  If I install CMF 1.4.8 and nothing else, and run ./bin/zopectl test,
  I get 1 error and 1 failure.  This is a bit surprising from a stable
  release, I'd expect not to get any... will forward those to the CMF
  list and follow up there, if there are not already issues filed for
  these.

- Some of my own Product tests run fine in isolation but break when
  run alongside other installed product tests.
  That's its own issue which I will take up in a separate thread.

I guess my overall questions are....

- Does this jibe with others' experience?

- Is there a better (or best) way to run all my Product tests?

- *Should* we expect that ./bin/zopectl test will run all installed
Products tests, and if so, what can we do as a community to get
authors to fix their products?

p.s. you may notice a series of messages from me re. testing...
i'm trying to sort out a bunch of issues at once, and want to
try to keep the issues separate.

-- 
Paul Winkler
http://www.slinkp.com