[Zope-dev] How to test changes to ZTK packages?

[Wolfgang Schnerring]

* Tres Seaver <tseaver at palladion.com> [2009-06-30 20:41]:
> Jim Fulton wrote:
>> I should know this, but I don't.  What is the recommended way to test  
>> changes to core ZTK packages to mitigate the risk that changes affect  
>> other packages? Is there a page somewhere with instructions?
>> 
>> I tried using using zope.release.  Building the trunk of zope.release  
>> with Python 2.4 and running the tests gives lots of test import errors  
>> and test failures.  (Lots of tests want to import z3c.pt.) The tests  
>> hang when I try to build and run with Python 2.6.
>
> There is a recipe for testing *other* packages which might be affected
> by changes to the package-under-development:
>  http://pypi.python.org/pypi/z3c.recipe.compattest

This recipe was written at the Grok dependency-cleanup sprint in January
with exactly the purpose Jim talks about: testing packages against each
other to make sure changes in one don't adversely affect the others.

I haven't studied zope.release closely yet, but I think testing-wise it
does quite a similar thing, namely running tests for a set of packages.

The difference is that compattest uses either pinned version from
buildout (but doesn't supply its own list of pins), or alternatively
trunk checkouts. Also, it seems to be more lightweight than zope.release
both in concept and in usage (mostly since it only does one thing,
namely running tests, and not other release management stuff like
creating tarballs and uploading them etc.), but I'm biased since I'm one
of the compattest authors.

For the record, the usage is
1. add it to your buildout, minimally like so:
   [compattest]
   recipe = z3c.recipe.compattest
2. run bin/compattest
3. there is no step three. ;-)

> I don't know how to supply the set of dependents to that recipe
> (something in the buildout config file, I guess).

You can specify include and exclude options; the default is to include a
list of zope.* packages that we pulled out of thin air at the sprint,
it'd probably be better to use the KGS as a default instead -- but that
would duplicate even more zope.release functionality.

I think it would be useful to standardize on *one* compatibility testing
method for the ZTK (and then document it).

My instinct would be to separate KGS handling from tarball creation from
testing, that is, remove the test-business from zope.release and use the
compattest recipe instead. (Alternatively, retire compattest and have
zope.release gain the ability to use trunks so it could be used for
continuous integration purposes as well -- but that doesn't feel quite
right, to be honest)

Thoughts?

Wolfgang