[Zope-dev] How to test changes to ZTK packages?

[Jim Fulton]

On Jul 1, 2009, at 1:08 AM, Wolfgang Schnerring wrote:

> * Tres Seaver <tseaver at palladion.com> [2009-06-30 20:41]:
>> Jim Fulton wrote:
>>> I should know this, but I don't.  What is the recommended way to  
>>> test
>>> changes to core ZTK packages to mitigate the risk that changes  
>>> affect
>>> other packages? Is there a page somewhere with instructions?
>>>
>>> I tried using using zope.release.  Building the trunk of  
>>> zope.release
>>> with Python 2.4 and running the tests gives lots of test import  
>>> errors
>>> and test failures.  (Lots of tests want to import z3c.pt.) The tests
>>> hang when I try to build and run with Python 2.6.
>>
>> There is a recipe for testing *other* packages which might be  
>> affected
>> by changes to the package-under-development:
>> http://pypi.python.org/pypi/z3c.recipe.compattest
>
> This recipe was written at the Grok dependency-cleanup sprint in  
> January
> with exactly the purpose Jim talks about: testing packages against  
> each
> other to make sure changes in one don't adversely affect the others.
>
> I haven't studied zope.release closely yet, but I think testing-wise  
> it
> does quite a similar thing, namely running tests for a set of  
> packages.
>
> The difference is that compattest uses either pinned version from
> buildout (but doesn't supply its own list of pins), or alternatively
> trunk checkouts. Also, it seems to be more lightweight than  
> zope.release
> both in concept and in usage (mostly since it only does one thing,
> namely running tests, and not other release management stuff like
> creating tarballs and uploading them etc.), but I'm biased since I'm  
> one
> of the compattest authors.
>
> For the record, the usage is
> 1. add it to your buildout, minimally like so:
>   [compattest]
>   recipe = z3c.recipe.compattest
> 2. run bin/compattest
> 3. there is no step three. ;-)
>
>> I don't know how to supply the set of dependents to that recipe
>> (something in the buildout config file, I guess).
>
> You can specify include and exclude options; the default is to  
> include a
> list of zope.* packages that we pulled out of thin air at the sprint,
> it'd probably be better to use the KGS as a default instead -- but  
> that
> would duplicate even more zope.release functionality.

Where is this list? In the recipe?

> I think it would be useful to standardize on *one* compatibility  
> testing
> method for the ZTK (and then document it).

Yes.

> My instinct would be to separate KGS handling from tarball creation  
> from
> testing, that is, remove the test-business from zope.release and use  
> the
> compattest recipe instead. (Alternatively, retire compattest and have
> zope.release gain the ability to use trunks so it could be used for
> continuous integration purposes as well -- but that doesn't feel quite
> right, to be honest)
>
> Thoughts?


I agree we need one way to do this. I think the KGS concept is right.   
I think there should be a known good configuration of packages and a  
way to evolve it.  The configuration should be changed only after  
testing changes.  The configuration needs to include Python versions  
that it is tested with.  Beyond that, I don't care what the process is  
as long as it is documented.

Jim

--
Jim Fulton
Zope Corporation