[Zope-dev] How to test changes to ZTK packages?

[Jim Fulton]

On Jul 1, 2009, at 1:08 AM, Wolfgang Schnerring wrote:

> * Tres Seaver <tseaver at palladion.com> [2009-06-30 20:41]:
>> Jim Fulton wrote:
>>> I should know this, but I don't.  What is the recommended way to  
>>> test
>>> changes to core ZTK packages to mitigate the risk that changes  
>>> affect
>>> other packages? Is there a page somewhere with instructions?
>>>
>>> I tried using using zope.release.  Building the trunk of  
>>> zope.release
>>> with Python 2.4 and running the tests gives lots of test import  
>>> errors
>>> and test failures.  (Lots of tests want to import z3c.pt.) The tests
>>> hang when I try to build and run with Python 2.6.
>>
>> There is a recipe for testing *other* packages which might be  
>> affected
>> by changes to the package-under-development:
>> http://pypi.python.org/pypi/z3c.recipe.compattest
>
> This recipe was written at the Grok dependency-cleanup sprint in  
> January
> with exactly the purpose Jim talks about: testing packages against  
> each
> other to make sure changes in one don't adversely affect the others.
>
> I haven't studied zope.release closely yet, but I think testing-wise  
> it
> does quite a similar thing, namely running tests for a set of  
> packages.
>
> The difference is that compattest uses either pinned version from
> buildout (but doesn't supply its own list of pins), or alternatively
> trunk checkouts.

So it provides no good configuration to test against. This is a fatal  
flaw IMO.


> Also, it seems to be more lightweight than zope.release
> both in concept and in usage (mostly since it only does one thing,
> namely running tests, and not other release management stuff like
> creating tarballs and uploading them etc.), but I'm biased since I'm  
> one
> of the compattest authors.
>
> For the record, the usage is
> 1. add it to your buildout, minimally like so:
>   [compattest]
>   recipe = z3c.recipe.compattest
> 2. run bin/compattest
> 3. there is no step three. ;-)

I tried this with the current trunk of zope.app.publisher, which I'm  
about to modify.  I added a compattest part:

[compattest]
recipe = z3c.recipe.compattest

I didn't get bin/compattest, but I did get bin/test_compattest.  When  
I run this, I get lots and lots of errors. :(  I could attach them,  
but what's the point?

>> I don't know how to supply the set of dependents to that recipe
>> (something in the buildout config file, I guess).
>
> You can specify include and exclude options; the default is to  
> include a
> list of zope.* packages that we pulled out of thin air at the sprint,
> it'd probably be better to use the KGS as a default instead -- but  
> that
> would duplicate even more zope.release functionality.
>
> I think it would be useful to standardize on *one* compatibility  
> testing
> method for the ZTK (and then document it).
>
> My instinct would be to separate KGS handling from tarball creation  
> from
> testing, that is, remove the test-business from zope.release and use  
> the
> compattest recipe instead. (Alternatively, retire compattest and have
> zope.release gain the ability to use trunks so it could be used for
> continuous integration purposes as well -- but that doesn't feel quite
> right, to be honest)
>
> Thoughts?


I'm kind of stuck at this point. The KGS approach seems most promising  
because it is based on the idea of a known working configuration.  It  
is still baffling to me that people checked version numbers into the  
trunk of zope.release that don't pass tests.  (Many of the packages  
there don't even run their tests.)

Jim

--
Jim Fulton
Zope Corporation