Jim Fulton wrote:
On Thu, Aug 6, 2009 at 12:16 PM, Martijn Faassen<faassen@startifact.com> wrote: [snip]
Could you give an example ofa 'test package'? I'm having trouble thinking of what these might be.
Perhaps Grok (or keas ), or some higher-level framework or application built with the framework. The idea is to catch cases where we've changed the APIs in a backward incompatible way.
Ah, I understand better now. I was thinking more of dedicated test packages, but you're talking about libraries and frameworks that use parts of the ZTK, and that we could use to see whether we break anything. [snip]
I propose that any test infrastructure we come up with needs to handle these 3 cases. In any cases, the test infrastructure needs to fix version for all of the categories of packages and there needs to be a formal process (uh, like tests passing :) for updating the configuration. Could you say a bit more about what's motivating this proposal?
There's been way too much breakage lately. We have a number of applications here at ZC that can't be upgraded without a huge amount of effort to figure out what combination of packages actually work together. There have also been backward incompatible changes. We'll deal with those, but aren't going to spend time on that until there's a solid foundation.
We need a known tested configuration of packages, with their dependencies, that are knows to work -- or at least pass their tests. The purpose of my proposal was to try to move this forward.
Understood. Putting the independent pieces back together again in a known-good-set will indeed require work. During the sprint earlier this year we developed z3c.recipe.compattest to help with that, and I believe the KGS toolset also has tools to do this (the difference being compattest runs tests of each package in isolation whereas KGS runs them when all are imported). Basically the primary thing we need right now is a sensible KGS that we can use as a foundation to upgrade towards, instead of a moving target. Regards, Martijn