Re: [Zope-dev] IRC discussion about testing
2009/8/3 Fabio Tranchitella <fabio@tranchitella.it>:
Hi,
Hi, I've added zope-dev. I hope you don't mind.
* 2009-08-03 08:59, Christian Theune wrote:
Cool! Thanks for helping out! I should be online most of the time and I'd be happy to answer questions.
Tonight I spent some time working on this, and I wrote a mock-up of my idea for testing the KGS. I did not want to change anything in compattest, so I created a new recipe called z3c.recipe.kgs. Later on we can merge things back into compattest.
You can get the recipe from here:
svn://svn.zope.org/repos/main/z3c.recipe.kgs/trunk
There are no tests nor documentation, because it is a mock-up and I did not want to spend time on writing documentation for something that maybe is not exactly what we need.
Let's say that you are working on transaction and you want to test your changes against the KGS, you can simply add the following snippet to your buildout.cfg:
[test-kgs] recipe = z3c.recipe.kgs kgs = http://svn.zope.org/*checkout*/zope.release/trunk/releases/controlled-packag...
You can now run buildout, and you will get ./bin/test-kgs-<package> for each package in the KGS, as well as ./bin/test-kgs to run all the tests in all the packages in isolated environments.
The KGS is rather big right now, so I suggest to try it with a subset of the packages using something like:
[test-kgs] recipe = z3c.recipe.kgs kgs = http://svn.zope.org/*checkout*/zope.release/trunk/releases/controlled-packag... packages = zope.component zope.interface zope.event zope.configuration
Now.. this simple recipe allows us to test a local change in a package we are developing on running the tests of all the packages that are part of a KGS. In fact, it reuses the KGS as defined in zope.release, without reinventing the wheel, and it is quite generic.
Ideas? Comments? Is this the right direction?
I think this is the right direction. :) A number of comments below: - I just tried this out. Unfortunately, I'm getting lots of test failures using Python 2.5. That's not the recipe's fault. :) - A bigger problem is that the runner of all tests eventually dies with too many open files on Mac OS X. Here's the last part of the output: ... Running test-kgs-zope.app.intid Running test-kgs-zope.app.keyreference Running test-kgs-zope.app.layers Running test-kgs-zope.app.locales 1Running test-kgs-zope.app.localpermission Running test-kgs-zope.app.locking Traceback (most recent call last): File "bin/test-kgs", line 259, in <module> File "/private/tmp/kgs/zc.recipe.kgs/src/z3c/recipe/kgs/runner.py", line 86, in main File "/private/tmp/kgs/zc.recipe.kgs/src/z3c/recipe/kgs/runner.py", line 33, in start File "/usr/local/python/2.5.1/lib/python2.5/subprocess.py", line 593, in __init__ File "/usr/local/python/2.5.1/lib/python2.5/subprocess.py", line 1002, in _execute_child OSError: [Errno 24] Too many open files - The packages from zope.release don't build for me on Windows because of a dependency on M2Crypto from keas.kmi. Not a big deal at this point, since ultimately, we want to use a smaller set. - The versions specified in the controlled-packages.cfg don't seem to be honored except for the package being tested. For example, ZODB3-3.9.0b2 is used for test-kgs-ZODB3, but 3.9.0b5 is used for everything else. Fortunately, it appears I can work around this using a buildout versions section. Thanks! Jim -- Jim Fulton
On Tue, Aug 11, 2009 at 5:22 PM, Jim Fulton<jim@zope.com> wrote: ...
- The versions specified in the controlled-packages.cfg don't seem to be honored except for the package being tested. For example, ZODB3-3.9.0b2 is used for test-kgs-ZODB3, but 3.9.0b5 is used for everything else. Fortunately, it appears I can work around this using a buildout versions section.
Except that there's a bug in buildout that causes extras to be dropped when versions are specified via a versions section. This causes the extras to be omitted from the test scripts. :( Hopefully, I'll get a fixed buildout release out tomorrow. The good news is that with a hacked buildout and a versions section, I can run tests for all of the packages in http://docs.zope.org/zopetoolkit/about/packages.html except for zope.ucol. I'm getting test failures in 5 packages. I suspect that these are due to missing dependencies. In playing with this today, I'm inclined to think that it would be simpler to use a list of packages in an option to specify the packages to be tested and used a versions section to specify versions, rather than using a controlled-packages configuration file. Jim -- Jim Fulton
* Jim Fulton <jim@zope.com> [2009-08-12 01:36]:
On Tue, Aug 11, 2009 at 5:22 PM, Jim Fulton<jim@zope.com> wrote:
- The versions specified in the controlled-packages.cfg don't seem to be honored except for the package being tested. For example, ZODB3-3.9.0b2 is used for test-kgs-ZODB3, but 3.9.0b5 is used for everything else. Fortunately, it appears I can work around this using a buildout versions section.
In playing with this today, I'm inclined to think that it would be simpler to use a list of packages in an option to specify the packages to be tested and used a versions section to specify versions, rather than using a controlled-packages configuration file.
This doesn't strike me as simpler to *use* (as I said earlier, I'd much prefer a *single* gesture of "use this KGS" to set up both the versions and the list of packages to run tests for, and I think it's worth spending effort to get there), but I'm not sure that's what you meant. What makes you prefer two files instead of one? I'm quite willing to either merge the z3c.recipe.kgs into compattest or abandon the latter (or whatever) and to write a buildout extension to pin the versions using a controlled-packages file, but I only want to do that kind of work if we're reasonably certain that this is what we want. Wolfgang
On Wed, Aug 12, 2009 at 2:31 AM, Wolfgang Schnerring<ws@gocept.com> wrote:
* Jim Fulton <jim@zope.com> [2009-08-12 01:36]:
On Tue, Aug 11, 2009 at 5:22 PM, Jim Fulton<jim@zope.com> wrote:
- The versions specified in the controlled-packages.cfg don't seem to be honored except for the package being tested. For example, ZODB3-3.9.0b2 is used for test-kgs-ZODB3, but 3.9.0b5 is used for everything else. Fortunately, it appears I can work around this using a buildout versions section.
In playing with this today, I'm inclined to think that it would be simpler to use a list of packages in an option to specify the packages to be tested and used a versions section to specify versions, rather than using a controlled-packages configuration file.
This doesn't strike me as simpler to *use* (as I said earlier, I'd much prefer a *single* gesture of "use this KGS" to set up both the versions and the list of packages to run tests for, and I think it's worth spending effort to get there), but I'm not sure that's what you meant. What makes you prefer two files instead of one?
I didn't say anything about 2 files. I said I prefered a parts list in a single option in combination with a standard buildout versions section. There are a number of reasons for this: - We'll need the versions section to consume the KGS. That is, given a KGS, you'll often want to use the versions in other buildouts to limit them to the known good configuration. - I think a parts list in an option will be easier to control. For example, you will be able to use the standard buildout option incrementing and decrementating machinery to modify it. - After using it for a day, I find the controlled-packages syntax to be a bit heavy. Of course, it didn't help that I still had to create a versions list. - The parts section and versions section could be specified in a single file, so the "gesture" for using them could be pretty simple. Jim -- Jim Fulton
* Jim Fulton <jim@zope.com> [2009-08-12 11:52]:
On Wed, Aug 12, 2009 at 2:31 AM, Wolfgang Schnerring<ws@gocept.com> wrote:
* Jim Fulton <jim@zope.com> [2009-08-12 01:36]:
In playing with this today, I'm inclined to think that it would be simpler to use a list of packages in an option to specify the packages to be tested and used a versions section to specify versions, rather than using a controlled-packages configuration file.
This doesn't strike me as simpler to *use* (as I said earlier, I'd much prefer a *single* gesture of "use this KGS" to set up both the versions and the list of packages to run tests for, and I think it's worth spending effort to get there), but I'm not sure that's what you meant. What makes you prefer two files instead of one?
I didn't say anything about 2 files. I said I prefered a parts list in a single option in combination with a standard buildout versions section.
Sorry for my misunderstanding. In fact, I'm not hung up about the number of files all that much, but rather I'm searching for a way not to duplicate information. And that seems rather diffcult, since you assert...
- We'll need the versions section to consume the KGS. That is, given a KGS, you'll often want to use the versions in other buildouts to limit them to the known good configuration.
...while Martijn said here <http://article.gmane.org/gmane.comp.web.zope.devel/21328> that the KGS will need to store more information about each package than a versions section can handle.
- The parts section and versions section could be specified in a single file, so the "gesture" for using them could be pretty simple.
This seems to be the best we can do, given the above requirements. If I understand you correctly, that file would then look something like this: [versions] zope.foo = 1.2.3 grok.bar = 1.1.0 thirdparty.dependency = 4.4 [zope.foo] tested = true kgs = ztk develop = http://svn.zope.org/repos/main/zope.foo/branches/1.2 [grok.bar] tested = true develop = http://svn.zope.org/repos/main/grok.bar/trunk [thirdparty.dependency] tested = no z3c.recipe.compattest/kgs would learn to extract all sections from the above where tested=true. And zope.kgs/zope.release could then probably be retired (or am I missing something?).
- I think a parts list in an option will be easier to control. For example, you will be able to use the standard buildout option incrementing and decrementating machinery to modify it.
I don't understand how a parts list could help here. Wolfgang
On Wed, Aug 12, 2009 at 7:07 AM, Wolfgang Schnerring<ws@gocept.com> wrote:
* Jim Fulton <jim@zope.com> [2009-08-12 11:52]:
On Wed, Aug 12, 2009 at 2:31 AM, Wolfgang Schnerring<ws@gocept.com> wrote:
* Jim Fulton <jim@zope.com> [2009-08-12 01:36]:
In playing with this today, I'm inclined to think that it would be simpler to use a list of packages in an option to specify the packages to be tested and used a versions section to specify versions, rather than using a controlled-packages configuration file.
This doesn't strike me as simpler to *use* (as I said earlier, I'd much prefer a *single* gesture of "use this KGS" to set up both the versions and the list of packages to run tests for, and I think it's worth spending effort to get there), but I'm not sure that's what you meant. What makes you prefer two files instead of one?
I didn't say anything about 2 files. I said I prefered a parts list in a single option in combination with a standard buildout versions section.
Sorry for my misunderstanding. In fact, I'm not hung up about the number of files all that much, but rather I'm searching for a way not to duplicate information. And that seems rather diffcult, since you assert...
- We'll need the versions section to consume the KGS. That is, given a KGS, you'll often want to use the versions in other buildouts to limit them to the known good configuration.
...while Martijn said here <http://article.gmane.org/gmane.comp.web.zope.devel/21328> that the KGS will need to store more information about each package than a versions section can handle.
So there he said:
I do think we need a computer readable system that includes information like this:
* whether a project is in a KGS (such as for the ZTK)
* whether a project is used to test a KGS (a package not in the ZTK but used to test whether changes don't induce breakage, let's say, grokcore.component)
* the locked down version of the package.
* where the *next* version of the locked down version is being developed. Generally this is the SVN trunk, but could be a stable branch.
I'm not sure why this all has to be computer readable. What is the use case for computer readable knowledge of whether a package is in the kgs or just used to test it? What is the need for computer readable information about where the package is being developed.
- The parts section and versions section could be specified in a single file, so the "gesture" for using them could be pretty simple.
This seems to be the best we can do, given the above requirements. If I understand you correctly, that file would then look something like this:
[versions] zope.foo = 1.2.3 grok.bar = 1.1.0 thirdparty.dependency = 4.4
[zope.foo] tested = true kgs = ztk develop = http://svn.zope.org/repos/main/zope.foo/branches/1.2
[grok.bar] tested = true develop = http://svn.zope.org/repos/main/grok.bar/trunk
[thirdparty.dependency] tested = no
z3c.recipe.compattest/kgs would learn to extract all sections from the above where tested=true. And zope.kgs/zope.release could then probably be retired (or am I missing something?).
This seems heavier than needed. Also, if someone extends this, they're going to get an awful lot of sections that might have names that conflict with names in their buildout. I do like the fact that the versions section is reusable. :) Here's an alternative: [versions] zope.foo = 1.2.3 zope.bar = 1.2.3 zope.app.baz = 1.2.3 grok.bar = 1.1.0 thirdparty.dependency = 4.4 [ztk] projects = zope.foo zope.bar zope.app.baz also-tested = grok.bar Now, if I want to test using this, I do something like: [buildout] parts = ztk-tests extends = the-file-above [ztk-tests] recipe = our.kgs.testing.recipe projects = ${ztk:projects} ${ztk:also-tested} I think this is much cleaner and gives us: - Easily reusable information, including versions information that can be used by any buildout. - Information about the projects in the ztk - Information about additional projects to be tested. It doesn't give us machine-readable information about where the next version of the project is being developed. Why do we need that? Also, with this, the recipe doesn't have to parse anything.
- I think a parts list in an option will be easier to control. For example, you will be able to use the standard buildout option incrementing and decrementating machinery to modify it.
I don't understand how a parts list could help here.
Now suppose we want to add another project to be tested, 'our.app': [buildout] parts = ztk-tests extends = the-file-above [ztk-tests] recipe = our.kgs.testing.recipe projects = ${ztk:projects} ${ztk:also-tested} our.app But we don't want to bother testing zope.app.baz. We don't use it and the tests take too long: [buildout] parts = ztk-tests extends = the-file-above [ztk-tests] recipe = our.kgs.testing.recipe projects = ${ztk:projects} ${ztk:also-tested} our.app [ztk] projects -= zope.app.baz Jim -- Jim Fulton
* Jim Fulton <jim@zope.com> [2009-08-12 20:56]:
This seems heavier than needed. Also, if someone extends this, they're going to get an awful lot of sections that might have names that conflict with names in their buildout. I do like the fact that the versions section is reusable. :)
Here's an alternative:
[versions] zope.foo = 1.2.3 zope.bar = 1.2.3 zope.app.baz = 1.2.3 grok.bar = 1.1.0 thirdparty.dependency = 4.4
[ztk] projects = zope.foo zope.bar zope.app.baz also-tested = grok.bar
Yup, that looks much better. As far as I'm concerned, let's use this. (I'll leave it to Martijn to explain whether/which/why additional information should be stored in the KGS in computer-readable form.) I've just updated z3c.recipe.compattest to support exactly this:
[buildout] parts = ztk-tests extends = the-file-above
[ztk-tests] recipe = z3c.recipe.compattest include = ${ztk:projects} ${ztk:also-tested}
Would you give it a try? Wolfgang
On Wednesday 12 August 2009, Jim Fulton wrote:
I'm not sure why this all has to be computer readable. What is the use case for computer readable knowledge of whether a package is in the kgs or just used to test it? What is the need for computer readable information about where the package is being developed.
We can use such a list to extract more information from included packages. For example, I have a script to extract all CHANGES.txt entries in comparison to an older KGS version. Thus, knowing the list of packages that we consider part of the KGS is very useful beyond testing, for example release management. Regards, Stephan PS: I am still in Germany but will be back home late Sunday to read discussions more carefully. -- Entrepreneur and Software Geek Google me. "Zope Stephan Richter"
Stephan Richter wrote:
On Wednesday 12 August 2009, Jim Fulton wrote:
I'm not sure why this all has to be computer readable. What is the use case for computer readable knowledge of whether a package is in the kgs or just used to test it? What is the need for computer readable information about where the package is being developed.
We can use such a list to extract more information from included packages. For example, I have a script to extract all CHANGES.txt entries in comparison to an older KGS version.
Ah, nice use case. In fact I also have a script that parses CHANGES.txt entries and turns them into atom feeds, which could then be merged. I intended to use this to monitor the development of the KGS. Regards, Martijn
On Tuesday 25 August 2009, Martijn Faassen wrote:
We can use such a list to extract more information from included packages. For example, I have a script to extract all CHANGES.txt entries in comparison to an older KGS version.
Ah, nice use case. In fact I also have a script that parses CHANGES.txt entries and turns them into atom feeds, which could then be merged. I intended to use this to monitor the development of the KGS.
See zope.release for a full list of scripts that I originally developed for the KGS. I really wish that we are not loosing this sort of capabilities, since it would make the release process a lengthy task again. Regards, Stephan -- Entrepreneur and Software Geek Google me. "Zope Stephan Richter"
Jim Fulton wrote:
I do think we need a computer readable system that includes information like this:
* whether a project is in a KGS (such as for the ZTK)
* whether a project is used to test a KGS (a package not in the ZTK but used to test whether changes don't induce breakage, let's say, grokcore.component)
* the locked down version of the package.
* where the *next* version of the locked down version is being developed. Generally this is the SVN trunk, but could be a stable branch.
I'm not sure why this all has to be computer readable. What is the use case for computer readable knowledge of whether a package is in the kgs or just used to test it?
Making it computer readable to get those packages in the KGS is not that useful, I imagine, though making it so could potentially see tools being developed.
What is the need for computer readable information about where the package is being developed.
That's a real usecase. There are tools that can already use this information, in particular mr.developer: http://pypi.python.org/pypi/mr.developer It's nice to be able to just get the relevant checkout of a package without having to hunt around trying to figure out where it is. Regards, Martijn
On Tue, Aug 11, 2009 at 7:36 PM, Jim Fulton<jim@zope.com> wrote:
On Tue, Aug 11, 2009 at 5:22 PM, Jim Fulton<jim@zope.com> wrote: ...
- The versions specified in the controlled-packages.cfg don't seem to be honored except for the package being tested. For example, ZODB3-3.9.0b2 is used for test-kgs-ZODB3, but 3.9.0b5 is used for everything else. Fortunately, it appears I can work around this using a buildout versions section.
Except that there's a bug in buildout that causes extras to be dropped when versions are specified via a versions section. This causes the extras to be omitted from the test scripts. :( Hopefully, I'll get a fixed buildout release out tomorrow.
Done
The good news is that with a hacked buildout and a versions section, I can run tests for all of the packages in http://docs.zope.org/zopetoolkit/about/packages.html except for zope.ucol. I'm getting test failures in 5 packages. I suspect that these are due to missing dependencies.
So, I created a KGS based on the ZTK and Fabio's recipe. If you want to try it, check out svn://svn.zope.org/repos/main/zopetoolkit/branches/jim-kgs/kgs. (I still think we want a simpler way to configure this, as proposed in an earlier message today.) Jim -- Jim Fulton
participants (4)
-
Jim Fulton -
Martijn Faassen -
Stephan Richter -
Wolfgang Schnerring