Hi everyone, last week I worked on getting the KGS trunk located in zope.release to pass without failures. (Python 2.5 on Linux, i686) It is in the best interest of everyone to keep the trunk in a passing condition. So please try hard to do that. I am certainly very tired of spending many hours fixing those sorts of problems. Failures were due to a wide variety of issues: - Missing imports! - Unreleased fixes. - Insufficient dependency lists. - Incompatible API changes. - Bug fixes in other packages. For the same reasons, there is a whole lot of packages I cannot add to the KGS without making further releases. If someone could setup a buildbot to run all the tests nightly, that would be great. Regards, Stephan -- Entrepreneur and Software Geek Google me. "Zope Stephan Richter"
On Mon, Jul 27, 2009 at 22:43, Stephan Richter<srichter@cosmos.phy.tufts.edu> wrote:
Hi everyone,
last week I worked on getting the KGS trunk located in zope.release to pass without failures. (Python 2.5 on Linux, i686)
Thank you for the work! Not the same result on my buildbot (64 bits): Tests with errors: test_send_failQuit (zope.sendmail.tests.test_mailer.TestSMTPMailer) test_send_failQuit (zope.sendmail.tests.test_mailer.TestSMTPMailerWithNoEHLO) Tests with failures: /srv/buildbot/slave/zope3/py2.5-64bit-linux-kgs35/build/test/eggs/keas.profile-0.1.0-py2.5.egg/keas/profile/README.txt /srv/buildbot/slave/zope3/py2.5-64bit-linux-kgs35/build/test/eggs/keas.unit-1.0.0-py2.5.egg/keas/unit/README.txt /srv/buildbot/slave/zope3/py2.5-64bit-linux-kgs35/build/test/eggs/zope.fssync-3.5.1-py2.5.egg/zope/fssync/tests/../caseinsensitivity.txt /srv/buildbot/slave/zope3/py2.5-64bit-linux-kgs35/build/test/eggs/zope.index-3.5.2-py2.5-linux-x86_64.egg/zope/index/text/tests/../textindex.txt The log: http://zope.buildbot.securactive.org/builders/py2.5-64bit-linux-kgs35/builds...
If someone could setup a buildbot to run all the tests nightly, that would be great.
http://zope.buildbot.securactive.org/waterfall 8 Buildbots : - KGS Zope 3.4.0 Python2.4 32bits - KGS Zope 3.4.0 Python2.4 64bits - KGS Zope 3.4.0 Python2.5 32bits - KGS Zope 3.4.0 Python2.5 64bits - KGS Zope 3.5.0 Python2.5 32bits - KGS Zope 3.5.0 Python2.5 64bits - KGS Zope 3.5.0 Python2.6 32bits - KGS Zope 3.5.0 Python2.6 64bits Need to work on Firewall ( for internal buildbot), python-memcached needs to retrieve external FTP data (?!). -- Sebastien Douche <sdouche@gmail.com> Twitter: http://bit.ly/afkrK (agile, python, open source)
On Mon, Jul 27, 2009 at 4:43 PM, Stephan Richter<srichter@cosmos.phy.tufts.edu> wrote:
Hi everyone,
last week I worked on getting the KGS trunk located in zope.release to pass without failures. (Python 2.5 on Linux, i686)
Thanks so much! As we discussed on IRC, there are still some open issues we (not just you :) need to work through: 1. Some of the tests don't pass any more. I think this is because you added some packages after sending your note. 2. Some of the tests only pass if run separately, due to test interactions. Presumably, this means that other tests aren't cleaning up after themselves. I think we need a standard automated way to run each package's tests separately. I think some folks are working on that. 3. zope.release tests more than I think it needs to. This is OK if everything passes, but that's not the situation we have. Many of us have an acute problem that we can't update our applications to current package versions because it's hard ti find combinations that work together. We really need to solve this problem soon, even if it means being less ambitious. 4. We need to get things running for multiple Python versions on multiple platforms, including different Python versions, architectures (32 vs 64 bit) and operating systems, especially linux and windows. Some sort of testing bots would be helpful here.
It is in the best interest of everyone to keep the trunk in a passing condition. So please try hard to do that. I am certainly very tired of spending many hours fixing those sorts of problems.
So this means that if someone modifies a package that is in the KGS, they need to run the KGS tests before checking in.
If someone could setup a buildbot to run all the tests nightly, that would be great.
Yes it would. :) Jim -- Jim Fulton
On Sun, Aug 2, 2009 at 6:34 PM, Jim Fulton<jim@zope.com> wrote:
2. Some of the tests only pass if run separately, due to test interactions. Presumably, this means that other tests aren't cleaning up after themselves. I think we need a standard automated way to run each package's tests separately. I think some folks are working on that.
I had a similar problem with the Plone trunk tests and my solution to that is http://pypi.python.org/pypi/plone.recipe.alltests (licensed as ZPL). My example usage of it is at http://svn.plone.org/svn/plone/buildouts/plone-coredev/trunk/tests.cfg The main idea here is to generate a "./bin/alltests" script, which internally runs the tests on a package by package basis (using a simple os.system call to the zc.recipe.testrunner test script). It parses the test result and exists at the end with a status code of 0 if there where no test failures at all or 1 otherwise, making it buildbot friendly. Generally it will try to run the tests for all packages in the working set and add the package to the test-path automatically. The exclude option allows to exclude packages from being tested. For example if those do not include their tests in the released distribution or may not be compatible with zope.testing. The assumption here is that you want to test everything you have with an opt-out. If some package pulls in a new dependency that dependency will be tested too. The other advanced option is to use a "groups" definition, which allows to put multiple packages into one group and then run all of them in one run (internally using ./bin/test -s foo -s bar). This is basically to avoid test layer startup cost for similar packages which are known to behave well when run together. This stuff is fairly new and might be missing features. I'd welcome feedback and patches ;) Hanno
* Jim Fulton <jim@zope.com> [2009-08-02 18:34]:
2. Some of the tests only pass if run separately, due to test interactions. Presumably, this means that other tests aren't cleaning up after themselves. I think we need a standard automated way to run each package's tests separately. I think some folks are working on that.
I've just released z3c.recipe.compattest-0.6, which can now populate the list of packages to run test for from a buildout section, which makes it easy to test against a KGS. So, if you're developing zope.foo, you can set up your buildout.cfg like this: [buildout] develop = . parts = <whatever else you need> compat extends = http://url/to/kgs/versions.cfg # assuming said versions.cfg uses a section called [versions]: versions = versions [versions] # use development version of our package zope.foo = [compat] recipe = z3c.recipe.compattest and then run bin/test-compat to run the tests of each package in versions against the development version of zope.foo (in a separate process).
So this means that if someone modifies a package that is in the KGS, they need to run the KGS tests before checking in.
After we reach a consensus on how to do it (I'm in favor of the way outlined above, of course ;-), I'd like to add a step-by-step walkthrough of the day-to-day development of a package somewhere below <http://docs.zope.org/zopetoolkit/process/index.html>, which would include a description of how to run KGS tests. Regarding the KGS, I have a question: Where is the current KGS and how do we update it? By "where is" I mean the location of the versions.cfg, by "current" I mean the "trunk", the version currently in development, the version where we probably want to bump the version number of a package as soon as a version is released, and by "update" I mean that I'd like this operation to involve no manual interaction other than committing the new version number to SVN. Wolfgang
Hello Wolfgang, * 2009-08-07 11:42, Wolfgang Schnerring wrote:
[buildout] develop = . parts = <whatever else you need> compat extends = http://url/to/kgs/versions.cfg # assuming said versions.cfg uses a section called [versions]: versions = versions
I've done something similar in z3c.recipe.kgs (which is a fork of compattest, I did not want to commit anything there, but I'm more than happy to bring back my changes). The KGS is defined here: http://svn.zope.org/*checkout*/zope.release/trunk/releases/controlled-packag... IMHO the KGS testing should be done using the controlled-packages.cfg and not versions, because some of the packages in the KGS are marked as not to be tested.
Regarding the KGS, I have a question: Where is the current KGS and how do we update it? By "where is" I mean the location of the versions.cfg,
AFAICK, It can be built from zope.release.
and by "update" I mean that I'd like this operation to involve no manual interaction other than committing the new version number to SVN.
I suppose there is no policy (yet) regarding how we manage the KGS. Fabio
Thanks Fabio and Wolfgang, How to do you specify the projects to be tested? Does every project in versions get tested? If so, how do you specify versions for projects that you don't want to run tests for but do want to fix the version of. (Note that I'm using more precise jargon: "project" rather than "package". In the Python world, the word "package" means a module that is implemented as a directory rather than a file. A project is a collection of software for which we create distributions. We should generally be using the word "project" rather than "package".) Jim -- Jim Fulton
Hello Jim, * 2009-08-07 12:28, Jim Fulton wrote:
How to do you specify the projects to be tested? Does every project in versions get tested? If so, how do you specify versions for projects that you don't want to run tests for but do want to fix the version of.
In my recipe, it automatically tests all the libraries that are marked as tested=true in the controlled-packages.cfg, but you can filter and test only a subset of those packages, for example: [test-kgs] recipe = z3c.recipe.kgs kgs = http://svn.zope.org/*checkout*/zope.release/trunk/releases/controlled-packag... packages = zope.component zope.interface zope.event zope.configuration If somebody tells me how it is possible to get "projects" from the controlled-packages.cfg using zope.kgs, I can modify the recipe code to understand it. :) -- Fabio Tranchitella http://www.kobold.it Free Software Developer and Consultant http://www.tranchitella.it _____________________________________________________________________ 1024D/7F961564, fpr 5465 6E69 E559 6466 BF3D 9F01 2BF8 EE2B 7F96 1564
2009/8/7 Fabio Tranchitella <kobold@kobold.it>:
Hello Jim,
* 2009-08-07 12:28, Jim Fulton wrote:
How to do you specify the projects to be tested? Does every project in versions get tested? If so, how do you specify versions for projects that you don't want to run tests for but do want to fix the version of.
In my recipe, it automatically tests all the libraries that are marked as tested=true in the controlled-packages.cfg,
Ah cool, so you don't just use the versions section. Wolfgang? Jim -- Jim Fulton
* Jim Fulton <jim@zope.com> [2009-08-07 12:39]:
2009/8/7 Fabio Tranchitella <kobold@kobold.it>:
* 2009-08-07 12:28, Jim Fulton wrote:
How to do you specify the projects to be tested? Does every project in versions get tested? If so, how do you specify versions for projects that you don't want to run tests for but do want to fix the version of.
In my recipe, it automatically tests all the libraries that are marked as tested=true in the controlled-packages.cfg,
Ah cool, so you don't just use the versions section. Wolfgang?
My guideline was "let's rely on buildout" rather than "let's use zope.kgs or anything else rather specific". Wolfgang
On Friday 07 August 2009, Fabio Tranchitella wrote:
If somebody tells me how it is possible to get "projects" from the controlled-packages.cfg using zope.kgs, I can modify the recipe code to understand it. :)
I did not use the precise jargon; read project wherever I write about packages in the zope.kgs and zope.release packages. Regards, Stephan -- Entrepreneur and Software Geek Google me. "Zope Stephan Richter"
* Jim Fulton <jim@zope.com> [2009-08-07 06:01]:
How to do you specify the projects to be tested? Does every project in versions get tested? If so, how do you specify versions for projects that you don't want to run tests for but do want to fix the version of.
With z3c.recipe.compattest, to build the list of projects to run tests for you can at the moment: - specify project names to include - specify buildout section names whose keys will be included (and it defaults to the [versions] section for ease of use) - specify regexes to remove entries from the included list So if your [versions] section contains projects you don't want to run tests for, I guess you either need to exclude those explicitly, or not feed this section to compattest in the first place and build up the include list in a different way instead. Could you elaborate what use case you are envisioning here? Wolfgang
On Fri, Aug 7, 2009 at 6:52 AM, Wolfgang Schnerring<ws@gocept.com> wrote:
* Jim Fulton <jim@zope.com> [2009-08-07 06:01]:
How to do you specify the projects to be tested? Does every project in versions get tested? If so, how do you specify versions for projects that you don't want to run tests for but do want to fix the version of.
With z3c.recipe.compattest, to build the list of projects to run tests for you can at the moment: - specify project names to include - specify buildout section names whose keys will be included (and it defaults to the [versions] section for ease of use) - specify regexes to remove entries from the included list
So if your [versions] section contains projects you don't want to run tests for, I guess you either need to exclude those explicitly, or not feed this section to compattest in the first place and build up the include list in a different way instead.
Could you elaborate what use case you are envisioning here?
Yes, although I'll note that this was discussed here just yesterday. We have a set of projects we want to run tests for. We want the tests to be as reproducible as possible, so we want to control what can cause changes in test outcome, and so that we have a "working configuration" that applications using the projects can use as a starting point. For this reason, we specify the versions of all projects we test *and* all projects those projects depend on. We may not test all of the dependencies for various reasons, but we still want to fix the versions of the dependencies. Jim -- Jim Fulton
Jim Fulton wrote:
(Note that I'm using more precise jargon: "project" rather than "package". In the Python world, the word "package" means a module that is implemented as a directory rather than a file. A project is a collection of software for which we create distributions. We should generally be using the word "project" rather than "package".)
+1 on the more precise jargon. Regards, Martijn
* Fabio Tranchitella <kobold@kobold.it> [2009-08-07 11:46]:
* 2009-08-07 11:42, Wolfgang Schnerring wrote: http://svn.zope.org/*checkout*/zope.release/trunk/releases/controlled-packag... IMHO the KGS testing should be done using the controlled-packages.cfg and not versions, because some of the packages in the KGS are marked as not to be tested.
If controlled-packages.cfg is to be the authoritative represenation of the KGS (and maybe it should be, since it contains more information than versions.cfg) then I'd probably wish for a buildout recipe that can pin versions based on that data format, because I much prefer less moving parts. In other words, I would very much like a single gesture to tell buildout "use this KGS": extends=something-or-other.cfg. The rest should Just Work(tm). - Less "compiling". Having to explicitly regenerate versions.cfg feels superfluous - While developing something, you probably want to change the KGS from the official one to a local one, having to remember to change both the versions.cfg and the packages.cfg for the compattests is error prone.
Regarding the KGS, I have a question: Where is the current KGS and how do we update it? By "where is" I mean the location of the versions.cfg,
AFAICK, It can be built from zope.release.
Right. But it's not publicly accessible. (And uploading to say download.zope.org is restricted to people with access there) But if versions.cfg is not to be the authoritative representation of the KGS then my point here is moot. Wolfgang
Hey, Wolfgang Schnerring wrote:
* Fabio Tranchitella <kobold@kobold.it> [2009-08-07 11:46]:
* 2009-08-07 11:42, Wolfgang Schnerring wrote: http://svn.zope.org/*checkout*/zope.release/trunk/releases/controlled-packag... IMHO the KGS testing should be done using the controlled-packages.cfg and not versions, because some of the packages in the KGS are marked as not to be tested.
If controlled-packages.cfg is to be the authoritative represenation of the KGS (and maybe it should be, since it contains more information than versions.cfg) then I'd probably wish for a buildout recipe that can pin versions based on that data format, because I much prefer less moving parts.
In other words, I would very much like a single gesture to tell buildout "use this KGS": extends=something-or-other.cfg. The rest should Just Work(tm).
I'm +1 on this, the less moving parts the better and the less compiling/generation we can get away with, the better too. It needs to be as straightforward as we can make it. I do think we need a computer readable system that includes information like this: * whether a project is in a KGS (such as for the ZTK) * whether a project is used to test a KGS (a package not in the ZTK but used to test whether changes don't induce breakage, let's say, grokcore.component) * the locked down version of the package. * where the *next* version of the locked down version is being developed. Generally this is the SVN trunk, but could be a stable branch. The system that manages this shouldn't be tied to Zope in particular. It'd be great for managing, say, Grok's, KGS too. Regards, Martijn
Hey Fabio, Fabio Tranchitella wrote:
* 2009-08-07 11:42, Wolfgang Schnerring wrote:
[buildout] develop = . parts = <whatever else you need> compat extends = http://url/to/kgs/versions.cfg # assuming said versions.cfg uses a section called [versions]: versions = versions
I've done something similar in z3c.recipe.kgs (which is a fork of compattest, I did not want to commit anything there, but I'm more than happy to bring back my changes).
Ah, cool, I do hope we can merge your changes back again into z3c.recipe.compattest soon. Next time perhaps consider just branching in the SVN? That way it's clearer that changes can be merged back in.. Regards, Martijn
Hey, Wolfgang Schnerring wrote: [snip]
After we reach a consensus on how to do it (I'm in favor of the way outlined above, of course ;-), I'd like to add a step-by-step walkthrough of the day-to-day development of a package somewhere below <http://docs.zope.org/zopetoolkit/process/index.html>, which would include a description of how to run KGS tests.
That'd be awesome!
Regarding the KGS, I have a question: Where is the current KGS and how do we update it?
For the Zope Toolkit, I don't think it exists yet. There's a Zope 3.4 KGS and I know Stephan also mentioned wanting to make a 'wide KGS' that includes a lot more packages than what's in the ZTK (this can expand on the ZTK KGS though). For Grok, we maintain a versions.cfg list in the Grok trunk.
By "where is" I mean the location of the versions.cfg, by "current" I mean the "trunk", the version currently in development, the version where we probably want to bump the version number of a package as soon as a version is released, and by "update" I mean that I'd like this operation to involve no manual interaction other than committing the new version number to SVN.
I'd like to see such a method of working too. Regards, Martijn
Hey, Stephan Richter wrote:
- Insufficient dependency lists.
I wish we were publishing z3c.recipe.depgraph results for all these packages in a convenient place. I'm worried that fixing dependencies introduced cycles again (that were of course really there). That's not to complain about all the work done to make sure tests pass at all, just to make sure we keep our dependencies free from cycles and as flat as possible. Thanks, Martijn
participants (7)
-
Fabio Tranchitella -
Hanno Schlichting -
Jim Fulton -
Martijn Faassen -
Sebastien Douche -
Stephan Richter -
Wolfgang Schnerring