Unit Test Failures
The Zope2 unit tests have been failing for some time on buildbot.zope.com. Looks like a Five-related problem: http://buildbot.zope.org/Zope%20trunk%202.4%20Linux%20zc-buildbot/builds/109... -- Benji York Senior Software Engineer Zope Corporation
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Benji York wrote:
The Zope2 unit tests have been failing for some time on buildbot.zope.com. Looks like a Five-related problem: http://buildbot.zope.org/Zope%20trunk%202.4%20Linux%20zc-buildbot/builds/109...
The failing test is a functional test, not a unit test; I don't run them by default when I check in. I see a couple of problem points: the biggest is the use in the output of 'checkSiteManager.html' of a repr'ed dict, which might render differently between different machines (or even the same machine across Python versions). This is a classic case where the "doctest" does *not* provide clarity over a "traditional" 'unittest' style test, IMNSHO. I'm not sure what it is testing, either; CC'ing Phillip, whose fingerprints are on it, according the 'svn blame', for clarification. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDps5A+gerLs4ltQ4RAn2DAKDJ4PUXUGY8pGHI4RcSU6qUQm2FRgCdHTEM GNgrBRn/Y70IbuAhuJ48GGM= =0Um3 -----END PGP SIGNATURE-----
Tres Seaver wrote:
I see a couple of problem points: the biggest is the use in the output of 'checkSiteManager.html' of a repr'ed dict, which might render differently between different machines (or even the same machine across Python versions). This is a classic case where the "doctest" does *not* provide clarity over a "traditional" 'unittest' style test, IMNSHO.
It is often clearer to show the contents of a dict than making an assertion. In those cases where it's not, you can make a unittest style assertion, as in:
thisdict == thatdict True
This is really just a matter of knowing how to write the test. Generally, when you want to show a dict sample, the way to do it is with:
from zope.testing/doctestunit import pprint pprint(thisdict)
This formats the dictionary nicely and, most importantly, sorts the items before displaying it. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote:
This is really just a matter of knowing how to write the test. Generally, when you want to show a dict sample, the way to do it is with:
from zope.testing/doctestunit import pprint pprint(thisdict)
This formats the dictionary nicely and, most importantly, sorts the items before displaying it.
This is what I did in this test, btw. Philipp P.S.: My other reply got swallowed by Mailman, can someone accept the posting please? ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
[Benji York]
The Zope2 unit tests have been failing for some time on buildbot.zope.com. Looks like a Five-related problem: http://buildbot.zope.org/Zope%20trunk%202.4%20Linux%20zc-buildbot/builds/109...
[Tres Seaver]
The failing test is a functional test, not a unit test; I don't run them by default when I check in. I see a couple of problem points: the biggest is the use in the output of 'checkSiteManager.html' of a repr'ed dict, which might render differently between different machines (or even the same machine across Python versions). This is a classic case where the "doctest" does *not* provide clarity over a "traditional" 'unittest' style test, IMNSHO.
Possibly, but that clearly doesn't apply to the specific failures we're seeing ("True" versus "False" outcomes, and getting lots more output than expected).
I'm not sure what it is testing, either; CC'ing Phillip, whose fingerprints are on it, according the 'svn blame', for clarification.
These tests have "always" failed, and Phillip doesn't know why. Because they were failing, he changed them to run at level 2. That's not what level 2 is for, though, and the failures became "visible" to everyone when the buildbot was changed to pass `--all` to test.py (--all runs tests at all levels, level 2 among them ;-)). I opened a Collector issue on this about a month ago (I always run with `--all`, so these failures are old news to me): http://www.zope.org/Collectors/Zope/1947
Tim Peters wrote:
I'm not sure what it is testing, either; CC'ing Phillip, whose fingerprints are on it, according the 'svn blame', for clarification.
These tests have "always" failed, and Phillip doesn't know why. Because they were failing, he changed them to run at level 2. That's not what level 2 is for, though, and the failures became "visible" to everyone when the buildbot was changed to pass `--all` to test.py (--all runs tests at all levels, level 2 among them ;-)).
Yes, mea culpa.
I opened a Collector issue on this about a month ago (I always run with `--all`, so these failures are old news to me):
Rereading that issue again, it's totally surprising to see that there's no failure on Windows, which makes its source even harder to debug. I wouldn't know where to start (having tried to debug this problem in the past). Anyone got an idea? Philipp ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
[Tim]
I opened a Collector issue on this about a month ago (I always run with `--all`, so these failures are old news to me):
[Philipp]
Rereading that issue again, it's totally surprising to see that there's no failure on Windows, which makes its source even harder to debug.
No, these tests fail on Windows too. In fact, the problem was orginally reported against Windows: see entry #2 in: http://www.zope.org/Collectors/Zope/1931 What does not fail on Windows is the specific variation you asked me to try in your comment #2 on issue 1947: When you run it by itself, it passes, though. Maybe you can confirm that. In my comment #3 on issue 1947, I confirmed that: when I ran the functional.txt test _by itself_ on Windows, it passed.
I wouldn't know where to start (having tried to debug this problem in the past). Anyone got an idea?
For a start, disabuse yourself of the illusion that it acts differently on Windows than on Linux ;-)
Tim Peters wrote:
I wouldn't know where to start (having tried to debug this problem in the past). Anyone got an idea?
For a start, disabuse yourself of the illusion that it acts differently on Windows than on Linux ;-)
Yup, sorry, misread the issue. Speed kills... :) Philipp ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
[resending this from the right address, the original posting got swallowed by Mailman] Tres Seaver wrote:
Benji York wrote:
The Zope2 unit tests have been failing for some time on buildbot.zope.com. Looks like a Five-related problem: http://buildbot.zope.org/Zope%20trunk%202.4%20Linux%20zc-buildbot/builds/109...
The failing test is a functional test, not a unit test; I don't run them by default when I check in.
Actually, I put this particular test (functional.txt) on level=2 because it would fail when run together with other Five tests. It passes fine when run by itself. This test exists identically in Five 1.2 (based on Zope 2.8) where it passes just fine when run together with other tests and by itself. During the hot phase of making the deadline on November 1st, I decided that this wasn't a big issue and could be resolved during the beta phase (after all, the test passes by itself). I obviously forgot to look into it again, but then again, I still don't know what the heck is wrong with this.
I see a couple of problem points: the biggest is the use in the output of 'checkSiteManager.html' of a repr'ed dict, which might render differently between different machines (or even the same machine across Python versions).
Well, if you look closer you find that it uses pprint.pformat which always outputs the same on all machines (because it provides output sorted by the dictionary key).
This is a classic case where the "doctest" does *not* provide clarity over a "traditional" 'unittest' style test, IMNSHO.
Comparing dictionaries is not trivial in classic javiotic tests, either. Using pprint is actually quite an elegant way, I think.
I'm not sure what it is testing, either;
It's testing how local site managers get found during traversal and how that affects component lookup of e.g. views (depending on which site is traversed, you would find local components or not). Philipp
Philipp von Weitershausen wrote: ...
Actually, I put this particular test (functional.txt) on level=2 because it would fail when run together with other Five tests. It passes fine when run by itself. This test exists identically in Five 1.2 (based on Zope 2.8) where it passes just fine when run together with other tests and by itself. During the hot phase of making the deadline on November 1st, I decided that this wasn't a big issue and could be resolved during the beta phase (after all, the test passes by itself). I obviously forgot to look into it again, but then again, I still don't know what the heck is wrong with this.
Obviously, some other test isn't cleaning up after itself. The way to debug this is by trial and error, running this test in combination with other tests until it fails. (It would be cool, in some future version of the test runner, to automate the search process. Even a simple option to run tests by number would probably help a lot, making a manual binary search pretty easy.) Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
Jim Fulton wrote:
Obviously, some other test isn't cleaning up after itself.
Yes, that was obvious to me too. It was confusing that the same test would pass on Five 1.2, though, and I couldn't find any obvious differences. As trial and error usually takes time, I left the issue to be resolved at a later time. And then, as I said, I just forgot, my bad. I'm investigating now, but no promises that I'll produce results soon. I won't forget it this time, though :). Philipp ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
Philipp von Weitershausen wrote:
Jim Fulton wrote:
Obviously, some other test isn't cleaning up after itself.
Yes, that was obvious to me too. It was confusing that the same test would pass on Five 1.2, though, and I couldn't find any obvious differences. As trial and error usually takes time, I left the issue to be resolved at a later time. And then, as I said, I just forgot, my bad.
I'm investigating now, but no promises that I'll produce results soon. I won't forget it this time, though :).
I found the problem. I remembered how Zope X3 3.0 and 3.1+ handle the setup of the local component hooks differently (local component hooks make sure that zapi.getSiteManager() would actually look in the thread-local for a site and not just default to the global site). So, in the end the problem was actually that previous tests would clean up too much, not too little. I've checked in a fix in the Five repository, but haven't merged that the Zope repository yet. Will do that tomorrow as my eyes are slowly closing to sleep and the matches won't hold any longer ;). G'night Philipp
participants (5)
-
Benji York -
Jim Fulton -
Philipp von Weitershausen -
Tim Peters -
Tres Seaver