Re: Unit Test Failures
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Philipp von Weitershausen wrote:
Tres Seaver wrote:
Benji York wrote:
The Zope2 unit tests have been failing for some time on buildbot.zope.com. Looks like a Five-related problem: http://buildbot.zope.org/Zope%20trunk%202.4%20Linux%20zc-buildbot/builds/109...
The failing test is a functional test, not a unit test; I don't run them by default when I check in.
Actually, I put this particular test (functional.txt) on level=2 because it would fail when run together with other Five tests. It passes fine when run by itself. This test exists identically in Five 1.2 (based on Zope 2.8) where it passes just fine when run together with other tests and by itself. During the hot phase of making the deadline on November 1st, I decided that this wasn't a big issue and could be resolved during the beta phase (after all, the test passes by itself). I obviously forgot to look into it again, but then again, I still don't know what the heck is wrong with this.
I see a couple of problem points: the biggest is the use in the output of 'checkSiteManager.html' of a repr'ed dict, which might render differently between different machines (or even the same machine across Python versions).
Well, if you look closer you find that it uses pprint.pformat which always outputs the same on all machines (because it provides output sorted by the dictionary key).
I see that in the implementation; it isn't documented as part of pprint's contract, however.
This is a classic case where the "doctest" does *not* provide clarity over a "traditional" 'unittest' style test, IMNSHO.
Comparing dictionaries is not trivial in classic javiotic tests, either. Using pprint is actually quite an elegant way, I think.
But the dictionary here is an *artifact* -- what you are really trying to do would be expressed as individual assertions in a classic unit test: self.failUnless(IFiveUtilityRegistry.providedBy(utility_service)) self.failIf(zapi.getSiteManager() is zapi.getGlobalSiteManager()) etc. If you really want to test equality of dictionaries, then 'self.assertEqual' does the trick just fine.
I'm not sure what it is testing, either;
It's testing how local site managers get found during traversal and how that affects component lookup of e.g. views (depending on which site is traversed, you would find local components or not).
OK, thanks! Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDptfi+gerLs4ltQ4RAtfkAKCiJLa+s+8GaNxew9DcIPWrbw7qXgCeNaSj csBJ641nurB0xPwBu6xLByY= =juKw -----END PGP SIGNATURE-----
Tres Seaver wrote:
Well, if you look closer you find that it uses pprint.pformat which always outputs the same on all machines (because it provides output sorted by the dictionary key).
I see that in the implementation; it isn't documented as part of pprint's contract, however.
Yes it is: http://python.org/doc/current/lib/module-pprint.html
But the dictionary here is an *artifact* -- what you are really trying to do would be expressed as individual assertions in a classic unit test:
self.failUnless(IFiveUtilityRegistry.providedBy(utility_service)) self.failIf(zapi.getSiteManager() is zapi.getGlobalSiteManager())
etc. If you really want to test equality of dictionaries, then 'self.assertEqual' does the trick just fine.
All I need is a means to get information from the view (which is our guinnea pig component that tries to look up local stuff) to the functional test. Views typically output strings, whether we're in doctests or not. So, in the end we're comparing strings. pprint'ing a dictionary is just a convenient, but perhaps misleading way of doing this. Philipp ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Philipp von Weitershausen wrote:
Tres Seaver wrote:
Well, if you look closer you find that it uses pprint.pformat which always outputs the same on all machines (because it provides output sorted by the dictionary key).
I see that in the implementation; it isn't documented as part of pprint's contract, however.
Yes it is: http://python.org/doc/current/lib/module-pprint.html
I must be blind, but I don't see any occurrence of 'sort' on that page or on the "PrettyPrinter Objects" page under it, nor in the pydoc output nor the module docstring.
But the dictionary here is an *artifact* -- what you are really trying to do would be expressed as individual assertions in a classic unit test:
self.failUnless(IFiveUtilityRegistry.providedBy(utility_service)) self.failIf(zapi.getSiteManager() is zapi.getGlobalSiteManager())
etc. If you really want to test equality of dictionaries, then 'self.assertEqual' does the trick just fine.
All I need is a means to get information from the view (which is our guinnea pig component that tries to look up local stuff) to the functional test. Views typically output strings, whether we're in doctests or not. So, in the end we're comparing strings. pprint'ing a dictionary is just a convenient, but perhaps misleading way of doing this.
In that case, I would have added methods to the view and asserted their return values, I think, even in a doctest. De gustibus, perhaps. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDptwT+gerLs4ltQ4RAopbAJ0UVEeeLZ0tZ4pMu2TBAAoYFy+60gCgyeUf BU+mhROlfBZ546O7hgoDZ1E= =qMNh -----END PGP SIGNATURE-----
Tres Seaver wrote:
Well, if you look closer you find that it uses pprint.pformat which always outputs the same on all machines (because it provides output sorted by the dictionary key).
I see that in the implementation; it isn't documented as part of pprint's contract, however.
Yes it is: http://python.org/doc/current/lib/module-pprint.html
I must be blind, but I don't see any occurrence of 'sort' on that page or on the "PrettyPrinter Objects" page under it, nor in the pydoc output nor the module docstring.
Now I see what you mean by "contract". You're right, I guess it isn't documented then, but perhaps it should be.
But the dictionary here is an *artifact* -- what you are really trying to do would be expressed as individual assertions in a classic unit test:
self.failUnless(IFiveUtilityRegistry.providedBy(utility_service)) self.failIf(zapi.getSiteManager() is zapi.getGlobalSiteManager())
etc. If you really want to test equality of dictionaries, then 'self.assertEqual' does the trick just fine.
All I need is a means to get information from the view (which is our guinnea pig component that tries to look up local stuff) to the functional test. Views typically output strings, whether we're in doctests or not. So, in the end we're comparing strings. pprint'ing a dictionary is just a convenient, but perhaps misleading way of doing this.
In that case, I would have added methods to the view and asserted their return values, I think, even in a doctest. De gustibus, perhaps.
Yup, I should probably do this. Philipp ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
On 12/19/05, Philipp von Weitershausen <philipp@weitershausen.de> wrote:
Now I see what you mean by "contract". You're right, I guess it isn't documented then, but perhaps it should be.
That's never been part of the contract and, as Tres notes, it's inconsistent. The implmentation will only sort when it uses the multi-line representation of the dict, not the single-line form. I don't oppose adding this to the contract and fixing it for the single-line representation for Python 2.5, but I'd rather think more about what the right interface is between displayed objects and the formatter. This is where a component architecture would really come in handy in the standard library. :-) -Fred -- Fred L. Drake, Jr. <fdrake at gmail.com> "There is no wealth but life." --John Ruskin
On the actual problem, this is a big red flag for me: 'isinstance(zapi.getSiteManager(), FiveSiteManager)': True, Fails. So, it's not a FivesiteManager. What is it? None, or something else? Since this seems to be a cleanup issue, my guess is: Something else. Probably another one of the site tests doesn't clean up (and that's probably my fault).
Lennart Regebro wrote:
On the actual problem, this is a big red flag for me:
'isinstance(zapi.getSiteManager(), FiveSiteManager)': True,
Fails. So, it's not a FivesiteManager. What is it? None, or something else?
It's the global site manager if not the FiveSiteManager.
Since this seems to be a cleanup issue, my guess is: Something else. Probably another one of the site tests doesn't clean up (and that's probably my fault).
Nope, it's really nobody's fault but mine. And the site tests weren't responsible for sure. So, before I explain this issue in detail, let me say: I've already fixed this in the Five repo after remembering a peculiarity in Zope 3.1+ last night. I just haven't had the time yet to merge it to the Zope repo, will do that first thing I'm back in SVN check-in range (on the road right now). So, here's the deal: zope.component doesn't have a clue about local sites. All it knows is the global site manager. However, it *does* provide a hook so that other software can influence where the site manager is found. zope.app.component (notice the app here) provides such a hook that knows about the site in the thread local. That's how zope.component in the end *does* find local sites, because it calls the hook. Now, in Zope X3 3.0, the hooks were set via ZCML using the <hook /> directive. This directive was ripped out in Zope 3.1 and replaced by a simple function, zope.app.component.hooks.setHooks(). This function is called upon Zope startup by zope.app.appsetup.appsetup.config() so that the hooks are set when Zope is running. A clean-up function is registered with the general test clean-up facility in Zope 3 that clears the hook. So, a unit test would normally not see the hooks *if* it is run *after* a test that has used e.g. placelesssetup.tearDown(). So, in the end, it was actually a problem of cleaning up too much, not too little. I've already had to deal with this problem in the utility tests and solved it there, but didn't remember it when I initially wrote the functional test. That's why my debugging efforts were futile back then. I guess this needs to be handled a better way, but I leave it to Stephan (who I think is responsible for the hooks handling) to decide what to do. Philipp ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
... [Philipp]
Well, if you look closer you find that it uses pprint.pformat which always outputs the same on all machines (because it provides output sorted by the dictionary key).
[Tres Seaver]
I see that in the implementation; it isn't documented as part of pprint's contract, however.
It's not part of the implementation either. For example,
d = {"z": 1, "m": 2} pprint.pprint(d) {'z': 1, 'm': 2}
That is, it's not true that pprint always sorts a dict for display. Looks like Jim's suggested from zope.testing.doctestunit import pprint inherits this insecurity. In the example above, two things conspire to give "unsorted" output: 1. pprint(dict) doesn't try to sort at all if _repr(dict) is "short". 2. On my 32-bit box, hash('z') < hash('m'), which leads to platform accidents in dict insertion putting 'z' before 'm' in the "natural" dict iteration order:
d {'z': 1, 'm': 2}
Because of #1, pprint passes on that order. On a 64-bit box, the order may differ (or across Python versions on one box if the string hash, or dict insertion, algorithms change).
Tim Peters wrote: ... a good lecture on pprint vs. dicts Thanks, I'll be more careful about using pprint for dicts then. Philipp ---------------------------------------------------------------- This message was sent using IMP, the Internet Messaging Program.
Tim Peters wrote:
...
[Philipp]
Well, if you look closer you find that it uses pprint.pformat which
always outputs the
same on all machines (because it provides output sorted by the
dictionary key).
[Tres Seaver]
I see that in the implementation; it isn't documented as part of pprint's contract, however.
It's not part of the implementation either. For example,
d = {"z": 1, "m": 2} pprint.pprint(d)
{'z': 1, 'm': 2}
That is, it's not true that pprint always sorts a dict for display. Looks like Jim's suggested
from zope.testing.doctestunit import pprint
inherits this insecurity.
No, it doesn't.
from zope.testing.doctestunit import pprint pprint({"z": 1, "m": 2}) {'m': 2, 'z': 1}
Note both the sorting and the wrapping. See below.
In the example above, two things conspire to give "unsorted" output:
1. pprint(dict) doesn't try to sort at all if _repr(dict) is "short".
2. On my 32-bit box, hash('z') < hash('m'), which leads to platform accidents in dict insertion putting 'z' before 'm' in the "natural" dict iteration order:
d
{'z': 1, 'm': 2}
Because of #1, pprint passes on that order.
On a 64-bit box, the order may differ (or across Python versions on one box if the string hash, or dict insertion, algorithms change).
zope.doctestunit.pprint creates and uses a pretty printer with width set to 1, so all dicts are "long" and thus sorted. Note that, in Python 2.4, you can now pass a width to pprint without creating a separate pretty printer:
from pprint import pprint pprint({"z": 1, "m": 2}) {'z': 1, 'm': 2} pprint({"z": 1, "m": 2}, width=1) {'m': 2, 'z': 1}
So maybe we can phase out the use of docutestunit's pprint.) Perhaps we should push to get the sorting behavior of pprint documented to allay your concerns. Better yet, perhaps we can get all dicts to be sorted. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
[Tim]
Looks like Jim's suggested
from zope.testing.doctestunit import pprint
inherits this insecurity.
[Jim]
No, it doesn't.
from zope.testing.doctestunit import pprint pprint({"z": 1, "m": 2}) {'m': 2, 'z': 1}
Note both the sorting and the wrapping.
< See below. Cool! I confess that the 'width' fiddling in: def pprint(ob, **opts): if 'width' not in opts: opts['width'] = 1 return PrettyPrinter(**opts).pprint(ob) was incomprehensible to me just now -- you might want to add a comment to that ;-)
zope.doctestunit.pprint creates and uses a pretty printer with width set to 1, so all dicts are "long" and thus sorted.
Well, I understand why that works, but it's not part of pprint's contract either.
Note that, in Python 2.4, you can now pass a width to pprint without creating a separate pretty printer:
from pprint import pprint pprint({"z": 1, "m": 2}) {'z': 1, 'm': 2} pprint({"z": 1, "m": 2}, width=1) {'m': 2, 'z': 1}
So maybe we can phase out the use of docutestunit's pprint.)
Perhaps we should push to get the sorting behavior of pprint documented to allay your concerns.
I think it's harder than you yet realize: dicts don't require that keys be orderable, so pprint can't promise to sort dict displays. As-is, pprint simply blows up if, e.g., you pass it a "big dict" using complex numbers as keys. Could be nobody has noticed that yet, but since the trend in Python is to raise exceptions for non-equality comparisons of objects of different types, this kind of failure is bound to become more common. In the end, I wouldn't be surprised if pprint(dict) got changed to sort on the repr of the keys instead of on the keys themselves.
Better yet, perhaps we can get all dicts to be sorted.
As above.
Tim Peters wrote: ...
Well, I understand why that works, but it's not part of pprint's contract either.
What contract. :) Aren't you always telling me to read the source?
Note that, in Python 2.4, you can now pass a width to pprint without creating a separate pretty printer:
from pprint import pprint pprint({"z": 1, "m": 2}) {'z': 1, 'm': 2} pprint({"z": 1, "m": 2}, width=1) {'m': 2, 'z': 1}
So maybe we can phase out the use of docutestunit's pprint.)
Perhaps we should push to get the sorting behavior of pprint documented to allay your concerns.
I think it's harder than you yet realize: dicts don't require that keys be orderable, so pprint can't promise to sort dict displays. As-is, pprint simply blows up if, e.g., you pass it a "big dict" using complex numbers as keys. Could be nobody has noticed that yet, but since the trend in Python is to raise exceptions for non-equality comparisons of objects of different types, this kind of failure is bound to become more common.
That's fine. In practice, when we do this sort of thing, the keys are almost always strings.
In the end, I wouldn't be surprised if pprint(dict) got changed to sort on the repr of the keys instead of on the keys themselves.
That wouldn't be bad. Anyway, I guess we should make an issue of this on python-dev, so that either we can count on documented behavior going forward or so that we write our pwn pretty printer for use in doctest. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jim Fulton wrote:
Tim Peters wrote: ...
Well, I understand why that works, but it's not part of pprint's contract either.
What contract. :)
The documented behavior, e.g. from '$ pydoc pprint' or the online equivalent, http://python.org/doc/current/lib/module-pprint.html ? Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDpunX+gerLs4ltQ4RAoyfAJ9rRJHNhppGcNebCC94JOnHv+R/cQCggL9R Fejtd3mKGZzbE8XFWhf8SJg= =8arR -----END PGP SIGNATURE-----
Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Jim Fulton wrote:
Tim Peters wrote: ...
Well, I understand why that works, but it's not part of pprint's contract either.
What contract. :)
The documented behavior, e.g. from '$ pydoc pprint' or the online equivalent, http://python.org/doc/current/lib/module-pprint.html ?
The Python documentation is incomplete in lots of places. It can hardly be considered a complete contract. Jim -- Jim Fulton mailto:jim@zope.com Python Powered! CTO (540) 361-1714 http://www.python.org Zope Corporation http://www.zope.com http://www.zope.org
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Jim Fulton wrote:
Tres Seaver wrote:
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1
Jim Fulton wrote:
Tim Peters wrote: ...
Well, I understand why that works, but it's not part of pprint's contract either.
What contract. :)
The documented behavior, e.g. from '$ pydoc pprint' or the online equivalent, http://python.org/doc/current/lib/module-pprint.html ?
The Python documentation is incomplete in lots of places. It can hardly be considered a complete contract.
(Pulling numbers out of the air) ... 97% of Zope3 has less well-specified contracts than the Python standard library. Relying on implementation details beyond the documented API is risky, period. As Tim notes, there isn't even any guarantee that dictionary keys are *ordereable*. Tres. - -- =================================================================== Tres Seaver +1 202-558-7113 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.1 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://enigmail.mozdev.org iD8DBQFDpv3P+gerLs4ltQ4RAslWAKClKbqWIzHLobI1zb9+ciFhJgZkEACcDzQR htZpQ+RaaeBAeujqPQxG9nI= =B8fr -----END PGP SIGNATURE-----
... [Tim]
Well, I understand why that works, but it's not part of pprint's contract either.
[Jim]
What contract. :)
pprint's docs.
Aren't you always telling me to read the source?
Indeed, if you hadn't, you wouldn't have known that forcing width=1 forces dict sorting ;-) It's common as mud to close a bug report in Pythonland too when it's just griping about changes in undocumented behavior. ...
I think it's harder than you yet realize: dicts don't require that keys be orderable, so pprint can't promise to sort dict displays. As-is, pprint simply blows up if, e.g., you pass it a "big dict" using complex numbers as keys. Could be nobody has noticed that yet, but since the trend in Python is to raise exceptions for non-equality comparisons of objects of different types, this kind of failure is bound to become more common.
That's fine. In practice, when we do this sort of thing, the keys are almost always strings.
My point is that you're relying on undocumented behavior now, and so there's no guarantee it will continue to behave that way. To the contrary, because sorting dicts _started_ creating problems when Python gave up its original "any pair of objects can be compared" behavior, and Python continues moving away from that, there's reason to worry that pprint may give up sorting dict displays entirely. Sorting isn't free, is currently a source of bugs in pprint, and adds a large memory burden (to materialize a giant dict.items() list) when pprint'ing a large dict. An obvious alternative is for pprint to change to do dict.iteritems() when building a display, which is much better on all those counts. If I have anything to say about it, it will continue to sort, but nobody listens to me anymore ;-)
In the end, I wouldn't be surprised if pprint(dict) got changed to sort on the repr of the keys instead of on the keys themselves.
That wouldn't be bad.
It's debatable for sure.
Anyway, I guess we should make an issue of this on python-dev, so that either we can count on documented behavior going forward or so that we write our pwn pretty printer for use in doctest.
I'll try to make time for this next week (I'm on vacation then).
participants (6)
-
Fred Drake -
Jim Fulton -
Lennart Regebro -
Philipp von Weitershausen -
Tim Peters -
Tres Seaver