Python 2.7 and the ZTK - three test failures
Hi there, with Python 2.7 final being released, I ran the ZTK tests against it. zope.exceptions, zope.formlib and zope.proxy all have one test output related failure. RestrictedPython will need a security review. Any help with those appreciated, Hanno Full test output: Running test-ztk-zope.formlib test-ztk-zope.formlib failed with: Running zope.testing.testrunner.layer.UnitTests tests: Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds. Failure in test /opt/eggs/zope.formlib-4.0.3-py2.7.egg/zope/formlib/tests/../form.txt Traceback (most recent call last): File "/opt/python/parts/opt/lib/python2.7/unittest/case.py", line 318, in run testMethod() File "/opt/python/parts/opt/lib/python2.7/doctest.py", line 2153, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for form.txt File "/opt/eggs/zope.formlib-4.0.3-py2.7.egg/zope/formlib/tests/../form.txt", line 0 ---------------------------------------------------------------------- File "/opt/eggs/zope.formlib-4.0.3-py2.7.egg/zope/formlib/tests/../form.txt", line 217, in form.txt Failed example: MyForm(None, request)() # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS Expected: There were errors: (u'Invalid floating point data', <exceptions.ValueError instance at ...>) ('max_size', u'Maximum size', RequiredMissing('max_size')) <input class="textType" id="form.name" name="form.name" size="20" type="text" value="bob" /> <input class="textType" id="form.min_size" name="form.min_size" size="10" type="text" value="bob" /> <span class="error">Invalid floating point data</span> <input class="textType" id="form.max_size" name="form.max_size" size="10" type="text" value="" /> <span class="error">Required input is missing.</span> <input class="textType" id="form.color" name="form.color" size="20" type="text" value="" /> {'name': u'bob'} Got: There were errors: (u'Invalid floating point data', ValueError('could not convert string to float: bob',)) ('max_size', u'Maximum size', RequiredMissing('max_size')) <input class="textType" id="form.name" name="form.name" size="20" type="text" value="bob" /> <input class="textType" id="form.min_size" name="form.min_size" size="10" type="text" value="bob" /> <span class="error">Invalid floating point data</span> <input class="textType" id="form.max_size" name="form.max_size" size="10" type="text" value="" /> <span class="error">Required input is missing.</span> <input class="textType" id="form.color" name="form.color" size="20" type="text" value="" /> {'name': u'bob'} Ran 398 tests with 1 failures and 0 errors in 2.037 seconds. Tearing down left over layers: Tear down zope.testing.testrunner.layer.UnitTests in 0.000 seconds. Running test-ztk-zope.proxy test-ztk-zope.proxy failed with: Running zope.testing.testrunner.layer.UnitTests tests: Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds. Failure in test test_setProxiedObject (zope.proxy.tests.test_proxy) Traceback (most recent call last): File "/opt/python/parts/opt/lib/python2.7/unittest/case.py", line 318, in run testMethod() File "/opt/python/parts/opt/lib/python2.7/doctest.py", line 2153, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for zope.proxy.tests.test_proxy.test_setProxiedObject File "/opt/eggs/zope.proxy-3.6.0-py2.7-macosx-10.6-x86_64.egg/zope/proxy/tests/test_proxy.py", line 683, in test_setProxiedObject ---------------------------------------------------------------------- File "/opt/eggs/zope.proxy-3.6.0-py2.7-macosx-10.6-x86_64.egg/zope/proxy/tests/test_proxy.py", line 709, in zope.proxy.tests.test_proxy.test_setProxiedObject Failed example: setProxiedObject(c1, None) Expected: Traceback (most recent call last): TypeError: setProxiedObject() argument 1 must be zope.proxy.ProxyBase, not C Got: Traceback (most recent call last): File "/opt/python/parts/opt/lib/python2.7/doctest.py", line 1248, in __run compileflags, 1) in test.globs File "<doctest zope.proxy.tests.test_proxy.test_setProxiedObject[9]>", line 1, in <module> setProxiedObject(c1, None) TypeError: must be zope.proxy.ProxyBase, not C Ran 32 tests with 1 failures and 0 errors in 0.026 seconds. Tearing down left over layers: Tear down zope.testing.testrunner.layer.UnitTests in 0.000 seconds. Running test-ztk-zope.exceptions test-ztk-zope.exceptions failed with: Running zope.testing.testrunner.layer.UnitTests tests: Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds. Failure in test testMultilineException (zope.exceptions.tests.test_exceptionformatter.Test) Traceback (most recent call last): File "/opt/python/parts/opt/lib/python2.7/unittest/case.py", line 318, in run testMethod() File "/opt/eggs/zope.exceptions-3.6.0-py2.7.egg/zope/exceptions/tests/test_exceptionformatter.py", line 160, in testMultilineException 'SyntaxError: unexpected EOF while parsing']) File "/opt/python/parts/opt/lib/python2.7/unittest/case.py", line 494, in assertEqual assertion_func(first, second, msg=msg) File "/opt/python/parts/opt/lib/python2.7/unittest/case.py", line 723, in assertListEqual self.assertSequenceEqual(list1, list2, msg, seq_type=list) File "/opt/python/parts/opt/lib/python2.7/unittest/case.py", line 705, in assertSequenceEqual self.fail(msg) File "/opt/python/parts/opt/lib/python2.7/unittest/case.py", line 393, in fail raise self.failureException(msg) AssertionError: Lists differ: [' syntax error', ' ... != [' syntax error', ' ... First differing element 2: SyntaxError: invalid syntax SyntaxError: unexpected EOF while parsing - [' syntax error', ' ^', 'SyntaxError: invalid syntax'] + [' syntax error', + ' ^', + 'SyntaxError: unexpected EOF while parsing'] Ran 10 tests with 1 failures and 0 errors in 0.003 seconds. Tearing down left over layers: Tear down zope.testing.testrunner.layer.UnitTests in 0.000 seconds.
On 2010-7-5 23:08, Hanno Schlichting wrote:
Hi there,
with Python 2.7 final being released, I ran the ZTK tests against it.
zope.exceptions, zope.formlib and zope.proxy all have one test output related failure.
We seem to have a lot of problems with doctests breaking due to changes in exception formatting. Perhaps a policy to check exceptions using a try/except or self.assertRaises instead of relying on exact formatting would be useful? Wichert. -- Wichert Akkerman <wichert@wiggy.net> It is simple to make things. http://www.wiggy.net/ It is hard to make things simple.
On 07/06/2010 09:46 AM, Wichert Akkerman wrote:
On 2010-7-5 23:08, Hanno Schlichting wrote:
Hi there,
with Python 2.7 final being released, I ran the ZTK tests against it.
zope.exceptions, zope.formlib and zope.proxy all have one test output related failure.
We seem to have a lot of problems with doctests breaking due to changes in exception formatting. Perhaps a policy to check exceptions using a try/except or self.assertRaises instead of relying on exact formatting would be useful?
I wish the Python developers saw the exception messages as a core part of their interface, but I guess they won't. While I acknowledge its fragility, I do like the way doctests express exceptions quite a lot, so I'd prefer something that kept them like that. We've used normalization in the past, so if core Python exception messages changed I'd imagine more normalization could help again? Anyway, since we've only had a few test failures the problems with changing message formatting seems to be limited, right? Regards, Martijn
Am 06.07.2010, 12:23 Uhr, schrieb Martijn Faassen <faassen@startifact.com>:
While I acknowledge its fragility, I do like the way doctests express exceptions quite a lot, so I'd prefer something that kept them like that. We've used normalization in the past, so if core Python exception messages changed I'd imagine more normalization could help again? Anyway, since we've only had a few test failures the problems with changing message formatting seems to be limited, right?
I think this has come up before especially in reference to Lennart's port to Python 3.x. Any test that depends upon exception formatting is likely to be a problem. In the specific zope.formlib test we have a doctest embedded within another: >>> MyForm(None, request)() # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS There were errors: (u'Invalid floating point data', <exceptions.ValueError instance at ...>) I think this is bound to cause problems sooner or later in any environment. The test here is whether, and not how, the constraint is fulfilled and the nature of the exception is irrelevant. The more common
u'Invalid floating point data' in browser.contents
would be a better solution, I think. Furthermore, while it's great that form.txt actually runs I wasn't aware that it contained any additional tests that are not already run and I've always treated it as testable documentation not as an integral part of the formlib tests. But I'm ready to be believe this is a large misconception on my part. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226
Hey, On 07/06/2010 01:04 PM, Charlie Clark wrote:
In the specific zope.formlib test we have a doctest embedded within another:
>>> MyForm(None, request)() # doctest: +NORMALIZE_WHITESPACE +ELLIPSIS There were errors: (u'Invalid floating point data', <exceptions.ValueError instance at ...>)
What do you mean, a doctest embedded within another? I'm probably missing something.
Furthermore, while it's great that form.txt actually runs I wasn't aware that it contained any additional tests that are not already run and I've always treated it as testable documentation not as an integral part of the formlib tests. But I'm ready to be believe this is a large misconception on my part.
I follow the principle that all testable stuff should actually be run during the tests - just in case. :) Regards, Martijn
Am 06.07.2010, 13:12 Uhr, schrieb Martijn Faassen <faassen@startifact.com>:
What do you mean, a doctest embedded within another? I'm probably missing something.
No, it's probably me getting the explicit doctest call wrong. It looks to my novice eyes like print statement is being passed to a "doctest" method. In matters like these it's usually safe to assume I'm wrong! :-) Proposed solution: rewrite this particular doctest to use a test browser.
Furthermore, while it's great that form.txt actually runs I wasn't aware that it contained any additional tests that are not already run and I've always treated it as testable documentation not as an integral part of the formlib tests. But I'm ready to be believe this is a large misconception on my part.
I follow the principle that all testable stuff should actually be run during the tests - just in case.
hm, I don't think that can be argued with really, particularly given the amount of time I've actually studied this and other documents over reading the code. But I do think that, whether the module runs as specified, and whether the documentation is up to snuff, are of a different nature and, consequently, so are their failures. Shouldn't we be testing documentation differently? Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226
On 07/06/2010 01:27 PM, Charlie Clark wrote:
Am 06.07.2010, 13:12 Uhr, schrieb Martijn Faassen<faassen@startifact.com>:
What do you mean, a doctest embedded within another? I'm probably missing something.
No, it's probably me getting the explicit doctest call wrong. It looks to my novice eyes like print statement is being passed to a "doctest" method. In matters like these it's usually safe to assume I'm wrong! :-)
No, that comment (if you mean that?) is just a way to send configuration parameters to the doctest engine. It calls the view.
hm, I don't think that can be argued with really, particularly given the amount of time I've actually studied this and other documents over reading the code. But I do think that, whether the module runs as specified, and whether the documentation is up to snuff, are of a different nature and, consequently, so are their failures. Shouldn't we be testing documentation differently?
I think a failure in documentation should ideally show up when running the tests. I do agree that documentation and tests have different concerns and that mixing them can (but does not necessarily have to) lead to poor documentation. Regards, Martijn
Hi again. On Mon, Jul 5, 2010 at 11:08 PM, Hanno Schlichting <hanno@hannosch.eu> wrote:
with Python 2.7 final being released, I ran the ZTK tests against it.
zope.exceptions, zope.formlib and zope.proxy all have one test output related failure.
I fixed all of those and released new versions. bin/test-ztk should now pass under Python 2.7.
RestrictedPython will need a security review.
I hear David Glick has started to look into this. Applause! Running bin/test-zopeapp still gives failures in two packages: zope.app.publisher and zope.app.wsgi. I'm not familiar enough with that code to fix any of those. Hanno Full test output: Running test-zopeapp-zope.app.publisher test-zopeapp-zope.app.publisher failed with: Running zope.app.publisher.testing.AppPublisherLayer tests: Set up zope.app.publisher.testing.AppPublisherLayer in 1.360 seconds. Failure in test /opt/eggs/zope.app.publisher-3.10.1-py2.7.egg/zope/app/publisher/xmlrpc/tests/../README.txt Traceback (most recent call last): File "/opt/python/parts/opt/lib/python2.7/unittest/case.py", line 318, in run testMethod() File "/opt/python/parts/opt/lib/python2.7/doctest.py", line 2153, in runTest raise self.failureException(self.format_failure(new.getvalue())) AssertionError: Failed doctest test for README.txt File "/opt/eggs/zope.app.publisher-3.10.1-py2.7.egg/zope/app/publisher/xmlrpc/tests/../README.txt", line 0 ---------------------------------------------------------------------- File "/opt/eggs/zope.app.publisher-3.10.1-py2.7.egg/zope/app/publisher/xmlrpc/tests/../README.txt", line 65, in README.txt Failed example: proxy.contents() Exception raised: Traceback (most recent call last): File "/opt/python/parts/opt/lib/python2.7/doctest.py", line 1248, in __run compileflags, 1) in test.globs File "<doctest README.txt[7]>", line 1, in <module> proxy.contents() File "/opt/python/parts/opt/lib/python2.7/xmlrpclib.py", line 1224, in __call__ return self.__send(self.__name, args) File "/opt/python/parts/opt/lib/python2.7/xmlrpclib.py", line 1570, in __request verbose=self.__verbose File "/opt/eggs/zope.app.testing-3.7.5-py2.7.egg/zope/app/testing/xmlrpc.py", line 62, in request return self._parse_response( AttributeError: ZopeTestTransport instance has no attribute '_parse_response' [... snip lots more failures with the same problem ...] Ran 1 tests with 1 failures and 0 errors in 0.163 seconds. Running zope.testing.testrunner.layer.UnitTests tests: Tear down zope.app.publisher.testing.AppPublisherLayer in 0.001 seconds. Set up zope.testing.testrunner.layer.UnitTests in 0.000 seconds. Ran 11 tests with 0 failures and 0 errors in 0.089 seconds. Tearing down left over layers: Tear down zope.testing.testrunner.layer.UnitTests in 0.000 seconds. Total: 12 tests, 1 failures, 0 errors in 1.791 seconds. Running test-zopeapp-zope.app.wsgi test-zopeapp-zope.app.wsgi failed with: Running zope.app.wsgi.BrowserLayer tests: Set up zope.app.wsgi.BrowserLayer in 0.222 seconds. Failure in test test_file_returns (zope.app.wsgi.tests) Failed doctest test for zope.app.wsgi.tests.test_file_returns File "/opt/eggs/zope.app.wsgi-3.9.2-py2.7.egg/zope/app/wsgi/tests.py", line 61, in test_file_returns ---------------------------------------------------------------------- File "/opt/eggs/zope.app.wsgi-3.9.2-py2.7.egg/zope/app/wsgi/tests.py", line 77, in zope.app.wsgi.tests.test_file_returns Failed example: browser.open('http://localhost/@@test-file-view.html') Exception raised: Traceback (most recent call last): File "/opt/eggs/zope.testing-3.9.4-py2.7.egg/zope/testing/doctest/__init__.py", line 1355, in __run compileflags, 1) in test.globs File "<doctest zope.app.wsgi.tests.test_file_returns[line 16, example 6]>", line 1, in <module> browser.open('http://localhost/@@test-file-view.html') File "/opt/eggs/zope.testbrowser-3.9.0-py2.7.egg/zope/testbrowser/browser.py", line 237, in open self.mech_browser.open(url, data) File "/opt/eggs/mechanize-0.2.1-py2.7.egg/mechanize/_mechanize.py", line 204, in open return self._mech_open(url, data, timeout=timeout) File "/opt/eggs/mechanize-0.2.1-py2.7.egg/mechanize/_mechanize.py", line 231, in _mech_open response = UserAgentBase.open(self, request, data) File "/opt/eggs/mechanize-0.2.1-py2.7.egg/mechanize/_opener.py", line 193, in open response = urlopen(self, req, data) File "/opt/eggs/mechanize-0.2.1-py2.7.egg/mechanize/_urllib2_fork.py", line 344, in _open '_open', req) File "/opt/eggs/mechanize-0.2.1-py2.7.egg/mechanize/_urllib2_fork.py", line 332, in _call_chain result = func(*args) File "/opt/eggs/wsgi_intercept-0.4-py2.7.egg/wsgi_intercept/urllib2_intercept/wsgi_urllib2.py", line 34, in http_open return self.do_open(WSGI_HTTPConnection, req) File "/opt/python/parts/opt/lib/python2.7/urllib2.py", line 1142, in do_open h.request(req.get_method(), req.get_selector(), req.data, headers) File "/opt/python/parts/opt/lib/python2.7/httplib.py", line 946, in request self._send_request(method, url, body, headers) File "/opt/python/parts/opt/lib/python2.7/httplib.py", line 986, in _send_request self.putheader(hdr, value) File "/opt/python/parts/opt/lib/python2.7/httplib.py", line 924, in putheader str = '%s: %s' % (header, '\r\n\t'.join(values)) TypeError: sequence item 0: expected string, bool found ---------------------------------------------------------------------- File "/opt/eggs/zope.app.wsgi-3.9.2-py2.7.egg/zope/app/wsgi/tests.py", line 78, in zope.app.wsgi.tests.test_file_returns Failed example: browser.headers['content-type'] Exception raised: Traceback (most recent call last): File "/opt/eggs/zope.testing-3.9.4-py2.7.egg/zope/testing/doctest/__init__.py", line 1355, in __run compileflags, 1) in test.globs File "<doctest zope.app.wsgi.tests.test_file_returns[line 17, example 7]>", line 1, in <module> browser.headers['content-type'] File "/opt/eggs/zope.testbrowser-3.9.0-py2.7.egg/zope/testbrowser/browser.py", line 208, in headers return self.mech_browser.response().info() AttributeError: 'NoneType' object has no attribute 'info' ---------------------------------------------------------------------- File "/opt/eggs/zope.app.wsgi-3.9.2-py2.7.egg/zope/app/wsgi/tests.py", line 81, in zope.app.wsgi.tests.test_file_returns Failed example: browser.headers['content-length'] Exception raised: Traceback (most recent call last): File "/opt/eggs/zope.testing-3.9.4-py2.7.egg/zope/testing/doctest/__init__.py", line 1355, in __run compileflags, 1) in test.globs File "<doctest zope.app.wsgi.tests.test_file_returns[line 20, example 8]>", line 1, in <module> browser.headers['content-length'] File "/opt/eggs/zope.testbrowser-3.9.0-py2.7.egg/zope/testbrowser/browser.py", line 208, in headers return self.mech_browser.response().info() AttributeError: 'NoneType' object has no attribute 'info' ---------------------------------------------------------------------- File "/opt/eggs/zope.app.wsgi-3.9.2-py2.7.egg/zope/app/wsgi/tests.py", line 84, in zope.app.wsgi.tests.test_file_returns Failed example: print browser.contents Expected: Hello World! <BLANKLINE> Got: None Ran 2 tests with 1 failures and 0 errors in 0.027 seconds. Running zope.app.wsgi.ZCMLFileLayer tests: Tear down zope.app.wsgi.BrowserLayer in 0.000 seconds. Set up zope.app.wsgi.ZCMLFileLayer in 0.091 seconds. Ran 19 tests with 0 failures and 0 errors in 0.288 seconds. Tearing down left over layers: Tear down zope.app.wsgi.ZCMLFileLayer in 0.000 seconds. Total: 21 tests, 1 failures, 0 errors in 0.753 seconds.
participants (4)
-
Charlie Clark -
Hanno Schlichting -
Martijn Faassen -
Wichert Akkerman