Hi All, I'm suffering with two different types of UnicodeDecodeError with a Zope 2.12 project... The first type of problem occurs when a form is submitted and the form fields contain encoded strings. Somewhere down the line I get: Module zope.i18n, line 166, in interpolate Module zope.i18n, line 161, in replace UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5: ordinal not in range(128) So, my question is how do I get strings submitted to the publisher (ideally) end up as unicode objects not encoded strings? Failing that, how do I find out what encoding the browser has provided? The second type of problem only occurs in IE and Safari, where viewing a page that contains some non-ascii-able data: Module ZPublisher.Publish, line 127, in publish Module ZPublisher.mapply, line 77, in mapply Module ZPublisher.Publish, line 47, in call_object Module Products.Five.browser.metaconfigure, line 427, in __call__ Module Products.Five.browser.pagetemplatefile, line 126, in __call__ Module Products.Five.browser.pagetemplatefile, line 60, in __call__ Module zope.pagetemplate.pagetemplate, line 116, in pt_render - Warning: Macro expansion failed - Warning: <class 'zope.tal.taldefs.METALError'>: macro nesting limit (100) exceeded by u'root/standard_template.pt/macros/nav' Module StringIO, line 270, in getvalue UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 10: ordinal not in range(128) Two things are weird here: - Why am I getting a macro nesting limit? This seems to be a recurring problem (excuse the pun) with Zope 2.12 page templates where if there's an error in a standard template, I get this error when a template that uses it is rendered. - Why am I getting this decoding error only with these browsers? Firefox and Chrome work just fine... Is the publisher doing something special with a header these browsers send that IE and Safari are not? If so, how do I make it do the "right thing" for IE and Safari? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
Hi Chris. 2009/8/22 Chris Withers <chris@simplistix.co.uk>:
The first type of problem occurs when a form is submitted and the form fields contain encoded strings. Somewhere down the line I get:
The first type, I don't know.
The second type of problem only occurs in IE and Safari, where viewing a page that contains some non-ascii-able data:
The second type, (maybe) similer probrem reported to launchpad. https://bugs.launchpad.net/zope2/+bug/160968
- Why am I getting this decoding error only with these browsers? Firefox and Chrome work just fine... Is the publisher doing something special with a header these browsers send that IE and Safari are not? If so, how do I make it do the "right thing" for IE and Safari?
Are concerned that HTTP_ACCEPT_CHARSET http header will probably not be sent from client browser (IE6,7, Safari). When HTTP_ACCEPT_CHARSET not in HTTP request headers, zope.publisher use UTF-8 encoding for publishing. regards. -- Takayuki Shimizukawa
Takayuki Shimizukawa wrote:
2009/8/22 Chris Withers <chris@simplistix.co.uk>:
The first type of problem occurs when a form is submitted and the form fields contain encoded strings. Somewhere down the line I get:
The first type, I don't know.
The second type of problem only occurs in IE and Safari, where viewing a page that contains some non-ascii-able data:
The second type, (maybe) similer probrem reported to launchpad. https://bugs.launchpad.net/zope2/+bug/160968
- Why am I getting this decoding error only with these browsers? Firefox and Chrome work just fine... Is the publisher doing something special with a header these browsers send that IE and Safari are not? If so, how do I make it do the "right thing" for IE and Safari?
Are concerned that HTTP_ACCEPT_CHARSET http header will probably not be sent from client browser (IE6,7, Safari). When HTTP_ACCEPT_CHARSET not in HTTP request headers, zope.publisher use UTF-8 encoding for publishing.
Well, I just solved the problem with a large setdefaultencoding hammer :-) Annoyingly, this required even more evil, since sitecustomize.py is no longer found anywhere sensible in Python 2.5, so I ended up with a buildout that looks like the following: [buildout] parts = instance extends = http://svn.zope.org/*checkout*/Zope/tags/2.12.0b4/versions.cfg?rev=102515 versions = versions [versions] zope2 = 2.12.0b4 [instance] recipe = zc.recipe.egg eggs = ${buildout:eggs} interpreter = py entry-points= runzope=Zope2.Startup.run:run zopectl=Zope2.Startup.zopectl:main scripts = runzope zopectl initialization = import sys reload(sys) sys.setdefaultencoding('utf-8') sys.argv[1:1] = ['-C','${buildout:directory}/etc/instance.conf'] Evil huh? ;-) Hope this helps someone... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
Chris Withers wrote:
Annoyingly, this required even more evil, since sitecustomize.py is no longer found anywhere sensible in Python 2.5,
correction: it never was, but prior to Zope 2.12, the INSTANCE_HOME ended up on PYTHONPATH and so sitecustomize.py could be put there. Using the buildout approach for Zope 2.12, it not longer is on PYTHONPATH, so it doesn't get found... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
There is a related bug report on LP (with a controversial discussion) about the issue (can't find the URL right now). Andreas On 21.08.09 21:40, Chris Withers wrote:
Hi All,
I'm suffering with two different types of UnicodeDecodeError with a Zope 2.12 project...
The first type of problem occurs when a form is submitted and the form fields contain encoded strings. Somewhere down the line I get:
Module zope.i18n, line 166, in interpolate Module zope.i18n, line 161, in replace UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 5: ordinal not in range(128)
So, my question is how do I get strings submitted to the publisher (ideally) end up as unicode objects not encoded strings? Failing that, how do I find out what encoding the browser has provided?
The second type of problem only occurs in IE and Safari, where viewing a page that contains some non-ascii-able data:
Module ZPublisher.Publish, line 127, in publish Module ZPublisher.mapply, line 77, in mapply Module ZPublisher.Publish, line 47, in call_object Module Products.Five.browser.metaconfigure, line 427, in __call__ Module Products.Five.browser.pagetemplatefile, line 126, in __call__ Module Products.Five.browser.pagetemplatefile, line 60, in __call__ Module zope.pagetemplate.pagetemplate, line 116, in pt_render - Warning: Macro expansion failed - Warning: <class 'zope.tal.taldefs.METALError'>: macro nesting limit (100) exceeded by u'root/standard_template.pt/macros/nav' Module StringIO, line 270, in getvalue UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 10: ordinal not in range(128)
Two things are weird here:
- Why am I getting a macro nesting limit? This seems to be a recurring problem (excuse the pun) with Zope 2.12 page templates where if there's an error in a standard template, I get this error when a template that uses it is rendered.
- Why am I getting this decoding error only with these browsers? Firefox and Chrome work just fine... Is the publisher doing something special with a header these browsers send that IE and Safari are not? If so, how do I make it do the "right thing" for IE and Safari?
cheers,
Chris
-- ZOPYX Ltd. & Co KG \ ZOPYX & Friends Charlottenstr. 37/1 \ The experts for your Python, Zope and D-72070 Tübingen \ Plone projects www.zopyx.com, info@zopyx.com \ www.zopyx.de/friends, friends@zopyx.de ------------------------------------------------------------------------ E-Publishing, Python, Zope & Plone development, Consulting
Am 30.08.2009, 12:51 Uhr, schrieb Andreas Jung <lists@zopyx.com>:
- Why am I getting this decoding error only with these browsers? Firefox and Chrome work just fine... Is the publisher doing something special with a header these browsers send that IE and Safari are not? If so, how do I make it do the "right thing" for IE and Safari?
I thought this had been fixed? I'm pretty sure it's related to 'HTTP_ACCEPT_CHARSET' missing in the headers from IE and Safari and the charset detector incorrectly extracting it from the REQUEST which is always True for Zope 2. zope.publisher.http.getPreferredCharsets() is where this magic happens. Charlie -- Charlie Clark Managing Director Clark Consulting & Research Ltd. German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 GSM: +49-178-782-6226 Company No. 06770088
On 30.08.09 13:01, Charlie Clark wrote:
Am 30.08.2009, 12:51 Uhr, schrieb Andreas Jung <lists@zopyx.com>:
- Why am I getting this decoding error only with these browsers? Firefox and Chrome work just fine... Is the publisher doing something special with a header these browsers send that IE and Safari are not? If so, how do I make it do the "right thing" for IE and Safari?
I thought this had been fixed? I'm pretty sure it's related to 'HTTP_ACCEPT_CHARSET' missing in the headers from IE and Safari and the charset detector incorrectly extracting it from the REQUEST which is always True for Zope 2.
zope.publisher.http.getPreferredCharsets() is where this magic happens.
Possibly fixed in new versions of zope.publisher. I think there was some open issue with backporting some fix - but my brain can't remember :-> Andreas
Am 30.08.2009, 13:26 Uhr, schrieb Andreas Jung <lists@zopyx.com>:
Possibly fixed in new versions of zope.publisher. I think there was some open issue with backporting some fix - but my brain can't remember :->
See https://bugs.launchpad.net/zope2/+bug/160968 and https://bugs.launchpad.net/bugs/143873 So, it should be in the current code-base and shouldn't be happening at all in Zope 2.12. So it could be something else. I only know about it because I keep on it hitting it in Windows installs as there isn't a binary of 2.11 with the fix. Charlie -- Charlie Clark Managing Director Clark Consulting & Research Ltd. German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 GSM: +49-178-782-6226 Company No. 06770088
On 30.08.09 13:58, Charlie Clark wrote:
Am 30.08.2009, 13:26 Uhr, schrieb Andreas Jung <lists@zopyx.com>:
Possibly fixed in new versions of zope.publisher. I think there was some open issue with backporting some fix - but my brain can't remember :->
See https://bugs.launchpad.net/zope2/+bug/160968 and https://bugs.launchpad.net/bugs/143873 No, I am referring to a controversial discussion with Dieter - however I really can not find it anymore. Anyway...my conclusion from this discussion was that a custom unicode conflict resolver should be used for resolving such custom problems (unless with a real bug in the processing of the http headers). A clean solution would be to introduce the notion of a default charset for a site manager (as the 'default_charset' property of CMF sites or Plone sites). A unicode conflict resolver could get hold of the (local) site manager and check for a (registered) local utiltiy return the charset through some API...something like that (instead of guessing/sniffing in the HTTP client headers).
Andreas
Hi Charlie,
I only know about it because I keep on it hitting it in Windows installs as there isn't a binary of 2.11 with the fix.
You mean the latest binary doesn't have the fix? Or you haven't seen that a binary was released recently? http://www.zope.org/Products/Zope/2.11.4/ -- Sidnei
Am 30.08.2009, 19:16 Uhr, schrieb Sidnei da Silva <sidnei.da.silva@gmail.com>:
You mean the latest binary doesn't have the fix? Or you haven't seen that a binary was released recently? http://www.zope.org/Products/Zope/2.11.4/
ah, I hadn't seen that one - my last Windows install was in July. Will give a try this week and am very much looking forward to removing my hackish override. Charlie -- Charlie Clark Managing Director Clark Consulting & Research Ltd. German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 GSM: +49-178-782-6226 Company No. 06770088
participants (5)
-
Andreas Jung -
Charlie Clark -
Chris Withers -
Sidnei da Silva -
Takayuki Shimizukawa