UnicodeDecodeError from PageTemplate rendering after upgrading to Zope 2.12.7
Hi All, After a recent upgraded to Zope 2.12, I'm now seeing errors like the following when using IE (version 8) or Safari (version 4.0.3) to view page templates. Chrome and Firefox work fine. Traceback (innermost last): Module ZPublisher.Publish, line 127, in publish Module ZPublisher.mapply, line 77, in mapply Module ZPublisher.Publish, line 47, in call_object <snip> Module Shared.DC.Scripts.Bindings, line 324, in __call__ Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec Module Products.PageTemplates.ZopePageTemplate, line 335, in _exec Module Products.PageTemplates.ZopePageTemplate, line 432, in pt_render Module Products.PageTemplates.PageTemplate, line 80, in pt_render Module zope.pagetemplate.pagetemplate, line 115, in pt_render Module zope.tal.talinterpreter, line 271, in __call__ <snip> Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 751, in do_insertStructure_tal UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 366: ordinal not in range(128) Has anyone seen this or anything similar before? Any solutions spring to mind? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
On Tue, Aug 31, 2010 at 11:40 AM, Chris Withers <chris@simplistix.co.uk>wrote:
Hi All,
After a recent upgraded to Zope 2.12, I'm now seeing errors like the following when using IE (version 8) or Safari (version 4.0.3) to view page templates. Chrome and Firefox work fine.
Traceback (innermost last): Module ZPublisher.Publish, line 127, in publish Module ZPublisher.mapply, line 77, in mapply Module ZPublisher.Publish, line 47, in call_object <snip> Module Shared.DC.Scripts.Bindings, line 324, in __call__ Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec Module Products.PageTemplates.ZopePageTemplate, line 335, in _exec Module Products.PageTemplates.ZopePageTemplate, line 432, in pt_render Module Products.PageTemplates.PageTemplate, line 80, in pt_render Module zope.pagetemplate.pagetemplate, line 115, in pt_render Module zope.tal.talinterpreter, line 271, in __call__ <snip> Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 751, in do_insertStructure_tal UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 366: ordinal not in range(128)
Has anyone seen this or anything similar before? Any solutions spring to mind?
Hi Chris. No solution, actually, but I would suggest to sniff the traffic between you and the server; I suspect different headers cause problems, here. Regards Marco
cheers,
Chris
-- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
-- Marco Bizzarri http://code.google.com/p/qt-asterisk/ http://notenotturne.blogspot.com/ http://iliveinpisa.blogspot.com/
Marco Bizzarri wrote:
No solution, actually, but I would suggest to sniff the traffic between you and the server; I suspect different headers cause problems, here.
Of that I have no doubt. What does Zope2's publisher do with requested encodings and the like now? Why would this be causing problems? Errors that affect all browsers I'd be inclined to attribute to my code, ones that randomly affect specific browsers appear more likely to be a bug in Zope for me... Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
Am 31.08.2010, 12:05 Uhr, schrieb Chris Withers <chris@simplistix.co.uk>:
What does Zope2's publisher do with requested encodings and the like now? Why would this be causing problems?
I can't remember exactly where but preferred_charset is looked for. It isn't set by either IE or Safari and, although it should default to UTF-8 it seems they get served Latin 1 and you get badda boom pretty quickly.
Errors that affect all browsers I'd be inclined to attribute to my code, ones that randomly affect specific browsers appear more likely to be a bug in Zope for me...
It is and I thought it had been squashed. Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226
Charlie Clark wrote:
Am 31.08.2010, 12:05 Uhr, schrieb Chris Withers <chris@simplistix.co.uk>:
What does Zope2's publisher do with requested encodings and the like now? Why would this be causing problems?
I can't remember exactly where but preferred_charset is looked for. It isn't set by either IE or Safari and, although it should default to UTF-8 it seems they get served Latin 1 and you get badda boom pretty quickly.
That's weird, the content in this app should be fine to be served latin-1.
Errors that affect all browsers I'd be inclined to attribute to my code, ones that randomly affect specific browsers appear more likely to be a bug in Zope for me...
It is and I thought it had been squashed.
Any idea what I should be looking for and in what changelog? Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
Am 31.08.2010, 12:51 Uhr, schrieb Chris Withers <chris@simplistix.co.uk>:
Any idea what I should be looking for and in what changelog?
This my horribly patched version of charset.py which we just use via an override. I know it's wrong but it was the easiest thing to do at a time. class HTTPCharsets(object): """This is a complete copy of zope.publisher.http.HTTPCharset except if there is no charset specified. Zope defaults to ISO-8859-1 and we use the portal default_charset""" adapts(IHTTPRequest) implements(IUserPreferredCharsets) def __init__(self, request): self.request = request def getPreferredCharsets(self): '''See interface IUserPreferredCharsets''' charsets = [] sawstar = sawiso88591 = 0 header_present = 'HTTP_ACCEPT_CHARSET' in self.request for charset in self.request.get('HTTP_ACCEPT_CHARSET', '').split(','): charset = charset.strip().lower() if charset: if ';' in charset: charset, quality = charset.split(';') if not quality.startswith('q='): # not a quality parameter quality = 1.0 else: try: quality = float(quality[2:]) except ValueError: continue else: quality = 1.0 if quality == 0.0: continue if charset == '*': sawstar = 1 if charset == 'iso-8859-1': sawiso88591 = 1 charsets.append((quality, charset)) # Quoting RFC 2616, $14.2: If no "*" is present in an Accept-Charset # field, then all character sets not explicitly mentioned get a # quality value of 0, except for ISO-8859-1, which gets a quality # value of 1 if not explicitly mentioned. # And quoting RFC 2616, $14.2: "If no Accept-Charset header is # present, the default is that any character set is acceptable." if not sawstar and not sawiso88591 and header_present: ptool = getUtility(IPropertiesTool) default_charset = ptool.getProperty('default_charset', None) charsets.append((1.0, default_charset)) # charsets.append((1.0, 'iso-8859-1')) # UTF-8 is **always** preferred over anything else. # Reason: UTF-8 is not specific and can encode the entire unicode # range , unlike many other encodings. Since Zope can easily use very # different ranges, like providing a French-Chinese dictionary, it is # always good to use UTF-8. # charsets.sort(sort_charsets) charsets = [charset for quality, charset in charsets] if sawstar and charsets[0] != 'utf-8': if 'utf-8' in charsets: idx = charsets.index('utf-8') del charsets[idx] charsets.insert(0, 'utf-8') return charsets The related bug in zope.publisher may or may not have found its way into the current release. https://bugs.launchpad.net/zope2/+bug/143873 In addition, but I don't know how relevant this is in your case, Jens improved FSPageTemplate https://bugs.launchpad.net/zope-cmf/+bug/322263 Charlie -- Charlie Clark Managing Director Clark Consulting & Research German Office Helmholtzstr. 20 Düsseldorf D- 40215 Tel: +49-211-600-3657 Mobile: +49-178-782-6226
Maybe not the best solution but I use this hack: zope.tal-3.5.2-py2.7.egg/zope/tal/talinterpreter.py line 751 change: text = unicode(structure) to try: text = unicode(structure) except: text=unicode(structure.decode('utf-8')) Vlad 2010/8/31 Chris Withers <chris@simplistix.co.uk>:
Hi All,
After a recent upgraded to Zope 2.12, I'm now seeing errors like the following when using IE (version 8) or Safari (version 4.0.3) to view page templates. Chrome and Firefox work fine.
Traceback (innermost last): Module ZPublisher.Publish, line 127, in publish Module ZPublisher.mapply, line 77, in mapply Module ZPublisher.Publish, line 47, in call_object <snip> Module Shared.DC.Scripts.Bindings, line 324, in __call__ Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec Module Products.PageTemplates.ZopePageTemplate, line 335, in _exec Module Products.PageTemplates.ZopePageTemplate, line 432, in pt_render Module Products.PageTemplates.PageTemplate, line 80, in pt_render Module zope.pagetemplate.pagetemplate, line 115, in pt_render Module zope.tal.talinterpreter, line 271, in __call__ <snip> Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 751, in do_insertStructure_tal UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 366: ordinal not in range(128)
Has anyone seen this or anything similar before? Any solutions spring to mind?
cheers,
Chris
-- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk _______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org https://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope )
-- Best Regards Vlad Vorobiev
Vladislav Vorobiev wrote:
Maybe not the best solution but I use this hack:
zope.tal-3.5.2-py2.7.egg/zope/tal/talinterpreter.py
Indeed, certainly not a good solution. This implies that things should be unicode by the time they get here. Now, the decoding shouldn't be dependent on anything the browser sends, should it? I would have thought only the encoding side of things should matter.. ...so why on earth does the browser used have an effect on the decoding (or not, as seems to be the case with the bug) of the structure? (and where is this decoding done?) cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
Hi Chris, On 08/31/2010 11:40 AM, Chris Withers wrote:
After a recent upgraded to Zope 2.12, I'm now seeing errors like the following when using IE (version 8) or Safari (version 4.0.3) to view page templates. Chrome and Firefox work fine.
This may be related to IE and Safari not sending an accept-charset header and Zope then falling back to the Python default (often ASCII) for encoding, which then fails. I use the following patch to http.py (tested up to 2.12.3): --- lib/python/zope/publisher/http.py.orig 2009-05-02 00:49:35.000000000 -0700 +++ lib/python/zope/publisher/http.py 2009-06-12 02:12:50.000000000 -0700 @@ -965,8 +965,11 @@ # value of 1 if not explicitly mentioned. # And quoting RFC 2616, $14.2: "If no Accept-Charset header is # present, the default is that any character set is acceptable." - if not sawstar and not sawiso88591 and header_present: + # ROC: the following checked also for header_present which seemed wrong + if not sawstar and not sawiso88591: charsets.append((1.0, 'iso-8859-1')) + # ROC: if there is no header we assume star (to get utf-8) + sawstar = 1 # UTF-8 is **always** preferred over anything else. # Reason: UTF-8 is not specific and can encode the entire unicode # range , unlike many other encodings. Since Zope can easily use very Best Robert
On Tue, Aug 31, 2010 at 10:40:46AM +0100, Chris Withers wrote:
After a recent upgraded to Zope 2.12, I'm now seeing errors like the following when using IE (version 8) or Safari (version 4.0.3) to view page templates. Chrome and Firefox work fine.
Traceback (innermost last): Module ZPublisher.Publish, line 127, in publish Module ZPublisher.mapply, line 77, in mapply Module ZPublisher.Publish, line 47, in call_object <snip> Module Shared.DC.Scripts.Bindings, line 324, in __call__ Module Shared.DC.Scripts.Bindings, line 361, in _bindAndExec Module Products.PageTemplates.ZopePageTemplate, line 335, in _exec Module Products.PageTemplates.ZopePageTemplate, line 432, in pt_render Module Products.PageTemplates.PageTemplate, line 80, in pt_render Module zope.pagetemplate.pagetemplate, line 115, in pt_render Module zope.tal.talinterpreter, line 271, in __call__ <snip> Module zope.tal.talinterpreter, line 343, in interpret Module zope.tal.talinterpreter, line 751, in do_insertStructure_tal UnicodeDecodeError: 'ascii' codec can't decode byte 0xa0 in position 366: ordinal not in range(128)
Has anyone seen this or anything similar before?
Yes. Basically, every Zope site I've upgraded to 2.12 has suffered from this. Annoyingly this only happens with MSIE, so developers/admins tend not to notice. Clarifying the "only with MSIE" comment: this happens if and only if the HTTP request doesn't explicitly include 'UTF-8' in Accept-Charset. IIRC.
Any solutions spring to mind?
Magically sprinkle .encode() / .decode() in TAL expressions/Python scripts, where appropriate. Most of the data stored in the ZODB is UTF-8 raw strings (things like DTML Documents and DTML Methods don't support storing Unicode data). Page templates now suddenly (since Zope 2.11) refuse to deal with raw strings and insist on having Unicode. Which means you need to do explicit conversions not at your system boundaries, but inside your Zope site, as you pass data between Python scripts and page templates. Marius Gedminas -- http://pov.lt/ -- Zope 3/BlueBream consulting and development
Marius Gedminas wrote:
Has anyone seen this or anything similar before?
Yes. Basically, every Zope site I've upgraded to 2.12 has suffered from this. Annoyingly this only happens with MSIE, so developers/admins tend not to notice.
...well, and Safari, but no-one uses that, right? ;-)
Clarifying the "only with MSIE" comment: this happens if and only if the HTTP request doesn't explicitly include 'UTF-8' in Accept-Charset. IIRC.
OK, but I'll ask this again; what has the present of an Accept-Charset header got to do with whether or not rending a page template to a unicode string succeeds? Surely, up to that point, it really shouldn't matter what the browser wants or doesn't want, we're just getting a unicode string ready from whatever data we have around, right?
Any solutions spring to mind?
Magically sprinkle .encode() / .decode() in TAL expressions/Python scripts, where appropriate.
I want to get to the bottom of why the Accept-Charset header is getting involved at this point. Anyone able to point me at the code that does this? cheers, Chris -- Simplistix - Content Management, Batch Processing & Python Consulting - http://www.simplistix.co.uk
Hi Chris, 2010/9/1 Chris Withers <chris@simplistix.co.uk>:
Marius Gedminas wrote:
Has anyone seen this or anything similar before?
Yes. Basically, every Zope site I've upgraded to 2.12 has suffered from this. Annoyingly this only happens with MSIE, so developers/admins tend not to notice.
...well, and Safari, but no-one uses that, right? ;-)
I think of this problem like a topic same as following URLs. * https://mail.zope.org/pipermail/zope-dev/2009-August/037565.html * https://bugs.launchpad.net/zope2/+bug/160968 * https://bugs.launchpad.net/zope2/+bug/143873 * http://dev.plone.org/plone/ticket/8185 And why would this problem appear again? Will this be regression? -- Takayuki SHIMIZUKAWA
participants (7)
-
Charlie Clark -
Chris Withers -
Marco Bizzarri -
Marius Gedminas -
Robert Casties -
Takayuki Shimizukawa -
Vladislav Vorobiev