Problem with non-latin character sets in Page Templates
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hello, Due to ICANN's decision to allow for non-Latin characters in domain names, I thought I'd give our Zope installation a test to see how it would handle it. We've had problems with people copy and pasting from MS Word, and its use of strange characters. I'm sure at some point the faculty here at Purdue will have a need to link to web sites using non-Latin characters. This is my example HTML: <html> <head> <title>URL Test</title> </head> <body> <a href="http://موقع.وزارة-الأتصالات.مصر">http://موقع.وزارة-الأتصالات.مصر</a>. </body> </html> Not only is it a different character set, but it is a right to left character set. This code works fine while editing a Page Template. However, when viewing it, all of those characters in the href are converted to question marks. If the same HTML is pasted into a DTML Method or a File object, Zope will convert the characters to ASCII characters. When viewed it will be viewed correctly. Any reason that Page Templates may fail to render this properly? - -Chris - -- - -------------------------------------------------------------------- Christopher N. Deckard | Lead Web Systems Developer cnd@ecn.purdue.edu | Engineering Computer Network http://eng.purdue.edu/ECN/ | Purdue University - ---- zlib.decompress('x\234K\316Kq((-J)M\325KM)\005\000)"\005w') --- -----BEGIN PGP SIGNATURE----- Version: GnuPG/MacGPG2 v2.0.12 (Darwin) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAkvxoQcACgkQ+zGkEDwn8UZcGACgufURwanURBIBcZUSLYYRkEvV vKwAoNnGMlmOhE/GciZuTO3ldptuBr4Z =mypE -----END PGP SIGNATURE-----
On 05/17/2010 10:03 PM, Christopher N. Deckard wrote:
Hello, Due to ICANN's decision to allow for non-Latin characters in domain names, I thought I'd give our Zope installation a test to see how it would handle it. We've had problems with people copy and pasting from MS Word, and its use of strange characters. I'm sure at some point the faculty here at Purdue will have a need to link to web sites using non-Latin characters.
This is my example HTML:
<html> <head> <title>URL Test</title> </head> <body> <a href="http://EHB9.H2'1)-'D#*5'D'*.E51">http://EHB9.H2'1)-'D#*5'D'*.E51</a>. </body> </html>
Not only is it a different character set, but it is a right to left character set. This code works fine while editing a Page Template. However, when viewing it, all of those characters in the href are converted to question marks. If the same HTML is pasted into a DTML Method or a File object, Zope will convert the characters to ASCII characters. When viewed it will be viewed correctly.
Any reason that Page Templates may fail to render this properly?
It is a bug of page template. I patched them zope.pagetemplate-3.5.0-py2.6.egg/zope/pagetemplate/pagetemplate.py line 116 return output.getvalue() to 116 # --- unicode/utf-8 hotfix --- for idx in range(0, len(output.buflist)): try: output.buflist[idx] = unicode(output.buflist[idx], 'utf-8') except (UnicodeDecodeError): output.buflist[idx] = output.buflist[idx].decode('utf-8', 'replace') except: pass return output.getvalue() # --- unicode/utf-8 hotfix --- _______________________________________________ Zope maillist - Zope@zope.org https://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - https://mail.zope.org/mailman/listinfo/zope-announce https://mail.zope.org/mailman/listinfo/zope-dev )
participants (2)
-
Christopher N. Deckard -
Vladislav Vorobiev