Hi, because nobody in der Localizer-Mailings list react and I thinks it could be a common problem with utf-8 in zope, I ask my question here also. I use Localizer 1.2.1 and I try out some things and all looks great. After testing Localizer my next plan was to switch from encoding ISO-8859-15 to utf-8 in Zope 2.10.3 but run into Problems when inserting text through a MessageCatalog (object of Localizer). Here is what I do in Zope: 1. I add "default-zpublisher-encoding utf-8" to my zope.conf and restart zope. 2. In ZMI I add the property "management_page_charset" and set it to "utf-8" 3. I add a new "DTML Document" and start editing. I check the encoding in my Browser and it switched to "unicode (utf-8)". Looks good. 4. I insert html-code and some russian chars for testing <html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body> ...russian chars... </body> </html> After Saving all looks great in ZMI and also in my webbrowser. 5. Now I add one line to use a MessageCatalog: <dtml-var "MessageCatalog('Hallo Welt!')"> After that, all of my utf-8-chars are broken in my webbrowser. I check browser-encoding and it is still utf-8. In ZMI my "DTML Document" looks good. This line must switch something in the output-encoding in the internal page-render (I don't know, how to better explain it). Is this an error in Localizer or in Zope or have I overlook a fault? Thanks Patrick
--On 24. August 2007 08:32:17 +0200 Patrick Ulmer <ulmer@truckport.de> wrote:
Hi,
because nobody in der Localizer-Mailings list react and I thinks it could be a common problem with utf-8 in zope, I ask my question here also. I use Localizer 1.2.1 and I try out some things and all looks great. After testing Localizer my next plan was to switch from encoding ISO-8859-15 to utf-8 in Zope 2.10.3 but run into Problems when inserting text through a MessageCatalog (object of Localizer).
Here is what I do in Zope:
1. I add "default-zpublisher-encoding utf-8" to my zope.conf and restart zope. 2. In ZMI I add the property "management_page_charset" and set it to "utf-8" 3. I add a new "DTML Document" and start editing. I check the encoding in my Browser and it switched to "unicode (utf-8)". Looks good. 4. I insert html-code and some russian chars for testing
<html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body> ...russian chars... </body> </html>
After Saving all looks great in ZMI and also in my webbrowser.
5. Now I add one line to use a MessageCatalog:
<dtml-var "MessageCatalog('Hallo Welt!')">
After that, all of my utf-8-chars are broken in my webbrowser. I check browser-encoding and it is still utf-8. In ZMI my "DTML Document" looks good. This line must switch something in the output-encoding in the internal page-render (I don't know, how to better explain it). Is this an error in Localizer or in Zope or have I overlook a fault?
The http-equiv tag means not much to browser. Try setting the content-type including the charset within the HTTP response (REQUEST.RESPONSE.setHeader(...)). -aj
Hi,
4. I insert html-code and some russian chars for testing
<html> <head> <meta http-equiv="content-type" content="text/html; charset=utf-8"> </head> <body> ...russian chars... </body> </html>
After Saving all looks great in ZMI and also in my webbrowser.
5. Now I add one line to use a MessageCatalog:
<dtml-var "MessageCatalog('Hallo Welt!')">
After that, all of my utf-8-chars are broken in my webbrowser. I check browser-encoding and it is still utf-8. In ZMI my "DTML Document" looks good. This line must switch something in the output-encoding in the internal page-render (I don't know, how to better explain it). Is this an error in Localizer or in Zope or have I overlook a fault?
The http-equiv tag means not much to browser. Try setting the content-type including the charset within the HTTP response (REQUEST.RESPONSE.setHeader(...)).
I have checked it but it is the same problem. Browser is encoding in utf-8 correct, but the result (html sourcecode from zope) is not utf-8 anymore after inserting <dtml-var "MessageCatalog('Hallo Welt!')"> and I don't know why it changed. How can a dtml-var-tag changed the encoding for the document? Thanks, Patrick
----- Original Message ----- From: "Patrick Ulmer" <ulmer@truckport.de>
The http-equiv tag means not much to browser. Try setting the content-type including the charset within the HTTP response (REQUEST.RESPONSE.setHeader(...)).
I have checked it but it is the same problem. Browser is encoding in utf-8 correct, but the result (html sourcecode from zope) is not utf-8 anymore after inserting <dtml-var "MessageCatalog('Hallo Welt!')"> and I don't know why it changed. How can a dtml-var-tag changed the encoding for the document?
What about <dtml-var "_.str(MessageCatalog('Hallo Welt!'))"> Use _.str() solves many potential problems Regards JL.
--On 24. August 2007 09:36:30 +0200 Jaroslav Lukesh <lukesh@seznam.cz> wrote:
----- Original Message ----- From: "Patrick Ulmer" <ulmer@truckport.de>
The http-equiv tag means not much to browser. Try setting the content-type including the charset within the HTTP response (REQUEST.RESPONSE.setHeader(...)).
I have checked it but it is the same problem. Browser is encoding in utf-8 correct, but the result (html sourcecode from zope) is not utf-8 anymore after inserting <dtml-var "MessageCatalog('Hallo Welt!')"> and I don't know why it changed. How can a dtml-var-tag changed the encoding for the document?
What about
<dtml-var "_.str(MessageCatalog('Hallo Welt!'))">
Use _.str() solves many potential problems
That's the mallet method. -aj
I have checked it but it is the same problem. Browser is encoding in utf-8 correct, but the result (html sourcecode from zope) is not utf-8 anymore after inserting <dtml-var "MessageCatalog('Hallo Welt!')"> and I don't know why it changed. How can a dtml-var-tag changed the encoding for the document?
What about
<dtml-var "_.str(MessageCatalog('Hallo Welt!'))">
Use _.str() solves many potential problems
I think it's the right way for a solution. Now my page is utf-8, but if the string returned from MessageCatalog have utf-8 chars and not online chars in ascii range I get this error-message: *Error Type: UnicodeEncodeError* *Error Value: 'ascii' codec can't encode character u'\xf4' in position 3: ordinal not in range(128)* Regards Patrick
Hi, I think I solved the problem. <dtml-var "MessageCatalog('Hallo Welt!').encode('utf-8')"> works. I read some further and found out that I should use something like import sys sys.setdefaultencoding('utf-8') I set it global in sitecustomize.py under /usr/lib/python2.4/site-packages and then <dtml-var "MessageCatalog('Hallo Welt!')"> works. Now I only must find the right place for setdefaultencoding() so it only affect my zope-instance. Regars Patrick Patrick Ulmer schrieb:
I think it's the right way for a solution. Now my page is utf-8, but if the string returned from MessageCatalog have utf-8 chars and not online chars in ascii range I get this error-message:
Error Type: UnicodeEncodeError Error Value: 'ascii' codec can't encode character u'\xf4' in position 3: ordinal not in range(128)
If you can, I would strongly recommend using the Zope3/Five way of doing translations and i18n in general! See here: http://worldcookery.com/files/fivei18n/ http://codespeak.net/z3/five/i18n.html I've made my site fully internationalized, localized and utf-8 using this method without resorting to any "trickery". setdefaultencoding() is apparently frowned upon ... Not sure why though. Good luck! J.F. -----Original Message----- From: zope-bounces@zope.org [mailto:zope-bounces@zope.org] On Behalf Of Patrick Ulmer Sent: August 24, 2007 04:27 To: zope@zope.org Subject: Re: [Zope] utf-8 problem in Zope when using Localizer Hi, I think I solved the problem. <dtml-var "MessageCatalog('Hallo Welt!').encode('utf-8')"> works. I read some further and found out that I should use something like import sys sys.setdefaultencoding('utf-8') I set it global in sitecustomize.py under /usr/lib/python2.4/site-packages and then <dtml-var "MessageCatalog('Hallo Welt!')"> works. Now I only must find the right place for setdefaultencoding() so it only affect my zope-instance. Regars Patrick Patrick Ulmer schrieb:
I think it's the right way for a solution. Now my page is utf-8, but if the string returned from MessageCatalog have utf-8 chars and not online chars in ascii range I get this error-message:
Error Type: UnicodeEncodeError Error Value: 'ascii' codec can't encode character u'\xf4' in position 3: ordinal not in range(128)
Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
Doyon, Jean-Francois wrote at 2007-8-24 09:36 -0400:
... setdefaultencoding() is apparently frowned upon ... Not sure why though.
Occasionally, a package writer expects that "str" converts to ASCII, such that try: ascii = str(unicodeStr) except UnicodeError: # non ascii can be used to check for ASCIIness. Of course, this is true only when the defaultencoding is "ASCII". Fortunately, "unicode(unicodeStr, 'ascii')" can safely be used instead. The code above actually occured in Python's "xmlrpclib". When I reported the bug, people said we should eliminate "setdefaultencoding".... But, then, they nevertheless fixed the bug... -- Dieter
Patrick Ulmer wrote at 2007-8-24 10:26 +0200:
I think I solved the problem.
<dtml-var "MessageCatalog('Hallo Welt!').encode('utf-8')">
works. I read some further and found out that I should use something like
import sys sys.setdefaultencoding('utf-8')
I set it global in sitecustomize.py under /usr/lib/python2.4/site-packages and then
<dtml-var "MessageCatalog('Hallo Welt!')">
works. Now I only must find the right place for setdefaultencoding() so it only affect my zope-instance.
"$INSTANCE_HOME/lib/python" would be a good place for the "site-customize.py". -- Dieter
(Fri, Aug 24, 2007 at 10:26:42AM +0200) Patrick Ulmer wrote/schrieb/egrapse:
Now I only must find the right place for setdefaultencoding() so it only affect my zope-instance.
Install a separate python that is used only by your zope instance. Use setdefaultencoding only there. Regards, Sascha
--On 25. August 2007 15:48:04 +0200 Sascha Welter <zopelist@betabug.ch> wrote:
(Fri, Aug 24, 2007 at 10:26:42AM +0200) Patrick Ulmer wrote/schrieb/egrapse:
Now I only must find the right place for setdefaultencoding() so it only affect my zope-instance.
Install a separate python that is used only by your zope instance. Use setdefaultencoding only there.
This is nonense. A custom sitecustomize.py script does the job without installing a complete new Python. -aj
Install a separate python that is used only by your zope instance. Use setdefaultencoding only there.
This is nonense. A custom sitecustomize.py script does the job without installing a complete new Python. You need to read a previous reply:
I set it global in sitecustomize.py under /usr/lib/python2.4/site-packages and then
<dtml-var "MessageCatalog('Hallo Welt!')">
works. Now I only must find the right place for setdefaultencoding() so it only affect my zope-instance. For the path used there "/usr/lib/python2.4/site-packages", it seems that the python used by zope is a binary that comes into a rpm package, so, this may break some things on the OS. So, I guess a new python for only zope makes sense here
Josef Meile wrote at 2007-9-4 09:31 +0200:
Install a separate python that is used only by your zope instance. Use setdefaultencoding only there.
This is nonense. A custom sitecustomize.py script does the job without installing a complete new Python. You need to read a previous reply:
I set it global in sitecustomize.py under /usr/lib/python2.4/site-packages and then
<dtml-var "MessageCatalog('Hallo Welt!')">
works. Now I only must find the right place for setdefaultencoding() so it only affect my zope-instance. For the path used there "/usr/lib/python2.4/site-packages", it seems that the python used by zope is a binary that comes into a rpm package, so, this may break some things on the OS. So, I guess a new python for only zope makes sense here
There is no need to put "sitecustomize.py" in the global Python distribution. Any place on the "PYTHONPATH" as effective when Python starts is sufficient. We ensure that "$INSTANCE_HOME/lib/python" is on this path and then can put "sitecustomize.py" there. -- Dieter
participants (7)
-
Andreas Jung -
Dieter Maurer -
Doyon, Jean-Francois -
Jaroslav Lukesh -
Josef Meile -
Patrick Ulmer -
Sascha Welter