[Zope] utf-8 over xmlrpc with zope2.9

[Kevin]

Doyon, Jean-Francois wrote:
> To be more explicit.
>
> The problem is that the strings received by XML-RPC where the encoding is utf-8 end up internally as unicode u'' strings.
>
> Other strings you're working with internally might not be however.
>
> When you're trying to render a ZPT, if you have a mix of u'' and '' strings, you might end up with such UnicodeDecodeErrors ...
>
>   
we're checking this now, but what about print statements that are used
for debugging other strings that are formed for return values?
I would think that if the error were related to ZPT we would see the
traceback lead to publisher.

here's the traceback:
Traceback (most recent call last):
  File "/var/lib/zope001/Products/web_site/webXMLRPC.py", line 124, in reg
isterUser

status=reg.register(email,activate,returnto,firstname,lastname,middlename)
  File "/var/lib/zope001/Products/web_site/RegistrationService.py", line 84,
in register

ldap.setAttributes(email,{'givenName':[firstname],'sn':[lastname],'middleNam
e':[middlename]})
  File "/var/lib/zope001/Products/web_site/LDAPServices.py", line 58, in set
Attributes
    result= self.userManage.manage_editUser(user_dn, REQUEST, attributes)
  File "/var/lib/zope001/Products/LDAPUserFolder/LDAPUserFolder.py",
line 1601,
in manage_editUser
  File "/var/lib/zope001/Products/LDAPUserFolder/LDAPDelegate.py", line
500, in
modify
    values = map(to_utf8, values)
  File "/var/lib/zope001/Products/LDAPUserFolder/utils.py", line 101, in
to_utf8
    return encodeUTF8(decodeLocal(s)[0])[0]
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf3' in
position 4:
ordinal not in range(128)
> In my case, I make sure I use u'' strnigs everywhere, and set character sets to utf-8 everywhere, and so on.
>
>   

> Basically if needed you can make your whole system utf-8, avoiding the risks of those errors cropping up.
>
> J.F. 
>
> -----Original Message-----
> From: zope-bounces+jdoyon=nrcan.gc.ca at zope.org [mailto:zope-bounces+jdoyon=nrcan.gc.ca at zope.org] On Behalf Of Doyon, Jean-Francois
> Sent: September 11, 2007 14:01
> To: Kevin; zope at zope.org
> Subject: RE: [Zope] utf-8 over xmlrpc with zope2.9
>
> Should work, I use XML-RPC pretty heavily with utf-8 without much problems.
>
> This error might be occurring because you're mixing unicode and non-unicode strings on the same page.
>
> Send the full traceback and we'll know more ...
>
> J.F. 
>
> -----Original Message-----
> From: zope-bounces at zope.org [mailto:zope-bounces at zope.org] On Behalf Of Kevin
> Sent: September 11, 2007 13:56
> To: zope at zope.org
> Subject: [Zope] utf-8 over xmlrpc with zope2.9
>
> I am trying to send unicode encoded utf-8 or latin1 strings over xmlrpc to a method in zope.  The method in zope tries to decode the string and it seems detects ascii.  I then recieve an error that
> "UnicodeDecodeError: 'ascii' codec can't decode byte 0xf3 in position 4: 
> ordinal not in range(128)"
>
> I can re-create this error in a python shell by doing something like:
>  >>> s = 'helló'
>  >>> s2 = unicode(s,'ascii')
> so after looking at code, I assume this is what is happening instead of  >>> s2 = unicode(s,'iso-8859-1')
>
> If  I  submit  to this method from  an html form, I have no error and the data is stored perfectly.
>
> I have tried setting the encoding of xmlrpclib.Server to utf-8 and
> latin1(iso-8859-1) and making a custom transport for xmlrpclib where I add charset='utf-8' or charset='iso-8859-1'.
>
> Is is not possible to send unicode characters > 128 to zope?
>
>
> The method is LDAPUserFolder.manage_edit_user()
>
>
>
>
>
>
> _______________________________________________
> Zope maillist  -  Zope at zope.org
> http://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope-dev ) _______________________________________________
> Zope maillist  -  Zope at zope.org
> http://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope-dev )
>