[Zope] utf-8 over xmlrpc with zope2.9

Wed Sep 12 13:12:27 EDT 2007

Looks like a bug in the LDAPUserFolder implementation to me ... Seems to explicitely expect a regular string, and not a unicode one.

What you could do is look at encoding the strings you get from XML-RPC with something like u''.encode('utf-8') (which would return a plain string).

Or go digg in LDAPUserFolder and figure out how it's messing with the encoding of your strings.  LDAP v3 requires UTF-8 encoded strings as part of the spec.

J.F. 

-----Original Message-----
From: zope-bounces at zope.org [mailto:zope-bounces at zope.org] On Behalf Of Kevin
Sent: September 12, 2007 12:44
To: zope at zope.org
Subject: Re: [Zope] utf-8 over xmlrpc with zope2.9

Doyon, Jean-Francois wrote:
> To be more explicit.
>
> The problem is that the strings received by XML-RPC where the encoding is utf-8 end up internally as unicode u'' strings.
>
> Other strings you're working with internally might not be however.
>
> When you're trying to render a ZPT, if you have a mix of u'' and '' strings, you might end up with such UnicodeDecodeErrors ...
>
>   
we're checking this now, but what about print statements that are used for debugging other strings that are formed for return values?
I would think that if the error were related to ZPT we would see the traceback lead to publisher.

here's the traceback:
Traceback (most recent call last):
  File "/var/lib/zope001/Products/web_site/webXMLRPC.py", line 124, in reg isterUser

status=reg.register(email,activate,returnto,firstname,lastname,middlename)
  File "/var/lib/zope001/Products/web_site/RegistrationService.py", line 84, in register

ldap.setAttributes(email,{'givenName':[firstname],'sn':[lastname],'middleNam
e':[middlename]})
  File "/var/lib/zope001/Products/web_site/LDAPServices.py", line 58, in set Attributes
    result= self.userManage.manage_editUser(user_dn, REQUEST, attributes)
  File "/var/lib/zope001/Products/LDAPUserFolder/LDAPUserFolder.py",
line 1601,
in manage_editUser
  File "/var/lib/zope001/Products/LDAPUserFolder/LDAPDelegate.py", line 500, in modify
    values = map(to_utf8, values)
  File "/var/lib/zope001/Products/LDAPUserFolder/utils.py", line 101, in
to_utf8
    return encodeUTF8(decodeLocal(s)[0])[0]
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf3' in position 4:
ordinal not in range(128)
> In my case, I make sure I use u'' strnigs everywhere, and set character sets to utf-8 everywhere, and so on.
>
>   

> Basically if needed you can make your whole system utf-8, avoiding the risks of those errors cropping up.
>
> J.F. 
>
> -----Original Message-----
> From: zope-bounces+jdoyon=nrcan.gc.ca at zope.org 
> [mailto:zope-bounces+jdoyon=nrcan.gc.ca at zope.org] On Behalf Of Doyon, 
> Jean-Francois
> Sent: September 11, 2007 14:01
> To: Kevin; zope at zope.org
> Subject: RE: [Zope] utf-8 over xmlrpc with zope2.9
>
> Should work, I use XML-RPC pretty heavily with utf-8 without much problems.
>
> This error might be occurring because you're mixing unicode and non-unicode strings on the same page.
>
> Send the full traceback and we'll know more ...
>
> J.F. 
>
> -----Original Message-----
> From: zope-bounces at zope.org [mailto:zope-bounces at zope.org] On Behalf 
> Of Kevin
> Sent: September 11, 2007 13:56
> To: zope at zope.org
> Subject: [Zope] utf-8 over xmlrpc with zope2.9
>
> I am trying to send unicode encoded utf-8 or latin1 strings over 
> xmlrpc to a method in zope.  The method in zope tries to decode the 
> string and it seems detects ascii.  I then recieve an error that
> "UnicodeDecodeError: 'ascii' codec can't decode byte 0xf3 in position 4: 
> ordinal not in range(128)"
>
> I can re-create this error in a python shell by doing something like:
>  >>> s = 'helló'
>  >>> s2 = unicode(s,'ascii')
> so after looking at code, I assume this is what is happening instead 
> of  >>> s2 = unicode(s,'iso-8859-1')
>
> If  I  submit  to this method from  an html form, I have no error and the data is stored perfectly.
>
> I have tried setting the encoding of xmlrpclib.Server to utf-8 and
> latin1(iso-8859-1) and making a custom transport for xmlrpclib where I add charset='utf-8' or charset='iso-8859-1'.
>
> Is is not possible to send unicode characters > 128 to zope?
>
>
> The method is LDAPUserFolder.manage_edit_user()
>
>
>
>
>
>
> _______________________________________________
> Zope maillist  -  Zope at zope.org
> http://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope-dev ) 
> _______________________________________________
> Zope maillist  -  Zope at zope.org
> http://mail.zope.org/mailman/listinfo/zope
> **   No cross posts or HTML encoding!  **
> (Related lists -
>  http://mail.zope.org/mailman/listinfo/zope-announce
>  http://mail.zope.org/mailman/listinfo/zope-dev )
>   

_______________________________________________
Zope maillist  -  Zope at zope.org
http://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists -
 http://mail.zope.org/mailman/listinfo/zope-announce
 http://mail.zope.org/mailman/listinfo/zope-dev )