Took a big step back and wrote a simple method to test adding some ldap attributes via xml-rpc. It's seems that there is some issue in our 'real' methods that is causing the strings to be treated as ascii, but in the test method, everything is working well. But now it becomes even stranger, if zope is run with zopectl fg, even the original methods work. utf-8 strings remain encoded properly, but with zopectl start or runzope or runzope &, the method fails...same old error...I'll spare the whole traceback but the final error is UnicodeEncodeError: 'ascii' codec can't encode character u'\xf3' in position 4:
ordinal not in range(128)
Doyon, Jean-Francois wrote:
Looks like a bug in the LDAPUserFolder implementation to me ... Seems to explicitely expect a regular string, and not a unicode one.
What you could do is look at encoding the strings you get from XML-RPC with something like u''.encode('utf-8') (which would return a plain string).
Or go digg in LDAPUserFolder and figure out how it's messing with the encoding of your strings. LDAP v3 requires UTF-8 encoded strings as part of the spec.
J.F.
-----Original Message----- From: zope-bounces@zope.org [mailto:zope-bounces@zope.org] On Behalf Of Kevin Sent: September 12, 2007 12:44 To: zope@zope.org Subject: Re: [Zope] utf-8 over xmlrpc with zope2.9
Doyon, Jean-Francois wrote:
To be more explicit.
The problem is that the strings received by XML-RPC where the encoding is utf-8 end up internally as unicode u'' strings.
Other strings you're working with internally might not be however.
When you're trying to render a ZPT, if you have a mix of u'' and '' strings, you might end up with such UnicodeDecodeErrors ...
we're checking this now, but what about print statements that are used for debugging other strings that are formed for return values? I would think that if the error were related to ZPT we would see the traceback lead to publisher.
here's the traceback: Traceback (most recent call last): File "/var/lib/zope001/Products/web_site/webXMLRPC.py", line 124, in reg isterUser
status=reg.register(email,activate,returnto,firstname,lastname,middlename) File "/var/lib/zope001/Products/web_site/RegistrationService.py", line 84, in register
ldap.setAttributes(email,{'givenName':[firstname],'sn':[lastname],'middleNam e':[middlename]}) File "/var/lib/zope001/Products/web_site/LDAPServices.py", line 58, in set Attributes result= self.userManage.manage_editUser(user_dn, REQUEST, attributes) File "/var/lib/zope001/Products/LDAPUserFolder/LDAPUserFolder.py", line 1601, in manage_editUser File "/var/lib/zope001/Products/LDAPUserFolder/LDAPDelegate.py", line 500, in modify values = map(to_utf8, values) File "/var/lib/zope001/Products/LDAPUserFolder/utils.py", line 101, in to_utf8 return encodeUTF8(decodeLocal(s)[0])[0] UnicodeEncodeError: 'ascii' codec can't encode character u'\xf3' in position 4: ordinal not in range(128)
In my case, I make sure I use u'' strnigs everywhere, and set character sets to utf-8 everywhere, and so on.
Basically if needed you can make your whole system utf-8, avoiding the risks of those errors cropping up.
J.F.
-----Original Message----- From: zope-bounces+jdoyon=nrcan.gc.ca@zope.org [mailto:zope-bounces+jdoyon=nrcan.gc.ca@zope.org] On Behalf Of Doyon, Jean-Francois Sent: September 11, 2007 14:01 To: Kevin; zope@zope.org Subject: RE: [Zope] utf-8 over xmlrpc with zope2.9
Should work, I use XML-RPC pretty heavily with utf-8 without much problems.
This error might be occurring because you're mixing unicode and non-unicode strings on the same page.
Send the full traceback and we'll know more ...
J.F.
-----Original Message----- From: zope-bounces@zope.org [mailto:zope-bounces@zope.org] On Behalf Of Kevin Sent: September 11, 2007 13:56 To: zope@zope.org Subject: [Zope] utf-8 over xmlrpc with zope2.9
I am trying to send unicode encoded utf-8 or latin1 strings over xmlrpc to a method in zope. The method in zope tries to decode the string and it seems detects ascii. I then recieve an error that "UnicodeDecodeError: 'ascii' codec can't decode byte 0xf3 in position 4: ordinal not in range(128)"
I can re-create this error in a python shell by doing something like:
s = 'helló' s2 = unicode(s,'ascii') so after looking at code, I assume this is what is happening instead of >>> s2 = unicode(s,'iso-8859-1')
If I submit to this method from an html form, I have no error and the data is stored perfectly.
I have tried setting the encoding of xmlrpclib.Server to utf-8 and latin1(iso-8859-1) and making a custom transport for xmlrpclib where I add charset='utf-8' or charset='iso-8859-1'.
Is is not possible to send unicode characters > 128 to zope?
The method is LDAPUserFolder.manage_edit_user()
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev ) _______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://mail.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope-dev )