[Zope3-dev] test errors due to ascii defaultencoding assumption
Dieter Maurer
dieter at handshake.de
Sat Oct 22 13:59:21 EDT 2005
Tim Peters wrote at 2005-10-21 15:27 -0400:
> ...
>[Dieter Maurer]
>> Nowadays, life is hell without a sensible "setdefaultencoding":
>>
>> Many isolated modules intersperse unicode in an otherwise
>> "string" dominated world causing wide spread
>> UnicodeDecodingErrors.
[Tim]
>And setdefaultencoding helps that? What, you set it to latin1 just to
>squash the exceptions?
[Dieter]
No, because I know that I live in a "latin-1" subworld and that
almost all "str"s containing text (rather than binary data)
*ARE* indeed "latin-1" encoded.
[Tim]
> "Brrrrr", if so. I suppose that's got a
>decent shot at working by accident, anyway.
[Dieter]
Up to now, I met a single "non accident" (you would add "<wink>"),
i.e. a single case where the "defaultencoding = 'latin-1'"
proved counter productive:
Python's "xmlrpclib" tried to check whether it should
convert the unicode data to an "str" with code like this:
try:
# try to return a simple 7bit string
return str(data)
except UnicodeError: return data
Of course, this code only works as expected
when "defaultencoding == 'ascii'".
The XML-RPC called function would have correctly worked with Unicode
and with an UTF-8 encoded "str" -- but it failed for a
"latin-1" encoded "str".
This problem was easily fixed by fixing the "xmlrpclib" code
("return data.encode('ascii')" rather than "return str(data)").
On the other hand, my "defaultencoding = 'latin-1'" saved me
dozens (if not hundreds) of "UnicodeDecodingError"s).
Thus, your "accident"s are appreciated incidents :-)
--
Dieter
More information about the Zope3-dev
mailing list