[Zope-CMF] Weird UnicodeDecodeError with zope.formlib

[Patrick Gerken]

Did you try to put a pdb in prcessInputs of ZPublisher/HTTPRequest, around
line 642 where my code shows something like this:
 640                     if flags & CONVERTED:
 641                         try:
 642                             if character_encoding:
 643                                 # We have a string with a specified
character
 644                                 # encoding.  This gets passed to the
converter
 645                                 # either as unicode, if it can handle
it, or
 646                                 # crunched back down to latin-1 if it
can not.
 647                                 item = unicode(item,character_encoding)
 648                                 if
hasattr(converter,'convert_unicode'):
 649                                     item =
converter.convert_unicode(item)
 650                                 else:
 651                                     item = converter(
 652                                         item.encode(default_encoding))
 653                             else:
 654                                 item = converter(item)
 655
 656                             # Flag potentially unsafe values
 657                             if converter_type in ('string',
'required', 'text',
 658                                                   'ustring', 'utext'):



...

The only place I can see where a default encoding gets changed is by the
default-zpublisher-encoding from zope.conf


On Fri, Nov 30, 2012 at 6:14 PM, Patrick Gerken <patrick.gerken at computer.org
> wrote:

>
>
>
> On Fri, Nov 30, 2012 at 11:19 AM, Charlie Clark <
> charlie.clark at clark-consulting.eu> wrote:
>
>> Hi Patrick,
>>
>> Am 30.11.2012, 09:50 Uhr, schrieb Patrick Gerken <do3ccqrv at googlemail.com
>> >:
>>
>>
>>  Add sentry logging with raven to the sites. Trigger an exception in both
>>> sites. With sentry you can not only see the traceback, but check the
>>> local
>>> variable of each frame. You can do the same with pdb of course but not so
>>> easily side by side to see where the local vars start to differ.
>>> I can give you access to my sentry server to send the logs to.
>>>
>>
>> thanks for the tip. I've got Sentry and Raven running and reporting but
>> I'm afraid I still can't see the difference. The posted form looks
>> indentical in both cases. I can only assume that, as you first suggested,
>> there is a difference lower down the stack which is causing one instance to
>> decode the URL-encoded form to unicode and the other to encode it as UTF-8.
>> How can I check this? locale.getdefaultlocale() reports ('de_DE', 'UTF8')
>> for both.
>
>
> I don't understand why you see no difference in the stacktrace, but a
> difference with pdb in the end. Doesn't one instance show that the input is
> a string and the other that its unicode?
> Do you see this until you extract it first from the request object?
>
> You are not having one form saying fieldname:string and the other just
> fieldname?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.zope.org/pipermail/zope-cmf/attachments/20121130/32fc584c/attachment.html>