[Zope3-Users] KeyError: Surrogate

[David Pratt]

Hi Bernd. Personally, I think zope and twisted is a good combination. 
Are you still running twisted in your installation or are you using 
zserver for your instances? Had you posted on twisted list to see if 
this triggered any discussion?

I am wondering what it means by surrogate? As in codec surrogate? If 
this were the case, I wonder why it would not be a decoding error you 
would receive instead. One kind of surrogate issue is when you have an 
incomplete byte sequence for a character in a codec. I found this 
explanation of a problem that appeared in an earlier version of python...

<snip>
Explanation of lone surrogates:

In Unicode, a surrogate pair is when you create the representation
of a character by using two values. So, for instance, UTF-32 can cover
the entire Unicode space (since Unicode is 20 bits, although MvL says
it is really more like 21 bits), but UTF-16 can't.  To solve the issue
for an encoding that cannot cover all possible characters in a single
value a character can be represented as a pair of UTF-16 values.  The
high surrogate cover the high 10 bits while the low surrogate cover
the lower 10 bits.  High and low surrogates can never be the same
since they are defined by a range of possible values and those ranges
do not overlap.  So with the proper high and low surrogate paired
together you can make any possible Unicode character.

     The problem in Python 2.2.1 is that when there is only a lone
surrogate (instead of there being a pair of surrogates), the encoder
for UTF-8 messes up and leaves off a UTF-8 value.  The following line
is an example:

	>>> u'\ud800'.encode('utf-8')
	'\xa0\x80'  #In Python 2.2.1
	'\xed\xa0\x80'  #In Python 2.3a0
	
Notice how in Python 2.3a0 the extra value is inserted so as to
make the representation a complete Unicode character instead of only
encoding the half of the surrogate pair that the encode was given.
</snip>

Don't know if this is meaninful or not, perhaps this issue has to do 
with a surrogate of another type in some form of object creation in Zope 
itself.

Regards,
David


Bernd Dorn wrote:
> 
> On 26.04.2006, at 16:52, David Pratt wrote:
> 
>> Hi Bernd. This is a bit unsettling. I realize as a practical matter, 
>> folks want their sites to run so the recommendation for zserver. It 
>> would be good to know what it is causing this problem with twisted. 
>> Anyone know how widespread this is. Anyone else with this experience.
> 
> Hi David
> 
> Sorry, but I don't know where to place this bug. I thought i was the 
> only one who got it. I asked stephan on the sprint about the cause, he 
> didn't know either, so i thought it was a missconfiguration of my 
> deployment first. I can't reproduce it now.
> 
> I think it has something to do with files which are not that small e.g. 
> 1MB and multiple requests, but I am not sure.
> 
> The twisted as default is really annoying, because the option for 
> zserver implies that twisted is more stable.