Hi Martin! Martin Aspeli wrote:

...

- After traversal and before calling the object ZPublisher.Publish.publish should figure out if the object expects zope.publisher behavior. Either we use a new interface for this or we use zope.publisher.interfaces.browser.IBrowserPage: AFAICS in Zope 2 land only zope.formlib and z3c.form based views implement IBrowserPage.

Isn't this in zope.browserpage now?

No.

...

- plone.z3cform uses a modified version of processInputs and doesn't use setPageEncoding. Can anybody explain why? I guess that are no z3c.form specific reasons. Maybe the changes can be merged back to Zope?

processInputs() in Five was very buggy. I thought I'd merged those changes into Zope 2?

Well. You were the last person who touched both. But the changes are quit different: http://svn.zope.org/Zope/trunk/src/Products/Five/browser/decode.py?rev=11528... http://svn.zope.org/plone.z3cform/trunk/plone/z3cform/z2.py?rev=109071&view=... Is there still anything in plone.z3cform that should be merged into Zope 2?

I don't know what setPageEncoding() does, though.

It sets a response Content-Type header with the first charset processInputs tries for decoding. Cheers, Yuppie

Laurence Rowe wrote:

On 2 March 2011 10:00, yuppie<y.2011@wcm-solutions.de> wrote:

...

Martin Aspeli wrote:

...

I don't know what setPageEncoding() does, though.

It sets a response Content-Type header with the first charset processInputs tries for decoding.

Is the charset of the request necessarily the right choice for the response? In Plone we always serve UTF-8 encoded.

getPreferredCharsets()[0] always returns 'utf-8' **if** UTF-8 is accepted. If 'utf-8' is not in getPreferredCharsets(), it is not very likely that the browser speaks UTF-8 and processInputs will not even try to decode with UTF-8. In that case it might be better to respond with an accepted encoding. Cheers, Yuppie

Hi Charlie! Charlie Clark wrote:

Am 02.03.2011, 12:29 Uhr, schrieb yuppie<y.2011@wcm-solutions.de>:

...

getPreferredCharsets()[0] always returns 'utf-8' **if** UTF-8 is accepted.

...

If 'utf-8' is not in getPreferredCharsets(), it is not very likely that the browser speaks UTF-8 and processInputs will not even try to decode with UTF-8. In that case it might be better to respond with an accepted encoding.

I think you are drawing the wrong conclusion

I did talk about getPreferredCharsets(), not about the Accept-Charset header.

some browsers (Internet Explorer and Safari spring to mind but this will also be the behaviour in Opera from 11.10) simply don't have an accept-charset header and the W3C says this means you can throw anything at them, in which case UTF-8 is a good choice.

You implemented this in getPreferredCharsets(), so that method says UTF-8 is accepted if no Accept-Charset header is set. And Five always used this line which has the same effect:: charsets = envadapter.getPreferredCharsets() or ['utf-8'] Cheers, Yuppie

Hi! Laurence Rowe wrote:

On 2 March 2011 11:29, yuppie<y.2011@wcm-solutions.de> wrote:

...

Laurence Rowe wrote:

...

On 2 March 2011 10:00, yuppie<y.2011@wcm-solutions.de> wrote:

...

Martin Aspeli wrote:

...

I don't know what setPageEncoding() does, though.

It sets a response Content-Type header with the first charset processInputs tries for decoding.

Is the charset of the request necessarily the right choice for the response? In Plone we always serve UTF-8 encoded.

getPreferredCharsets()[0] always returns 'utf-8' **if** UTF-8 is accepted.

If 'utf-8' is not in getPreferredCharsets(), it is not very likely that the browser speaks UTF-8 and processInputs will not even try to decode with UTF-8. In that case it might be better to respond with an accepted encoding.

If you serve differently encoded pages then you should set Vary: Accept-Charset.

That seems to be correct. So you found a bug in zope.publisher and five.formlib. If they do charset negotiation, they have to set Vary.

But then without normalization you'd get an explosion of different page variations.

AFAICS that normalization can't be done by the server and we can't prevent ineffective caching.

Without the Vary, it means that a visitor can poison the cache by supplying (only) a weird charset in Accept-Encoding. The page would then be served in this encoding, cached downstream, and if other visitor's browsers don't support that charset then they have a problem.

That sounds like charset negotiation isn't a good idea and neither zope.publisher nor five.formlib should do it. If we don't negotiate the charset, we should still have a setPageEncoding method that overrides the ZPublisher default_encoding with UTF-8. But what does all that mean for the processInputs methods in Five (used by five.formlib) and in plone.z3cform? If we always send UTF-8, their current implementation doesn't make much sense to me. Don't know if we really should try to fall back to all the charsets mentioned in Accept-Charset. But at least we should *always* try UTF-8 decoding first. Cheers, Yuppie

Charlie Clark wrote:

Am 04.03.2011, 08:58 Uhr, schrieb yuppie<y.2011@wcm-solutions.de>:

...

If we always send UTF-8, their current implementation doesn't make much sense to me. Don't know if we really should try to fall back to all the charsets mentioned in Accept-Charset. But at least we should *always* try UTF-8 decoding first.

I'm not sure if this is directly related but I remember Withers having a discussion (alright, shouting match) with Andreas about cycling through all kinds of encoding possibilities on the resolver. I can't find the thread at the moment but I think it related to the way templates could be edited TTW or how to handle situations of mixed encoding.

I considered to propose that we don't use the IUserPreferredCharsets adapter at all in Zope 2 and remove its registration in ZCML. But then I noticed the code Andreas wrote in Products.PageTemplates.unicodeconflictresolver.PreferredCharsetResolver. I'm not going to start that discussion again. Cheers, Yuppie