Making Zope accept international characters in Id's
Hiya everyone! I didn't get any response on my last post, so I'm charging ahead full speed. The fact that litmus complains about Zope's inability to accept UTF-8 filenames is something I view as a bug. It can be fixed simply by changing the default _checkid() implementation to allow extended characters. However, I don't want to break anything, so I think it should be optional, controlled by a setting. Either the already existing, but seemingly undocumented setting management_page_charset could be used to check that the given Id is acceptable in that charset. OR, a separate setting could be used, maybe allowed_id_charset? I'm gonna go with that last one if nobody complains. Suggested code (only the important parts): # Any other characters that should NEVER be allowed?bad_id=re.compile(r'[:;/\\]').search def checkValidId(self, id, allow_dup=0): # So should be allow unicode, then? if not id or not isinstance(id, StringType): if isinstance(id, UnicodeType): id = escape(id) raise BadRequestException, ('Empty or invalid id specified', id) # Here is the characterset checking. This works fine, I've tried it. try: charset = getattr(self, 'allowed_id_charset', 'ascii') id.decode(charset) except: raise BadRequestException, ( 'The id "%s" is not of the allowed character set (%s).' % \ (escape(id), self.allowed_id_charset)) # And here we check for unallowed special characters. if bad_id(id) is not None: raise BadRequestException, ( 'The id "%s" contains characters illegal in URLs.' % \ escape(id)) # And then comes the rest of the tests, as usual, for aq_ and such. # I won't copy them in in this mail. Opinions?
This issue has been discussed several times in the past. There was a consensus not allow extended characters because of compatibility issues and conformance of the HTTP specs (which don't allow extended chars in URLs AFAIk in unencoded form). So it is your task to proof that your implementation works fine, does not break compatibility and is compliant with the HTTP specs. I am not against using extended chars (which would be fine) but there have not been any reasonable approaches so far AFAIK. -aj --On Montag, 20. Oktober 2003 16:01 Uhr +0200 Lennart Regebro <lennart@regebro.nu> wrote:
Hiya everyone!
I didn't get any response on my last post, so I'm charging ahead full speed.
The fact that litmus complains about Zope's inability to accept UTF-8 filenames is something I view as a bug. It can be fixed simply by changing the default _checkid() implementation to allow extended characters. However, I don't want to break anything, so I think it should be optional, controlled by a setting.
Either the already existing, but seemingly undocumented setting management_page_charset could be used to check that the given Id is acceptable in that charset.
OR, a separate setting could be used, maybe allowed_id_charset?
I'm gonna go with that last one if nobody complains. Suggested code (only the important parts):
# Any other characters that should NEVER be # allowed?bad_id=re.compile(r'[:;/\\]').search
def checkValidId(self, id, allow_dup=0): # So should be allow unicode, then? if not id or not isinstance(id, StringType): if isinstance(id, UnicodeType): id = escape(id) raise BadRequestException, ('Empty or invalid id specified', id)
# Here is the characterset checking. This works fine, I've tried it. try: charset = getattr(self, 'allowed_id_charset', 'ascii') id.decode(charset) except: raise BadRequestException, ( 'The id "%s" is not of the allowed character set (%s).' % \ (escape(id), self.allowed_id_charset)) # And here we check for unallowed special characters. if bad_id(id) is not None: raise BadRequestException, ( 'The id "%s" contains characters illegal in URLs.' % \ escape(id)) # And then comes the rest of the tests, as usual, for aq_ and such. # I won't copy them in in this mail.
Opinions?
_______________________________________________ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Andreas Jung wrote:
This issue has been discussed several times in the past.
Probably, but infortunately I can't find anything about it. Toby Dickenson wrote:
On Monday 20 October 2003 15:01, Lennart Regebro wrote:
The fact that litmus complains about Zope's inability to accept UTF-8 filenames is something I view as a bug.
does webdav specify how utf-8-encoded dav filenames correspond to file > names used in URLs?
Good question.
If we need to support unicode ids, then the right approach is to allow > them to be of unicode type.
Ah, yes, of course, you are right. And that requires bigger changes than the suggested, of course. OK, back to the drawing board.
On Monday 20 October 2003 15:01, Lennart Regebro wrote:
The fact that litmus complains about Zope's inability to accept UTF-8 filenames is something I view as a bug.
does webdav specify how utf-8-encoded dav filenames correspond to file names used in URLs?
Either the already existing, but seemingly undocumented setting management_page_charset could be used to check that the given Id is acceptable in that charset.
That feature is intended only as an aid for the transition to unicode for users who have content that is stored in an encoded form.
OR, a separate setting could be used, maybe allowed_id_charset?
If we need to support unicode ids, then the right approach is to allow them to be of unicode type. It looks like OFS has the beginnings of supporting this (as your proposed code fragment pointed out), but Im not aware of it being used.
I'm gonna go with that last one if nobody complains.
consider this a complaint. character encodings should be applied at zopes boundary, not in the middle of its processing. -- Toby Dickenson
participants (3)
-
Andreas Jung -
Lennart Regebro -
Toby Dickenson