Hello list. I'm having an issue with ZTUtils.Zope.make_query when providing Unicode values. Specifically, We're building search results pagination using CMFPlone.PloneBatch Here's the scenario: The search form is provided a Unicode string. The query is properly written as "?search_text:utf8:ustring=value" The search will return many results. Our searchResults browser view will provide a PloneBatch object, making it very easy to display only, for example, 30 results, then provide page links at the bottom of the page for the next 30 results, or whatever. These links will contain the original query string, but when they are built using make_query(), the result is "?search_text=value" - What follows is Unicode decode errors when the 'next' link is followed - the search_text is marshaled as a 'str', not as a 'unicode' The bottom line is that make_query (actually, complex_marshal & simple_marshal) doesn't seem to apply the ':ustring' type prefix when creating query strings. Kind of important to us at Radio Free Asia, considering we host Cantonese and Korean languages, for example. This was fixed by using sys.setdefaultencoding('utf-8'), but that seems like quite a sledgehammer fix. ---------- I have a proposed fix, branched from the master branch on Github. Diff here: https://github.com/zopefoundation/Zope/pull/4/files Questions: 1. Is "Master" the correct place to fork from? 2. Is is safe - at least to start - to assume utf8 encoding when constructing query strings? 3. Dangerous idea: If simpleMarshal is passed a unicode object that can be represented as ASCII, should we "magically" turn it into a string? I don't like magic, but I also know it's common (and good) practice to use unicode in ASCII-based languages, making this exercise an overkill for those languages. Any comments or criticisms (not to hard please!) would be appreciated! -Flip -- Mike McFadden Radio Free Asia Technical Operations Division 2025 M Street NW Washington DC 20036 USA This e-mail message is intended only for the use of the addressee and may contain information that is privileged and confidential. Any unauthorized dissemination, distribution or copying is strictly prohibited. If you receive this transmission in error, please contact network@rfa.org.
participants (1)
-
Michael McFadden