[Zope-dev] ZTUtils make_query and unicode

[Michael McFadden]

Hello list.

I'm having an issue with ZTUtils.Zope.make_query when providing Unicode 
values.

Specifically, We're building search results pagination using 
CMFPlone.PloneBatch

Here's the scenario:

The search form is provided a Unicode string.  The query is properly 
written as "?search_text:utf8:ustring=value"
The search will return many results.

Our searchResults browser view will provide a PloneBatch object, making 
it very easy to display only, for example, 30 results, then provide page 
links at the bottom of the page for the next 30 results, or whatever.

These links will contain the original query string, but when they are 
built using make_query(), the result is "?search_text=value"
   - What follows is Unicode decode errors when the 'next' link is 
followed - the search_text is marshaled as a 'str', not as a 'unicode'

The bottom line is that make_query (actually, complex_marshal & 
simple_marshal) doesn't seem to apply the ':ustring' type prefix when 
creating query strings.

Kind of important to us at Radio Free Asia, considering we host 
Cantonese and Korean languages, for example.

This was fixed by using sys.setdefaultencoding('utf-8'), but that seems 
like quite a sledgehammer fix.

----------

I have a proposed fix, branched from the master branch on Github.
    Diff here: https://github.com/zopefoundation/Zope/pull/4/files

Questions:
   1.  Is "Master" the correct place to fork from?

   2.  Is is safe - at least to start - to assume utf8 encoding when 
constructing query strings?

   3.  Dangerous idea:  If simpleMarshal is passed a unicode object that 
can be represented as ASCII, should we "magically" turn it into a 
string?    I don't like magic, but I also know it's common (and good) 
practice to use unicode in ASCII-based languages, making this exercise 
an overkill for those languages.

Any comments or criticisms (not to hard please!) would be appreciated!

-Flip

-- 
Mike McFadden
Radio Free Asia
Technical Operations Division
2025 M Street NW
Washington DC 20036 USA

This e-mail message is intended only for the use of the addressee and may contain information that is privileged and confidential.  Any unauthorized dissemination, distribution or copying is strictly prohibited.  If you receive this transmission in error, please contact network at rfa.org.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.zope.org/pipermail/zope-dev/attachments/20131029/b72452b9/attachment.html>