[Zope] DTML-SQLVAR and Unicode and PostgreSQL

[Dieter Maurer]

Glenn R Williams wrote at 2003-5-21 17:58 -0400:
 > I got one step further: the PostgreSQL log shows my query with converted
 > HTML entites:
 > 
 > LOG:  query: select * from xp_name
 >               where original_name =             
 >               'عامررشيد'
 > 
 > As you can see, this is not what I want! But how do I tell Zope to use
 > utf8 encoding for the query? When building the query in Zope, I
 > specified as an argumen "original_name:ustring". However, the actual
 > query in the Z SQL method only allows a type of "string". 
 > 
 > Am I doing something wrong?

Not you, but someone does...

   I fear it is your browser.

We must find out who decides to code the arabic characters as
XML/SGML character references.

You told us that the value came from an HTML form, right?
I expect, it was a form "POST"ed to the server.
Apparently, the browser used a Content-Type ("text/sgml" or "text/xml")
which Zope does not expect and interpret wrongly.

I see several options:

  *  Use 'METHOD="GET"' in your form. This forces a standard
     compliant browser to use URL-encoding (with characters
     first encoded as UTF-8 and then url-escaped).

  *  HTML allows the server to control the content-types
     for form variables. Try to use the corresponding attribute
     to forbid "text/sgml", "text/xml".

  *  Convert the character references yourself into UTF-8.
     This is easy: first convert to unicode (with "unichr")
     then encode the resulting unicode string as UTF-8
     (with the unicode object's "encode" method).

The last option is probably the most robust one.


Dieter