[Zope3-Users] Unicode for Stupid Americans (like me)?

[Jeff Shell]

I continue to feel like an idiot in the face of Unicode. I finally
understand what a unicode 'string' really is, and what encode and
decode mean (they were previously interchangable in my mind). But I
don't know the best practices.

My desire is to:

- Not have any encode / decode errors. 'ascii codec doesn't recognize
character ... at position ...'. I don't want to keep on bullying
through whenever this pops up.

- Not turn customer input into garbage. It may render to the public
site fine, but sometimes in the admin skin's text areas, things turn
funky. I don't know if there's something I need to do at form-handling
time, or at rendering time, or what... I did a test based on a
document by Sam Ruby, and guess that I'm often getting Latin-1 from
our customers, which doesn't map to UTF-8 (the diacritic marks go
haywire).

  - HOW do I know what a browser has sent me? There doesn't seem to be
a real way of handling this. Do I guess?

- Know without a doubt when to encode, and when to decode. I guess the
"proper" thing to do is to store everything as unicode, and to decode
to unicode as early as possible when input is coming in. But again,
how do I know when to decode from latin-1 and when to decode from
UTF-8? When or why should I encode to one or the other at response
time? Should I worry at all?

If there are any documents, web pages, Zope 3 book chapters, and past
messages that I may have missed or need to look at in more detail,
please let me know. I've had a hard time sifting through all of the
information, and I apoligize if I've missed something written by
anyone here.

It's most important that the documents are from a Zope 3 perspective,
or that the lessons can easily apply to Zope 3. Some of the more
general documents I've found say "your language / database may or may
not natively support unicode, so we can't tell you what to do from
here," which makes it difficult to translate to good Zope idioms.

I feel like I know enough to squeak by, but that's no longer
acceptable. Sometimes I quiver in terror, waiting for everything to
fall down because of something so seemingly basic like strings/text.
It may be a lot of technical debt, or it may be extremely easy to pay
down. In any case, it's time to pay it down. :)

Thanks,
-- 
Jeff Shell