[Zope3-Users] TAL expression and ascii

[Marius Gedminas]

On Fri, Dec 14, 2007 at 01:20:52AM +0100, Justin Fletcher wrote:
> I am trying to migrate some data into the ZODB.  I am storing this data
> in a string, and later presenting the data in a ZPT similar to this:
> 
> <span tal:replace="context/content" />
> 
> This works fine for text that does not have any characters outside of
> the ascii range, but for text that does I receive this error:
> 
> UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position
> 3283: ordinal not in range(128)

You are mixing non-ASCII str objects with Unicode strings somewhere.

> The characters are typically things like what the HTML code &eacute;
> would generate, but it is a fairly significant amount of text so I am
> unsure specifically what characters are causing the problem.

Any that are non-ASCII.

> My understanding is that Zope3 strings are Unicode so I don't understand
> why the ascii range is a restriction.  Shouldn't these characters be
> stored and presented transparently?

There's no magic.  "Zope 3 strings are Unicode" is a convention, and
Zope makes it easy to follow by decoding all HTTP request strings into
Unicode objects.  If you're migrating existing non-Unicode data into
ZODB with a simple Python script, you'll have to take care of converting
your binary strings to Unicode yourself.

> Am I misunderstanding something or doing something wrong?

I think so.  If you could show us how you're migrating "some data" into
the ZODB, we could give you more advice.

Regards,
Marius Gedminas
-- 
lg_PC.gigacharset (lg = little green men language, PC = proxima centauri)
	-- Markus Kuhn provides an example of a locale
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 189 bytes
Desc: Digital signature
Url : http://mail.zope.org/pipermail/zope3-users/attachments/20071214/35d7a5f1/attachment.bin