[Zope3-Users] TAL expression and ascii

[Justin Fletcher]

On Fri, 2007-12-14 at 12:23 +0200, Marius Gedminas wrote:
> On Fri, Dec 14, 2007 at 01:20:52AM +0100, Justin Fletcher wrote:

<snip>

> 
> There's no magic.  "Zope 3 strings are Unicode" is a convention, and
> Zope makes it easy to follow by decoding all HTTP request strings into
> Unicode objects.  If you're migrating existing non-Unicode data into
> ZODB with a simple Python script, you'll have to take care of converting
> your binary strings to Unicode yourself.
> 
> > Am I misunderstanding something or doing something wrong?
> 
> I think so.  If you could show us how you're migrating "some data" into
> the ZODB, we could give you more advice.
> 
> Regards,
> Marius Gedminas

Thanks for your help so far.  It has already given me some things to
try.  Unfortunately I am still unable to get it to completely work,
though I have made progress.

I started to write a long description of how I am retrieving, parsing,
and storing the data, but instead I'll just post code.

I just added the unicode() conversions around the variables, so the code
as it is below will throw an exception because the strings contain
characters outside of the ascii range.  Before converting to Unicode
running 'type(variable)' on all the variables says that they are <type
'str'>.  I have tried passing different 'encoding=' settings to
unicode(), but while it will allow the code to run, and the ZPT pages to
display, the accented characters are not displayed correctly in the
browser.  The encoding= settings I have tried so far are 'utf-8', and
'latin-1'.

One last piece of the puzzle is that if I use the 'mdb-export' command
and dump the results to a file, the unix 'file' command says that the
file is encoded UTF-8.

Lastly, I run the code below like this:
$ zopectl debug
>>> from mysite.migrateFromMDB import migrateFromMDB
>>> migrateFromMDB(root['mysite'])


Thanks again for any help or pointers you might be able to provide,
-Justin

mysite/migrateFromMDB.py:
===============
from BeautifulSoup import BeautifulSoup
from os import popen4, walk
import csv
from mysite.app import Customer, Page


def migrateFromMDB(mysite):
    mdb_export = '/usr/bin/mdb-export -H'
    tables = ['setings', 'pages']
    mysite_data = '/home/justin/web/mysite/'
    
    for root, dirs, files in walk(mysite_data):
        data_file_name = None
        if 'DATA.mdb' in files:
            # We have a database
            data_file_name = 'DATA.mdb'
    
            cur_file = "%s/%s" % (root, data_file_name)
            infile, outfile = popen4("%s %s %s" % (mdb_export, cur_file,
'pages'))
            infile.close()
            print "Reading: %s/%s" % (root, data_file_name)
            reader = csv.reader(outfile)
            for row in reader:
                if row[0] == "Can't alloc filename": continue
                site = root.split('/')[-1]
                if site not in mysite.keys():
                    mysite[site] = Customer() 
                page_id = unicode(row[0])
                name = unicode(row[1])
                content = unicode(row[2])
                order_by = unicode(row[3])
                hidden = unicode(row[18])
                title = unicode(row[19])
                keywords = unicode(row[20])
                description = unicode(row[21])
                
                mysite[site][page_id] = Page()
                cursite = mysite[site][page_id]
                cursite.keywords = keywords
                cursite.description = description
                cursite.order = order_by
                cursite.title = title
                cursite.name = name
                
                soup = BeautifulSoup(content)
                cursite.content = unicode(soup.prettify())
                
            outfile.close()
        
        elif 'data.mdb' in files:
            data_file_name = 'data.mdb'