[Zope] UnicodeDecodeError: 'ascii' codec can't decode

[Jon Bowlas]

Hi all,

I'm not sure if any of you are familiar with Silva (CMS) but I’ve created a
SQL code source which works fine when tested in the ZMI but when attempting
to add this code source into a Silva document via the Silva Management
Interface I get a UnicodeDecodeError. I think I know why this is but I’m not
sure how to resolve it, so any suggestions would be appreciated. 

Basically my 'Silva SQL Source' generates a MultiListField by extracting its
values from a database however there are some quite unusual characters
contained in the returned recordset for the MultiListField, see below the
tuple I generate from the recordset.

[('Developmental Neurobiology', 'ANAT2008'), ('Neuroanatomy', 'ANAT2009'),
('The Evolution and Development of Body Form - Visceral Systems',
'ANAT2022'), ('The Evolution and\r\nDevelopment of Body Form \xbf
The\r\nMusculoskeletal System', 'ANAT2023'), ("Man's Place in
Nature,\r\n1750\xbf1900: The Debate in Britain", 'HMED3001'), ('Medicine,
Disease and\r\nSociety from Antiquity to the\r\nRenaissance', 'HMED3003'),
('Madness and Society', 'HMED3004'), ('Medicine and Modern\r\nSociety',
'HMED3006'), ('Colonial and Empire\r\nMedicine', 'HMED3009'), ('Disease in
History', 'HMED3010'), ('The Rise of Modern\r\nMedicine, 1650\xbf1900',
'HMED3013'), ('Medicine in Asia', 'HMED3014'), ('Advanced Cell Biology',
'ANAT3001'), ('Mechanisms of Development', 'ANAT3003'), ('Neural Basis of
Motivation and Learning', 'ANAT3018'), ('Advanced Neuroanatomy',
'ANAT3025'), ('Neurobiology of Neurodegenerative Diseases', 'ANAT3028'),
('Cellular and Developmental Neurobiology', 'ANAT3030'), ('The Control of
Movement', 'ANAT3031'), ('Neural Computation and Models of Brain Function',
'ANAT3041'), ('The Peripheral Nervous System', 'ANAT3033'), ('Pain',
'ANAT3042'), ('Biological Timing (Chronobiology)', 'ANAT3043'),
('Neurobiology of Vision', 'ANAT3044'), ('Integrated Downstream Processing',
'BENG4001'), ('Design and Control of Biochemical Reactors', 'BENG4002'),
('Bioprocess Validation and Quality Control', 'BENG4010'), ('Integrated
Biochemical Engineering Design', 'BENG4011'), ('Bioprocess Design and
Implementation', 'BENG4012'), ('- Research Project', 'Option 1'),
('Bioprocess Research Project and', 'BENG4005'), ('Bioprocess Experimental
Study', 'BENG4006'), ('- Biological Sciences', 'Option 2'), ('Molecular
Biology for Biochemical Engineers', 'BIOC4601'), ('Structures of Biological
Polymers', 'BIOC4602'), ('Metabolic Pathways and\r\ntheir Regulation',
'BIOC4603'), ('- Process Engineering', 'Option 3'), ('Introduction to Human
Anatomy', 'ANAT1003')]

As you can see this containes some strange chars particularly  
\r\n which I believe is a carriage return, and
\xbf which I believe represents a hyphen - 
So I’m assuming that this is the reason for the failure in the SMI. And
would like to find a way to either replace these characters or alternatively
change the recordset encoding prior to generating the MultiListField.

I tried replacing these characters by converting the tuple for my
MultiListField into a string first using a python script but it didn’t seem
to work (do anything):

modules = context.module_retriever()
mytuple = modules.tuples()
mystring =str(mytuple).replace('\xbf','-')
return mystring

N.B. module_retriever is my Z SQL method that returns the resultset.

Does anyone have any ideas as to how I can resolve this?

Jon
UCL