[Tutor] i18n Encoding/Decoding issues

Thu Aug 10 14:32:10 CEST 2006

Hi all,

It seems I can't get rid of my continuous issues i18n with Python :(

I've been through:
http://docs.python.org/lib/module-email.Header.html
and
http://www.reportlab.com/i18n/python_unicode_tutorial.html
to no avail.

Basically, I'm receiving and processing mail that comes with content (from 
an utf-8 accepting form) from many locales (France, Germany, etc)

def splitMessage() does what the name indicates, and send message is the 
code below.

def sendMessage(text):
    to, From, subject, body = splitMessage(text)
    msg = MIMEText(decodeChars(body), 'plain', 'UTF-8')
    msg['From'] = From
    msg['To'] = to
    msg['Subject'] = Header(decodeChars(subject), 'UTF-8')

def decodeChars(str=""):
    if not str: return None
    for characterCode in _characterCodes.keys():
        str = str.replace(characterCode, _characterCodes[characterCode])
    return str

Now as you have noticed, this only works, ie, I get an email sent with the 
i18n characters displayed correctly, after I pretty much wrote my own 
'urldecode' map

_characterCodes ={  "%80" : "&#65533;", "%82" : "&#65533;", "%83" : 
"&#65533;", "%84" : "&#65533;", \
                    "%85" : "&#65533;", "%86" : "&#65533;",	"%87" : 
"&#65533;", "%88" : "&#65533;", \
                    "%89" : "&#65533;", "%8A" : "&#65533;", "%8B" : 
"&#65533;", "%8C" : "&#65533;", \
                    "%8E" : "&#65533;", "%91" : "&#65533;", "%92" : 
"&#65533;", "%93" : "&#65533;", \
                    "%94" : "&#65533;", "%95" : "&#65533;", "%96" : 
"&#65533;", "%97" : "&#65533;", \
...

Which feels like an horrible kludge.
Note that using urlilib.unquote doesn't do it -I get an error saying that it 
is unable to . Replacing my decodeChars

msg = MIMEText(urllib.unquote(body), 'plain', 'UTF-8')

Returns content with i18n characters mangled.

Using unicode(body, 'latin-1').encode('utf-8') doesn't work either. Besides, 
am I the only one to feel that if I want to encode something in UTF-8 it 
doesn't feel intuitive to have to convert to latin-1 first and then encode?

Any ideas? I am dry on other option and really don't want to keep my kludge 
(unless I absolutely have to)

Thanks in advance
Cheers
j

_________________________________________________________________
Download the new Windows Live Toolbar, including Desktop search! 
http://toolbar.live.com/?mkt=en-gb