Convertion of Unicode to ASCII NIGHTMARE

[Paul Boddie]

ChaosKCW wrote:
> Hi
>
> I am reading from an oracle database using cx_Oracle. I am writing to a
> SQLite database using apsw.
>
> The oracle database is returning utf-8 characters for euopean item
> names, ie special charcaters from an ASCII perspective.

And does cx_Oracle return those as Unicode objects or as plain strings
containing UTF-8 byte sequences? It's very important to distinguish
between these two cases, and I don't have any experience with cx_Oracle
to be able to give advice here.

> I get the following error:
> >    SQLiteCur.execute(sql, row)
> >UnicodeDecodeError: 'ascii' codec can't decode byte 0xdc in position 12: ordinal not
> > in range(128)

It looks like you may have Unicode objects that you're presenting to
sqlite. In any case, with earlier versions of pysqlite that I've used,
you need to connect with a special unicode_results parameter, although
later versions should work with Unicode objects without special
configuration. See here for a thread (in which I seem to have
participated, coincidentally):

http://mail.python.org/pipermail/python-list/2002-June/107526.html

> I have googled for serval days now and still cant get it to encode to
> ascii.

This is a tough thing to find out - whilst previous searches did
uncover some discussions about it, I just tried and failed to find the
enlightening documents - and I certainly didn't see many references to
it on the official pysqlite site.

Paul