Unicode from Web to MySQL

[Bill Eldridge]

Note that I am able to do create Unicode data and insert it
with a carefully controlled unicode string

data = u"Make \u0633\u0644\u0627\u0645, not war"
c.execute ( INSERT INTO junk (junklet) VALUES ('%s') ''' % 
data.encode('utf-8','ignore')

but this won't work with what I find on the Web.

Thanks

Bill Eldridge wrote:

>
> I'm trying to grab a document off the Web and toss it
> into a MySQL database, but I keep running into the
> various encoding problems with Unicode (that aren't
> a problem for me with GB2312, BIG 5, etc.)
>
> What I'd like is something as simple as:
>
> CREATE TABLE junk (junklet VARCHAR(2500) CHARACTER SET UTF8));
>
> import MySQLdb, re,urllib
>
> data = urllib.urlopen('http://localhost/test.html').read()
>
> data2 = ???
> ...
> c.execute(''' INSERT INTO junk ( junklet) VALUES ( '%s') ''' % data2 )