[Tutor] urllib2, read data with specific encoding

Sander Sweers sander.sweers at gmail.com
Tue Sep 22 23:04:49 CEST 2009


Hello Tutors, Because a website was giving me issues with unicode
character I created a function to force the encoding. I am not sure it
is the correct way to handle these things.

def reader(fobject, encoding='UTF-8'):
    '''Read a fileobject with specified encoding, defaults UTF-8.'''
    r = codecs.getreader(encoding)
    data = r(fobject)
    return data

I would call it like reader(urllib2.urlopen(someurl), 'somencoding').
Now I am looking for advice if this is the proper way of dealing with
these type of issues? Is there better practice maybe?

Many thanks
Sander



More information about the Tutor mailing list