[issue4733] Add a "decode to declared encoding" version of urlopen to urllib
report at bugs.python.org
Thu Oct 20 04:58:32 CEST 2011
Ezio Melotti <ezio.melotti at gmail.com> added the comment:
> Christian Heimes wrote:
> There is no generic and simple way to detect the encoding of a
> remote site. Sometimes the encoding is mentioned in the HTTP header,
> sometimes it's embedded in the <head> section of the HTML document.
FWIW for HTML pages the encoding can be specified in at least 3 places:
* the HTTP headers: e.g. "content-type: text/html; charset=utf-8";
* the XML declaration: e.g. "<?xml version="1.0" encoding="utf-8" ?>";
* the <meta> tag: e.g. "<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
Browsers usually follow this order while searching the encoding, meaning that HTTP headers have the highest priority. The XML declaration is sometimes (mis)used in (X)HTML pages.
Anyway, since urlopen() is a generic function that can download anything, it shouldn't look at XML declarations and meta tags -- that's something parsers should take care of.
Regarding the implementation, wouldn't having a new method on the file-like object returned by urlopen better?
Maybe something like:
>>> page = urlopen(some_url)
>>> page.encoding # get the encoding from the HTTP headers
>>> page.decode() # same as page.read().decode(page.encoding)
The advantage of having these as new methods/attribute is that you can pass the 'page' around and other functions can get back the decoded content if/when they need to. OTOH other file-like objects don't have similar methods, so it might get a bit confusing.
versions: +Python 3.3 -Python 3.2
Python tracker <report at bugs.python.org>
More information about the Python-bugs-list