lxml can't output right unicode result

Thu Sep 6 21:14:06 EDT 2012

On 07/09/2012 01:21, contro opinion wrote:
> i eidt a file and save it in gbk encode named test. my system is
> :debian,locale,en.utf-8;python2.6,locale,utf-8.
>
> <html>
> <p>你</p>
> </html>
>
> in terminal i input:
>
> xxd  test
>
> 0000000: 3c68 746d 6c3e 0a3c 703e c4e3 3c2f 703e  <html>.<p>..</p>
> 0000010: 0a3c 2f68 746d 6c3e 0a                   .</html>.
>
> 你 is you in english,
> "\xc4\xe3" is the gbk encode of it.
> "\xe4\xbd\xe3" is the utf-8 encode of it.
> "u\x4f\x60" is the unicode encode of it.
> now i parse it in lxml
>
>  >>> "你"
> '\xe4\xbd\xa0'
>  >>> "你".decode("utf-8")
> u'\u4f60'
>  >>> "你".decode("utf-8").encode("gbk")
> '\xc4\xe3'
>  >>>
>
> code1:
>
>  >>> import lxml.html
>  >>> root=lxml.html.parse("test")
>  >>> d=root.xpath("//p")
>  >>> d[0].text_content()
> u'\xc4\xe3'
>
> in material ,lxml parse file to output the unicode form.
> why the d[0].text_content() can not output u'\x4f\x60'?
>
> code2:
>
> import codecs
> import lxml.html
> f = codecs.open('test', 'r', 'gbk')
> root=lxml.html.parse(f)
> d=root.xpath("//p")
> d[0].text_content()
> u'\xe4\xbd\xa0'
>
> why the d[0].text_content() can not output u'\x4f\x60'?
>
> i am confused by this problem for two days.
>
You can't just put some text into a file and expect it to know
"magically" what the encoding is. You have to specify that the encoding
is GBK, something like this (in a file actually encoded as GBK, of
course):

<html>
<meta http-equiv="content-type" content="text/html; charset=gbk">
<p>你</p>
</html>

I hope there's a good reason why you're using that encoding and not
UTF-8.