helping with unicode

[Terry Reedy]

On 7/2/2012 8:49 PM, self.python wrote:
> it's a simple source view program.
>
> the codec of the target website is utf-8
> so I read it and print the decoded

which re-encodes before printing

> --------------------------------------------------------------
> #-*-coding:utf8-*-
> import urllib2
>
> rf=urllib2.urlopen(r"http://gall.dcinside.com/list.php?id=programming")
>
> print rf.read().decode('utf-8')
>
> raw_input()
> ---------------------------------------------------------------
>
> It works fine on python shell

Do you mean the Windows Command Prompt shell?
>
> but when I make the file "wrong.py" and run it,
> Error rises.
>
> ----------------------------------------------------------------
> Traceback (most recent call last):
>    File "C:wrong.py", line 8, in <module>
>      print rf.read().decode('utf-8')
> UnicodeEncodeError: 'cp949' codec can't encode character u'u1368' in position 5
> 5122: illegal multibyte sequence
> ---------------------------------------------------------------------
>
> cp949 is the basic codec of sys.stdout and cmd.exe
> but I have no idea why it doesn't works.

cp949 is a Euro-Korean multibyte encoding whose mapping is given at
http://www.unicode.org/Public/MAPPINGS/VENDORS/MICSFT/WINDOWS/CP949.TXT
u1368 is not in the mapping. There is no reason the utf-8 site would 
restrict itself to the cp949 subset.

Perhap it prints in the interpreter because 2.x uses errors = 'replace' 
rather than 'strict' (as in 3.x).

Try print rf.read().decode('utf-8').encode('cp949', errors = 'replace')
Non-cp949 chars will print as '?'.

> printing without decode('utf-8') works fine on IDLE

because IDLE encodes to utf-8, and x.decode('utf-8').encode('utf-8') == x

> but on cmd, it print broken characters

Printing utf-8 encoded bytes as if cp949 encoded bytes is pretty hilariour
>
> the question may look silly:(
  but I want to know what is the problem



  or how to print the not broken strings.
>
> thanks for reading.
>


-- 
Terry Jan Reedy