helping with unicode

Terry Reedy tjreedy at
Tue Jul 3 03:39:08 CEST 2012

On 7/2/2012 8:49 PM, self.python wrote:
> it's a simple source view program.
> the codec of the target website is utf-8
> so I read it and print the decoded

which re-encodes before printing

> --------------------------------------------------------------
> #-*-coding:utf8-*-
> import urllib2
> rf=urllib2.urlopen(r"")
> print'utf-8')
> raw_input()
> ---------------------------------------------------------------
> It works fine on python shell

Do you mean the Windows Command Prompt shell?
> but when I make the file "" and run it,
> Error rises.
> ----------------------------------------------------------------
> Traceback (most recent call last):
>    File "", line 8, in <module>
>      print'utf-8')
> UnicodeEncodeError: 'cp949' codec can't encode character u'u1368' in position 5
> 5122: illegal multibyte sequence
> ---------------------------------------------------------------------
> cp949 is the basic codec of sys.stdout and cmd.exe
> but I have no idea why it doesn't works.

cp949 is a Euro-Korean multibyte encoding whose mapping is given at
u1368 is not in the mapping. There is no reason the utf-8 site would 
restrict itself to the cp949 subset.

Perhap it prints in the interpreter because 2.x uses errors = 'replace' 
rather than 'strict' (as in 3.x).

Try print'utf-8').encode('cp949', errors = 'replace')
Non-cp949 chars will print as '?'.

> printing without decode('utf-8') works fine on IDLE

because IDLE encodes to utf-8, and x.decode('utf-8').encode('utf-8') == x

> but on cmd, it print broken characters

Printing utf-8 encoded bytes as if cp949 encoded bytes is pretty hilariour
> the question may look silly:(
  but I want to know what is the problem

  or how to print the not broken strings.
> thanks for reading.

Terry Jan Reedy

More information about the Python-list mailing list