Non-unicode strings & Python.

"Martin v. Löwis" martin at
Tue Aug 31 07:48:57 CEST 2004

Jonathon Blake wrote:
> What happens when strings are read in from text files that were
> created using  GB 2312-1980, or KPS 9566-2003, or other, equally
> obscure code ranges?

Python has two kinds of strings: byte strings, and Unicode strings.
If you read data from a file, you get byte strings - i.e. a sequence
of bytes representing literally the encoded contents of the file.
If you want Unicode strings, you need to use

> The idea is to read text in the file format, and replace it with the
> appropriate Unicode character,then write it out as a new text file. 
> [Trivial to program, but incredibly time consuming to actually code]

Not at all:

data =, "r", encoding="gb2312"), "w", encoding="utf-8").write(data)

assuming that by "appropriate Unicode character" you actually mean
"I want to write the file encoded as UTF-8".


More information about the Python-list mailing list