[Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?
stefan_ml at behnel.de
Wed Jun 29 08:28:41 CEST 2011
Victor Stinner, 28.06.2011 15:43:
> In Python 2, open() opens the file in binary mode (e.g. file.readline()
> returns a byte string). codecs.open() opens the file in binary mode by
> default, you have to specify an encoding name to open it in text mode.
> In Python 3, open() opens the file in text mode by default. (It only
> opens the binary mode if the file mode contains "b".) The problem is
> that open() uses the locale encoding if the encoding is not specified,
> which is the case *by default*. The locale encoding can be:
> - UTF-8 on Mac OS X, most Linux distributions
> - ISO-8859-1 os some FreeBSD systems
> - ANSI code page on Windows, e.g. cp1252 (close to ISO-8859-1) in
> Western Europe, cp952 in Japan, ...
> - ASCII if the locale is manually set to an empty string or to "C", or
> if the environment is empty, or by default on some systems
> - something different depending on the system and user configuration...
> If you develop under Mac OS X or Linux, you may have surprises when you
> run your program on Windows on the first non-ASCII character. You may
> not detect the problem if you only write text in english... until
> someone writes the first letter with a diacritic.
I agree that this is a *very* common source of problems. People write code
that doesn't care about encodings all over the place, and are then
surprised when it stops working at some point, either by switching
environments or by changing the data. I've seen this in virtually all
projects I've ever come to work in. So, eventually, all of that code was
either thrown away or got debugged and fixed to use an explicit (and
usually configurable) encoding.
Consequently, I don't think it's a bad idea to break out of this ever
recurring development cycle by either requiring an explicit encoding right
from the start, or by making the default encoding platform independent. The
opportunity to fix this was very unfortunately missed in Python 3.0.
Personally, I don't buy the argument that it's harder to write quick
scripts if an explicit encoding is required. Most code that gets written is
not just quick scripts, and even those tend to live longer than initially
 Admittedly, most of those projects were in Java, where the situation is
substantially worse than in Python. Java entirely lacks a way to define a
per-module source encoding, and it even lacks a straight forward way to
encode/decode a file with an explicit encoding. So, by default, *both*
input encodings are platform dependent, whereas in Python it's only the
default file encoding, and properly decoding a file is straight forward there.
More information about the Python-Dev