[Python-Dev] open(): set the default encoding to 'utf-8' in Python 3.3?

[Stefan Behnel]

Victor Stinner, 28.06.2011 15:43:
> In Python 2, open() opens the file in binary mode (e.g. file.readline()
> returns a byte string). codecs.open() opens the file in binary mode by
> default, you have to specify an encoding name to open it in text mode.
>
> In Python 3, open() opens the file in text mode by default. (It only
> opens the binary mode if the file mode contains "b".) The problem is
> that open() uses the locale encoding if the encoding is not specified,
> which is the case *by default*. The locale encoding can be:
>
>   - UTF-8 on Mac OS X, most Linux distributions
>   - ISO-8859-1 os some FreeBSD systems
>   - ANSI code page on Windows, e.g. cp1252 (close to ISO-8859-1) in
> Western Europe, cp952 in Japan, ...
>   - ASCII if the locale is manually set to an empty string or to "C", or
> if the environment is empty, or by default on some systems
>   - something different depending on the system and user configuration...
>
> If you develop under Mac OS X or Linux, you may have surprises when you
> run your program on Windows on the first non-ASCII character. You may
> not detect the problem if you only write text in english... until
> someone writes the first letter with a diacritic.

I agree that this is a *very* common source of problems. People write code 
that doesn't care about encodings all over the place, and are then 
surprised when it stops working at some point, either by switching 
environments or by changing the data. I've seen this in virtually all 
projects I've ever come to work in[1]. So, eventually, all of that code was 
either thrown away or got debugged and fixed to use an explicit (and 
usually configurable) encoding.

Consequently, I don't think it's a bad idea to break out of this ever 
recurring development cycle by either requiring an explicit encoding right 
from the start, or by making the default encoding platform independent. The 
opportunity to fix this was very unfortunately missed in Python 3.0.

Personally, I don't buy the argument that it's harder to write quick 
scripts if an explicit encoding is required. Most code that gets written is 
not just quick scripts, and even those tend to live longer than initially 
intended.

Stefan


[1] Admittedly, most of those projects were in Java, where the situation is 
substantially worse than in Python. Java entirely lacks a way to define a 
per-module source encoding, and it even lacks a straight forward way to 
encode/decode a file with an explicit encoding. So, by default, *both* 
input encodings are platform dependent, whereas in Python it's only the 
default file encoding, and properly decoding a file is straight forward there.