Victor Stinner, 28.06.2011 15:43:
In Python 2, open() opens the file in binary mode (e.g. file.readline() returns a byte string). codecs.open() opens the file in binary mode by default, you have to specify an encoding name to open it in text mode.
In Python 3, open() opens the file in text mode by default. (It only opens the binary mode if the file mode contains "b".) The problem is that open() uses the locale encoding if the encoding is not specified, which is the case *by default*. The locale encoding can be:
- UTF-8 on Mac OS X, most Linux distributions - ISO-8859-1 os some FreeBSD systems - ANSI code page on Windows, e.g. cp1252 (close to ISO-8859-1) in Western Europe, cp952 in Japan, ... - ASCII if the locale is manually set to an empty string or to "C", or if the environment is empty, or by default on some systems - something different depending on the system and user configuration...
If you develop under Mac OS X or Linux, you may have surprises when you run your program on Windows on the first non-ASCII character. You may not detect the problem if you only write text in english... until someone writes the first letter with a diacritic.
I agree that this is a *very* common source of problems. People write code that doesn't care about encodings all over the place, and are then surprised when it stops working at some point, either by switching environments or by changing the data. I've seen this in virtually all projects I've ever come to work in[1]. So, eventually, all of that code was either thrown away or got debugged and fixed to use an explicit (and usually configurable) encoding. Consequently, I don't think it's a bad idea to break out of this ever recurring development cycle by either requiring an explicit encoding right from the start, or by making the default encoding platform independent. The opportunity to fix this was very unfortunately missed in Python 3.0. Personally, I don't buy the argument that it's harder to write quick scripts if an explicit encoding is required. Most code that gets written is not just quick scripts, and even those tend to live longer than initially intended. Stefan [1] Admittedly, most of those projects were in Java, where the situation is substantially worse than in Python. Java entirely lacks a way to define a per-module source encoding, and it even lacks a straight forward way to encode/decode a file with an explicit encoding. So, by default, *both* input encodings are platform dependent, whereas in Python it's only the default file encoding, and properly decoding a file is straight forward there.