Thanks Matt for the detailed explanation for why we cannot change `open` to do encoding detection by default. I think that should answer Guido's question.
It still leaves open the possibility of:
- a new mode to open() that opts-in to encoding detection;
- a new built-in function that is only used for opening text files (not pipes) with encoding detection by default;
- or a new function that attempts the detection:
enc = io.guess_encoding(FILENAME) or 'UTF-8' with open(FILENAME, encoding=enc) as f: ...
These may be useful, but I don't think that they are very helpful for solving the problem of naive programmers who don't know anything about encodings trying to open files which are encoded differently from the system encoding. Such users aren't knowledgable enough to know that they should opt-in to encoding detection. If they were, they would probably just set the encoding to "utf-8" in the first place.