[Python-ideas] Why decode()/encode() name is harmful

[random832 at fastmail.us]

On Fri, May 29, 2015, at 04:56, anatoly techtonik wrote:
> First, let me start with The Curse of Knowledge
> https://en.wikipedia.org/wiki/Curse_of_knowledge
> which can be summarized as:
> 
> "Once you get something, it becomes hard
> to think how it was to be without it".

Let's think about how it is to be without _the idea that text is a byte
stream in the first place_ - which some people here learned from Python
2, some learned from C, some may have learned from some other language.
It was the way things always were, after all, before Unicode came along.

The language I was using the most immediately before I started using
Python was C#. And C# uses Unicode (well, UTF-16, but the important
thing is that it's not an ASCII-compatible sequence of bytes) for
strings. One could argue that this paradigm - and the attendant "encode"
and "decode" concepts, and stream wrappers that take care of it in the
common cases, are _the future_, and that one day nobody will learn that
text's natural form is as a sequence of ASCII-compatible bytes... even
if text files continue to be encoded that way on the disk.

> Now imaging a person who has a text file. The
> person need to process that with Python. That
> person is probably a journalist and doesn't know
> anything that "any developer should know about
> unicode". In Python 2 he just copy pastes regular
> expressions to match the letter and is happy. In
> Python 3 he needs to *convert* that text to unicode.

You don't have to do so explicitly, if the text file's encoding matches
your locale. You can just open the file and read it, and it will open as
a text-mode stream that takes care of this for you and returns unicode
strings. It's a text file, so you open it in text mode.

Even if it doesn't match your locale, the proper way is to pass an
"encoding" argument to the open function; not to go so deep as to open
it in binary mode and decode the bytes yourself.