Unicode [was Re: Cult-like behaviour]
steve+comp.lang.python at pearwood.info
Mon Jul 16 14:31:46 EDT 2018
On Mon, 16 Jul 2018 10:27:18 -0700, Jim Lee wrote:
> Had you actually read my words with *intent* rather than *reaction*, you
> would notice that I suggested the *option* of turning off Unicode.
Yes, I know what you wrote, and I read it with intent.
Jim, you seem to be labouring under the misapprehension that anytime
somebody spots a flaw in your argument, or an unpleasant implication of
your words, it can only be because they must not have read your words
carefully. Believe me, that is not the case.
YOU are the one who raised the specter of politically correct groupthink,
not me. That's dog-whistle politics. But okay, let's move on from that.
You say that all you want is a switch to turn off Unicode (and replace it
with what? Kanji strings? Cyrillic? Shift_JS? no of course not, I'm being
absurd -- replace it with ASCII, what else could any right-thinking
person want, right?). Let's look at this from a purely technical
Python already has two string data types, bytes and text. You want
something that is almost functionally identical to bytes, but to call it
text, presumably because you don't want to have to prefix your strings
with a b"" (that was also Marko's objection to byte strings).
Let's say we do it. Now we have three string implementations that need to
be added, documented, tested, maintained, instead of two.
(Are you volunteering to do this work?)
Now we need to double the testing: every library needs to be tested
twice, once with the "Unicode text" switch on, once with it off, to
ensure that features behave as expected in the appropriate mode.
Is this switch a build-time option, so that we have interpreters built
with support for Unicode and interpreters built without it? We've been
there: it's a horribly bad idea. We used to have Python builds with
threading support, and others without threading support. We used to have
Python builds with "wide Unicode" and others with "narrow Unicode".
Nothing good comes of this design.
Or perhaps the switch is a runtime global option?
Surely you can imagine the opportunities for bugs, both obvious crashing
bugs and non-obvious silent failure bugs, that will occur when users run
libraries intended for one mode under the other mode. Not every library
is going to be fully tested under both modes.
Perhaps it is a compile-time option that only affects the current module,
like the __future__ imports. That's a bit more promising, it might even
use the __future__ infrastructure -- but then you have the problem of
interaction between modules that have this switch enabled and those that
have it disabled.
More complexity, more cruft, more bugs.
It's not clear that your switch gives us *any* advantage at all, except
the warm fuzzy feelings that no dirty foreign characters might creep into
our pure ASCII strings. Hmm, okay, but frankly apart from when I copy and
paste code from the internet and it ends up bringing in en-dashes and
curly quotes instead of hyphens and type-writer quotes, that never
happens to me by accident, and I'm having a lot of trouble seeing how it
If you want ASCII byte strings, you have them right now -- you just have
to use the b"" string syntax.
If you want ASCII strings without the b prefix, you have them right now.
Just use only ASCII characters in your strings.
I'm simply not seeing the advantage of:
from __future__ import no_unicode
print("Hello World!") # stand in for any string handling on ASCII
which works just as well if you control the data you are working with and
know that it is pure ASCII.
"Ever since I learned about confirmation bias, I've been seeing
it everywhere." -- Jon Ronson
More information about the Python-list