[Python-ideas] PEP 540: Add a new UTF-8 mode

[Stephen J. Turnbull]

Victor Stinner writes:

 > Python 3.6 is not exactly in the first or the later category: "it
 > depends".
 > 
 > To read data from the operating system, Python 3.6 behaves in "UNIX
 > mode": os.listdir() *does* return invalid filenames, it uses a funny
 > encoding using surrogates.
 > 
 > To write data back to the operating system, Python 3.6 wears its
 > "Unicode nazi" hat and becomes strict. It's no more possible to write
 > data from from the operating system back to the operating system.
 > Writing a filename read from os.listdir() into stdout or into a text
 > file fails with an encode error.
 > 
 > Subtle behaviour: since Python 3.6, with the POSIX locale, Python 3.6
 > uses the "UNIX mode" but only to write into stdout. It's possible to
 > write a filename into stdout, but not into a text file.

The point of this, I suppose, is that piping to xargs works by
default.

I haven't read the PEPs (don't have time, mea culpa), but my ideal
would be three options:

    --transparent ->  errors=surrogateescape on input and output
    --postel ->  errors=surrogateescape on input, =strict on output
    --unicode-me-harder ->  errors=strict on input and output

with --postel being default.  Unix afficianados with lots of xargs use
can use --transparent.  Since people have different preferences, I
guess there should be an envvar for this.

Others probably should configure open() by open().  I'll try to get to
the PEPs over the weekend but can't promise.

Steve