[Python-ideas] Fix default encodings on Windows

[Steve Dower]

On 17Aug2016 0901, Nick Coghlan wrote:
> On 17 August 2016 at 02:06, Chris Barker <chris.barker at noaa.gov> wrote:
>> So the Solution is to either:
>>
>>  (A) get everyone to use Unicode  "properly", which will work on all
>> platforms (but only on py3.5 and above?)
>>
>> or
>>
>> (B) kludge some *nix-compatible support for byte paths into Windows, that
>> will work at least much of the time.
>>
>> It's clear (to me at least) that (A) it the "Right Thing", but real world
>> experience has shown that it's unlikely to happen any time soon.
>>
>> Practicality beats Purity and all that -- this is a judgment call.
>>
>> Have I got that right?
>
> Yep, pretty much. Based on Stephen Turnbull's concerns, I wonder if we
> could make a whitelist of universal encodings that Python-on-Windows
> will use in preference to UTF-8 if they're configured as the current
> code page. If we accepted GB18030, GB2312, Shift-JIS, and ISO-2022-*
> as overrides, then problems would be significantly less likely.
>
> Another alternative would be to apply a similar solution as we do on
> Linux with regards to the "surrogateescape" error handler: there are
> some interfaces (like the standard streams) where we only enable that
> error handler specifically if the preferred encoding is reported as
> ASCII. In 2016, we're *very* skeptical about any properly configured
> system actually being ASCII-only (rather than that value showing up
> because the POSIX standards mandate it as the default), so we don't
> really believe the OS when it tells us that.
>
> The equivalent for Windows would be to disbelieve the configured code
> page only when it was reported as "mbcs" - for folks that had
> configured their system to use something other than the default,
> Python would believe them, just as we do on Linux.

The problem here is that "mbcs" is not configurable - it's a 
meta-encoder that uses whatever is configured as the "language (system 
locale) to use when displaying text in programs that do not support 
Unicode" (quote from the dialog where administrators can configure 
this). So there's nothing to disbelieve here.

And even on machines where the current code page is "reliable", UTF-16 
is still the actual encoding, which means UTF-8 is still a better choice 
for representing the path as a blob of bytes. Currently we have 
inconsistent encoding between different Windows machines and could 
either remove that inconsistency completely or simply reduce it for 
(approx.) English speakers. I would rather an extreme here - either make 
it consistent regardless of user configuration, or make it so broken 
that nobody can use it at all. (And note that the correct way to support 
*some* other FS encodings would be to change the return value from 
sys.getfilesystemencoding(), which breaks people who currently ignore 
that just as badly as changing it to utf-8 would.)

Cheers,
Steve