On 17Aug2016 0901, Nick Coghlan wrote:
On 17 August 2016 at 02:06, Chris Barker
So the Solution is to either:
(A) get everyone to use Unicode "properly", which will work on all platforms (but only on py3.5 and above?)
(B) kludge some *nix-compatible support for byte paths into Windows, that will work at least much of the time.
It's clear (to me at least) that (A) it the "Right Thing", but real world experience has shown that it's unlikely to happen any time soon.
Practicality beats Purity and all that -- this is a judgment call.
Have I got that right?
Yep, pretty much. Based on Stephen Turnbull's concerns, I wonder if we could make a whitelist of universal encodings that Python-on-Windows will use in preference to UTF-8 if they're configured as the current code page. If we accepted GB18030, GB2312, Shift-JIS, and ISO-2022-* as overrides, then problems would be significantly less likely.
Another alternative would be to apply a similar solution as we do on Linux with regards to the "surrogateescape" error handler: there are some interfaces (like the standard streams) where we only enable that error handler specifically if the preferred encoding is reported as ASCII. In 2016, we're *very* skeptical about any properly configured system actually being ASCII-only (rather than that value showing up because the POSIX standards mandate it as the default), so we don't really believe the OS when it tells us that.
The equivalent for Windows would be to disbelieve the configured code page only when it was reported as "mbcs" - for folks that had configured their system to use something other than the default, Python would believe them, just as we do on Linux.
The problem here is that "mbcs" is not configurable - it's a meta-encoder that uses whatever is configured as the "language (system locale) to use when displaying text in programs that do not support Unicode" (quote from the dialog where administrators can configure this). So there's nothing to disbelieve here. And even on machines where the current code page is "reliable", UTF-16 is still the actual encoding, which means UTF-8 is still a better choice for representing the path as a blob of bytes. Currently we have inconsistent encoding between different Windows machines and could either remove that inconsistency completely or simply reduce it for (approx.) English speakers. I would rather an extreme here - either make it consistent regardless of user configuration, or make it so broken that nobody can use it at all. (And note that the correct way to support *some* other FS encodings would be to change the return value from sys.getfilesystemencoding(), which breaks people who currently ignore that just as badly as changing it to utf-8 would.) Cheers, Steve