[Distutils] PEP 517 - specifying build system in pyproject.toml

[Steve Dower]

On 20May2017 1315, Paul Moore wrote:
> On 20 May 2017 at 17:36, Steve Dower <steve.dower at python.org> wrote:
>> In general, since most subprocesses (at least on Windows) do not have
>> customizable encodings, the tool that launches them needs to know what the
>> encoding is. Since we don't live in a Python 3.6 world quite yet, that means
>> the tool should read raw bytes from the compiler and encode them to UTF-8.
>
> Did you spot my point that Visual C produces output that's a mixture
> of OEM and ANSI codepages?

[SNIP]

Yes, and it's a perfect example of why the MSVC-specific wrapper should 
be the one to deal with tool encodings. If you forward unencoded bytes 
like this back to the UI, it will have to deal with the mixed encoding.

> I'd be very surprised if build tool developers got this sort of edge
> case correct without at least some guidance from the PEP on the sorts
> of things they need to consider. You suggest "read raw bytes and
> encode them to UTF-8" - but you don't encode bytes, you encode
> strings, so you still need to convert those bytes to a string first,
> and there's no encoding you can reliably use for this. You need to use
> "errors=replace" to ensure you can handle inconsistently encoded data
> without getting an exception.

The "read raw bytes and [transcode] them" comment was meant to be that 
sort of help. I didn't go as far as writing 
`output.decode(output_encoding, errors="replace").encode("utf-8", 
errors="replace")`, but that's basically what I meant to imply. The 
build tool developer is the *only* developer who can get this right, and 
if they can't, then they have to figure out the most appropriate way to 
work around the fact that they can't.

As for defining distutils as incompatible with the PEP, I'm okay with 
that. Updating distutils to use subprocess for launching tools rather 
than spawnv would be a very good start (and likely a good contribution 
for a new contributor), but allowing build tools to continue to be 
written badly is not worthwhile.

Cheers,
Steve