[Distutils] command line versus python API for build system abstraction (was Re: build system abstraction PEP)

[Nathaniel Smith]

On Wed, Nov 11, 2015 at 10:42 AM, Donald Stufft <donald at stufft.io> wrote:
> On November 11, 2015 at 1:38:38 PM, Nathaniel Smith (njs at pobox.com) wrote:
>> > Guaranteeing a clean stdout/stderr is hard: it means you have
>> to be careful to correctly capture and process the output of every
>> child you invoke (e.g. compilers), and deal correctly with the
>> tricky aspects of pipes (deadlocks, sigpipe, ...). And even
>> then you can get thwarted by accidentally importing the wrong
>> library into your main process, and discovering that it writes
>> directly to stdout/stderr on some error condition. And it may
>> or may not respect your resetting of sys.stdout/sys.stderr
>> at the python level. So to be really reliable the only thing to
>> do is to create some pipes and some threads to read the pipes and
>> do the dup2 dance (but not everyone will actually do this, they'll
>> just accept corrupted output on errors) and ugh, all of this is
>> a huge hassle that massively raises the bar on implementing simple
>> build systems.
>
> How is this not true for a worker.py process as well? If the worker process communicates via stdout then it has to make sure it captures the stdout and redirects it before calling into the Python API and then undoes that afterwords. It makes it harder to do incremental output actually because a Python function can’t return in the middle of execution so we’d need to make it some sort of akward generator protocol to make that happen too.

Did you, uh, read the second half of my email? :-) My actual position
is that we shouldn't even try to get structured incremental output
from the build system, and should stick with the current approach of
unstructured incremental output on stdout/stderr. But if we do insist
on getting structured incremental output, then I described a system
that's much easier for backends to implement, while leaving it up to
the frontend to pick whether they want to bother doing complicated
redirection tricks, and if so then which particular variety of
complicated redirection trick they like best.

In both approaches, yeah, any kind of incremental output is eventually
come down to some Python code issuing some sort of function call that
reports progress without returning, whether that's
sys.stdout.write(json.dumps(...)) or
progress_reporter.report_update(...). Between those two options, it's
sys.stdout.write(json.dumps(...)) that looks more awkward to me.

-n

-- 
Nathaniel J. Smith -- http://vorpus.org