[Distutils] command line versus python API for build system abstraction (was Re: build system abstraction PEP)

[Robert Collins]

On 12 November 2015 at 08:07, Nathaniel Smith <njs at pobox.com> wrote:
> On Wed, Nov 11, 2015 at 10:42 AM, Donald Stufft <donald at stufft.io> wrote:
>> On November 11, 2015 at 1:38:38 PM, Nathaniel Smith (njs at pobox.com) wrote:
>>> > Guaranteeing a clean stdout/stderr is hard: it means you have
>>> to be careful to correctly capture and process the output of every
>>> child you invoke (e.g. compilers), and deal correctly with the
>>> tricky aspects of pipes (deadlocks, sigpipe, ...). And even
>>> then you can get thwarted by accidentally importing the wrong
>>> library into your main process, and discovering that it writes
>>> directly to stdout/stderr on some error condition. And it may
>>> or may not respect your resetting of sys.stdout/sys.stderr
>>> at the python level. So to be really reliable the only thing to
>>> do is to create some pipes and some threads to read the pipes and
>>> do the dup2 dance (but not everyone will actually do this, they'll
>>> just accept corrupted output on errors) and ugh, all of this is
>>> a huge hassle that massively raises the bar on implementing simple
>>> build systems.
>>
>> How is this not true for a worker.py process as well? If the worker process communicates via stdout then it has to make sure it captures the stdout and redirects it before calling into the Python API and then undoes that afterwords. It makes it harder to do incremental output actually because a Python function can’t return in the middle of execution so we’d need to make it some sort of akward generator protocol to make that happen too.
>
> Did you, uh, read the second half of my email? :-) My actual position
> is that we shouldn't even try to get structured incremental output
> from the build system, and should stick with the current approach of
> unstructured incremental output on stdout/stderr. But if we do insist
> on getting structured incremental output, then I described a system
> that's much easier for backends to implement, while leaving it up to
> the frontend to pick whether they want to bother doing complicated
> redirection tricks, and if so then which particular variety of
> complicated redirection trick they like best.
>
> In both approaches, yeah, any kind of incremental output is eventually
> come down to some Python code issuing some sort of function call that
> reports progress without returning, whether that's
> sys.stdout.write(json.dumps(...)) or
> progress_reporter.report_update(...). Between those two options, it's
> sys.stdout.write(json.dumps(...)) that looks more awkward to me.

I think there is some big disconnect in the conversation. AIUI Donald
and Marcus and I are saying that build systems should just use

print("Something happened")

to provide incremental output.

-Rob

-- 
Robert Collins <rbtcollins at hp.com>
Distinguished Technologist
HP Converged Cloud