[Distutils] command line versus python API for build system abstraction (was Re: build system abstraction PEP)

[Nathaniel Smith]

On Wed, Nov 11, 2015 at 11:12 AM, Robert Collins
<robertc at robertcollins.net> wrote:
> On 12 November 2015 at 08:07, Nathaniel Smith <njs at pobox.com> wrote:
>> On Wed, Nov 11, 2015 at 10:42 AM, Donald Stufft <donald at stufft.io> wrote:
>>> On November 11, 2015 at 1:38:38 PM, Nathaniel Smith (njs at pobox.com) wrote:
>>>> > Guaranteeing a clean stdout/stderr is hard: it means you have
>>>> to be careful to correctly capture and process the output of every
>>>> child you invoke (e.g. compilers), and deal correctly with the
>>>> tricky aspects of pipes (deadlocks, sigpipe, ...). And even
>>>> then you can get thwarted by accidentally importing the wrong
>>>> library into your main process, and discovering that it writes
>>>> directly to stdout/stderr on some error condition. And it may
>>>> or may not respect your resetting of sys.stdout/sys.stderr
>>>> at the python level. So to be really reliable the only thing to
>>>> do is to create some pipes and some threads to read the pipes and
>>>> do the dup2 dance (but not everyone will actually do this, they'll
>>>> just accept corrupted output on errors) and ugh, all of this is
>>>> a huge hassle that massively raises the bar on implementing simple
>>>> build systems.
>>>
>>> How is this not true for a worker.py process as well? If the worker process communicates via stdout then it has to make sure it captures the stdout and redirects it before calling into the Python API and then undoes that afterwords. It makes it harder to do incremental output actually because a Python function can’t return in the middle of execution so we’d need to make it some sort of akward generator protocol to make that happen too.
>>
>> Did you, uh, read the second half of my email? :-) My actual position
>> is that we shouldn't even try to get structured incremental output
>> from the build system, and should stick with the current approach of
>> unstructured incremental output on stdout/stderr. But if we do insist
>> on getting structured incremental output, then I described a system
>> that's much easier for backends to implement, while leaving it up to
>> the frontend to pick whether they want to bother doing complicated
>> redirection tricks, and if so then which particular variety of
>> complicated redirection trick they like best.
>>
>> In both approaches, yeah, any kind of incremental output is eventually
>> come down to some Python code issuing some sort of function call that
>> reports progress without returning, whether that's
>> sys.stdout.write(json.dumps(...)) or
>> progress_reporter.report_update(...). Between those two options, it's
>> sys.stdout.write(json.dumps(...)) that looks more awkward to me.
>
> I think there is some big disconnect in the conversation. AIUI Donald
> and Marcus and I are saying that build systems should just use
>
> print("Something happened")
>
> to provide incremental output.

I agree that this is the best approach.

This particular subthread is all hanging off of Paul's message [1]
where he argues that we can't just print arbitrary text to
stdout/stderr, we need, like, structured JSON messages on stdout that
pip can parse while the build is running. (Which implies that you can
*only* have structured JSON messages on stdout, because otherwise
there's no way to tell which bits are supposed to be structured and
which bits are just arbitrary text.) And I said well, I think that's
probably overcomplicated and unnecessary, but if you insist then this
is what it would look like in the different approaches.

(Your current draft does create similar challenges for build backends
because it also uses stdout for passing structured data. But I know
you're in the middle of rewriting it anyway, so maybe this is
irrelevant.)

-n

[1] http://thread.gmane.org/gmane.comp.python.distutils.devel/24760/focus=24792

-- 
Nathaniel J. Smith -- http://vorpus.org