[Distutils] command line versus python API for build system abstraction (was Re: build system abstraction PEP)

[Donald Stufft]

On November 11, 2015 at 2:08:00 PM, Nathaniel Smith (njs at pobox.com) wrote:
> On Wed, Nov 11, 2015 at 10:42 AM, Donald Stufft wrote:
> > On November 11, 2015 at 1:38:38 PM, Nathaniel Smith (njs at pobox.com) wrote:
> >> > Guaranteeing a clean stdout/stderr is hard: it means you have
> >> to be careful to correctly capture and process the output of every
> >> child you invoke (e.g. compilers), and deal correctly with the
> >> tricky aspects of pipes (deadlocks, sigpipe, ...). And even
> >> then you can get thwarted by accidentally importing the wrong
> >> library into your main process, and discovering that it writes
> >> directly to stdout/stderr on some error condition. And it may
> >> or may not respect your resetting of sys.stdout/sys.stderr
> >> at the python level. So to be really reliable the only thing to
> >> do is to create some pipes and some threads to read the pipes and
> >> do the dup2 dance (but not everyone will actually do this, they'll
> >> just accept corrupted output on errors) and ugh, all of this is
> >> a huge hassle that massively raises the bar on implementing simple
> >> build systems.
> >
> > How is this not true for a worker.py process as well? If the worker process communicates  
> via stdout then it has to make sure it captures the stdout and redirects it before calling  
> into the Python API and then undoes that afterwords. It makes it harder to do incremental  
> output actually because a Python function can’t return in the middle of execution so  
> we’d need to make it some sort of akward generator protocol to make that happen too.
>  
> Did you, uh, read the second half of my email? :-) My actual position
> is that we shouldn't even try to get structured incremental output
> from the build system, and should stick with the current approach of
> unstructured incremental output on stdout/stderr. But if we do insist
> on getting structured incremental output, then I described a system
> that's much easier for backends to implement, while leaving it up to
> the frontend to pick whether they want to bother doing complicated
> redirection tricks, and if so then which particular variety of
> complicated redirection trick they like best.
>  
> In both approaches, yeah, any kind of incremental output is eventually
> come down to some Python code issuing some sort of function call that
> reports progress without returning, whether that's
> sys.stdout.write(json.dumps(...)) or
> progress_reporter.report_update(...). Between those two options, it's
> sys.stdout.write(json.dumps(...)) that looks more awkward to me.
>  

I’m confused how the progress indicator you just implemented would work if there wasn’t something triggering a “hey I’m still doing work” to incrementally output information.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA