On November 11, 2015 at 2:08:00 PM, Nathaniel Smith (njs@pobox.com) wrote:
On Wed, Nov 11, 2015 at 10:42 AM, Donald Stufft wrote:
On November 11, 2015 at 1:38:38 PM, Nathaniel Smith (njs@pobox.com) wrote:
Guaranteeing a clean stdout/stderr is hard: it means you have to be careful to correctly capture and process the output of every child you invoke (e.g. compilers), and deal correctly with the tricky aspects of pipes (deadlocks, sigpipe, ...). And even then you can get thwarted by accidentally importing the wrong library into your main process, and discovering that it writes directly to stdout/stderr on some error condition. And it may or may not respect your resetting of sys.stdout/sys.stderr at the python level. So to be really reliable the only thing to do is to create some pipes and some threads to read the pipes and do the dup2 dance (but not everyone will actually do this, they'll just accept corrupted output on errors) and ugh, all of this is a huge hassle that massively raises the bar on implementing simple build systems.
How is this not true for a worker.py process as well? If the worker process communicates via stdout then it has to make sure it captures the stdout and redirects it before calling into the Python API and then undoes that afterwords. It makes it harder to do incremental output actually because a Python function can’t return in the middle of execution so we’d need to make it some sort of akward generator protocol to make that happen too.
Did you, uh, read the second half of my email? :-) My actual position is that we shouldn't even try to get structured incremental output from the build system, and should stick with the current approach of unstructured incremental output on stdout/stderr. But if we do insist on getting structured incremental output, then I described a system that's much easier for backends to implement, while leaving it up to the frontend to pick whether they want to bother doing complicated redirection tricks, and if so then which particular variety of complicated redirection trick they like best.
In both approaches, yeah, any kind of incremental output is eventually come down to some Python code issuing some sort of function call that reports progress without returning, whether that's sys.stdout.write(json.dumps(...)) or progress_reporter.report_update(...). Between those two options, it's sys.stdout.write(json.dumps(...)) that looks more awkward to me.
I’m confused how the progress indicator you just implemented would work if there wasn’t something triggering a “hey I’m still doing work” to incrementally output information. ----------------- Donald Stufft PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA