[Python-Dev] subprocess crossplatformness and async communication

Nick Coghlan ncoghlan at gmail.com
Mon Jan 26 23:43:24 CET 2009


Daniel Stutzbach wrote:
> I'm confused.  What's wrong with the following?
> 
>     p = Popen('do_something', stdin=PIPE, stdout=PIPE)
>     p.stdin.write('la la la\n')
>     p.stdin.flush()
>     line = p.stdout.readline()
>     p.stdin.write(process(line))
>     p.stdin.flush()
>  
> If you want to see if data is available on p.stdout, use the select
> module (unless you're on Windows).
> 
> The child process has to flush its output buffer for this to work, but
> that isn't Python's problem.

Anatoly covered that in his response to Guido: "You can't launch a
process and communicate with it without blocking at some point. The
scenario where you can periodically check output and error pipes without
blocking and send input is not supported."

With the vanilla subprocess Popen implmentation, the stdin.write calls
can actually both block if the stdin buffer is full (waiting for the
child process to clear space) and the stdout.readline call can
definitely block (waiting for the child process to end the line).

So it isn't async communication in general that is the concern: it is
*non-blocking* async communication. (Since I'm happy with the idea of
threaded programs, I'd personally just farm the task off to some worker
threads and make it non-blocking that way, but that approach effectively
abandons the whole concept of non-blocking IO and it's ability to avoid
using threads).

As Guido said though, while there doesn't seem to be anything
fundamentally wrong with the idea of adding better support for
non-blocking IO to subprocess, it's difficult to have an opinion without
a concrete proposal for API changes to subprocess.Popen. The linked
recipe certainly can't be adopted as is (e.g. due to the dependency on
pywin32)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia
---------------------------------------------------------------


More information about the Python-Dev mailing list