Re: [Python-Dev] subprocess crossplatformness and async communication
Nick Craig-Wood wrote:
Can this really be made safe without an explicit flow control protocol, such as a pseudo-TTY? stdio reads data from pipes such as stdin in 4K or so chunks. I can easily imagine the child blocking while it waits for its stdin buffer to fill, while the parent in turn blocks waiting for the child's output arrive. Shell pipelines (and the subprocess module as it stands) don't have this problem because they're unidirectional: you read input from one process and write output to another, but you typically don't feed data back to the process you've read it from.
I'm confused. What's wrong with the following? p = Popen('do_something', stdin=PIPE, stdout=PIPE) p.stdin.write('la la la\n') p.stdin.flush() line = p.stdout.readline() p.stdin.write(process(line)) p.stdin.flush() If you want to see if data is available on p.stdout, use the select module (unless you're on Windows). The child process has to flush its output buffer for this to work, but that isn't Python's problem. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
Daniel Stutzbach wrote:
Anatoly covered that in his response to Guido: "You can't launch a process and communicate with it without blocking at some point. The scenario where you can periodically check output and error pipes without blocking and send input is not supported." With the vanilla subprocess Popen implmentation, the stdin.write calls can actually both block if the stdin buffer is full (waiting for the child process to clear space) and the stdout.readline call can definitely block (waiting for the child process to end the line). So it isn't async communication in general that is the concern: it is *non-blocking* async communication. (Since I'm happy with the idea of threaded programs, I'd personally just farm the task off to some worker threads and make it non-blocking that way, but that approach effectively abandons the whole concept of non-blocking IO and it's ability to avoid using threads). As Guido said though, while there doesn't seem to be anything fundamentally wrong with the idea of adding better support for non-blocking IO to subprocess, it's difficult to have an opinion without a concrete proposal for API changes to subprocess.Popen. The linked recipe certainly can't be adopted as is (e.g. due to the dependency on pywin32) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
On Mon, Jan 26, 2009 at 4:43 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
That's true, but for the use cases presented so far, .readline() is satisfactory, unless you have an unusual application that will fill the pipe without sending a single newline (in which case, see below).
If you really need to communicate with multiple subprocesses (which so far has not been suggested as a motivating example), then you can use select(). You don't need non-blocking IO to avoid using threads (although that is a common misconception). If a program never blocks, then it uses 100% of CPU by definition, which is undesirable. ;-) A program just needs select() so it knows which file descriptors it can call os.read() or os.write() on without blocking. -- Daniel Stutzbach, Ph.D. President, Stutzbach Enterprises, LLC <http://stutzbachenterprises.com>
Daniel Stutzbach wrote: [...]
If you really need to communicate with multiple subprocesses (which so far has not been suggested as a motivating example), then you can use select().
Not portably. select() on windows only works on sockets. -Andrew.
Andrew Bennetts wrote:
In addition, select() is exactly what the linked recipe uses to implement non-blocking I/O for subprocesses on non-Windows platforms. I agree the actual use cases need to be better articulated in any RFE that is actually posted, but a cleanly encapsulated approach to non-blocking communication with subprocesses over stdin/out/err certainly sounds like something that could reasonably be added to the subprocess module. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia ---------------------------------------------------------------
I don't think the subprocess module uses stdio.
That would be a bug in the parent process, for expecting output when none will arrive. As a consequence, some child programs might not be suitable for this kind of operation. This is no surprise - some programs are not suitable for automated operation at all, because they bring up windows, and communicate with their environment by means other than stdin and stdout (or, if you want to operate them automatically, you have to use AppleScript, or COM). Regards, Martin
participants (5)
-
Andrew Bennetts -
Daniel Stutzbach -
Hrvoje Niksic -
Martin v. Loewis -
Nick Coghlan