Reading Live Output from a Subprocess

[Nobody]

On Thu, 05 Apr 2012 23:57:49 -0700, bunslow wrote:

> Okay, I've been trying for days to figure this out, posting on forums,
> Googling, whatever. I have yet to find a solution that has worked for me.
> (I'm using Python 3.2.2, Ubuntu 11.04.) Everything I've tried has led to
> buffered output being spat back all at once after the subprocess
> terminates.

In all probability, this is because the child process (pypy) is
buffering its stdout, meaning that the data doesn't get passed to the OS
until either the buffer is full or the process terminates. If it doesn't
get passed to the OS, then the OS can't pass it on to whatever is on the
read end of the pipe. In that situation, there is nothing that the parent
process can do about it.

> I've heard that the Pexpect module works wonders,

If it does, that would confirm the buffering hypothesis. Pexpect causes
the child process' stdout to be associated with a pty.

The "stdout" stream created by the C library (libc) is initially
line-buffered if it is associated with a tty (according to the isatty()
function) and fully-buffered otherwise. The program can change the
buffering with e.g. setvbuf(), or it can explicitly fflush(stdout) after
each line, but this is up to the program, and is not something which can
be controlled externally (short of "hacking" the child process with
techniques such as ptrace() or LD_PRELOAD).

While the libc behaviour is occasionally inconvenient, it is mandated by
the C standard (7.19.3p7):

	As initially opened, the standard error stream is not fully
	buffered; the standard input and standard output streams are
	fully buffered if and only if the stream can be determined not
	to refer to an interactive device.

It's up to individual programs to force line-buffered output where
appropriate (e.g. GNU grep has the --line-buffered switch, GNU sed has the
-u switch, etc).