polling for output from a subprocess module

[Thomas Bellman]

Christian Heimes <lists at cheimes.de> writes:

> Thomas Bellman wrote:
>> The readlines() method will read until it reaches end of file (or
>> an error occurs), not just what is available at the moment.  You
>> can see that for your self by running:

> Bad idea ;)

Why is it a bad idea to see how the readlines() method behaves?


> readlines() on a subprocess Popen instance will block when you PIPE more
> than one stream and the buffer of the other stream is full.

> You can find some insight at http://bugs.python.org/issue1606. I
> discussed the matter with Guido a while ago.

Umm...  Yes, you are correct that the code in the original post
also has a deadlock problem.  I missed that.  But saying that it
is the readline() method that is blocking is a bit misleading,
IMHO.  Both processes will be blocking, in a deadly embrace.
It's a problem that has been known since the concept of inter-
process communication was invented, and isn't specific to the
readlines() method in Python.

But the OP *also* has the problem that I described in my reply.
Even if he only PIPE:d one of the output streams from his
subprocess, he would only receive its output when the subprocess
finished (if it ever does), not as it is produced.


(To those that don't understand why the OP's code risks a deadly
embrace: if a process (A) writes significant amounts of data to
both its standard output and standard error, but the process that
holds the other end of those streams (process B) only reads data
from one of those streams, process A will after a while fill the
operating system's buffers for the other stream.  When that
happens, the OS will block process A from running until process B
reads data from that stream too, freeing up buffer space.  If
process B never does that, then process A will never run again.

The OP must therefore do a select() on both the standard output
and standard error of his subprocess, and use os.read() to
retrieve the output from both streams to free up buffer space in
the pipes.)


-- 
Thomas Bellman,   Lysator Computer Club,   Linköping University,  Sweden
"We don't understand the software, and        ! bellman @ lysator.liu.se
 sometimes  we don't understand the hardware, ! 
 but we can *see* the blinking lights!"       ! Make Love -- Nicht Wahr!