spawning a process with subprocess

[Nick Craig-Wood]

MonkeeSage <MonkeeSage at gmail.com> wrote:
>  Couple of things. You should use poll() on the Popen instance, and
>  should check it explicitly against None (since a 0 return code,
>  meaning exit successfully, will be treated as a false condition the
>  same as None). Also, in your second example, you block the program
>  when you call readlines on the pipe, since readlines blocks until it
>  reaches eof (i.e., until pipe closes stdout, i.e., process is
>  complete). Oh, and you don't have to split the input to the args
>  option yourself, you can just pass a string.

Though passing an array is good practice if you want to avoid passing
user data through the shell.

> So, putting it all together, you want something like:
> 
>  import subprocess, time
> 
>  cmd = "cat somefile"
>  proc = subprocess.Popen(args=cmd, shell=True,
>    stdout=subprocess.PIPE, stdin=subprocess.PIPE,
>    stderr=subprocess.STDOUT, close_fds=True)
> 
>  while 1:
>    time.sleep(1)
>    if proc.poll() != None:
>      break
>    else:
>      print "waiting on child..."
> 
>  print "returncode =", proc.returncode

This works fine unless the command generates a lot of output (more
than 64k on linux) when the output pipe will fill up and the process
will block until it is emptied.

If you run the below with `seq 10000` then it works fine but as
written the subprocess will block forever writing its output pipe
(under linux 2.6.23).

#------------------------------------------------------------
import subprocess, time

cmd = """
for i in `seq 20000`; do
  echo $i
done
exit 42
"""

proc = subprocess.Popen(args=cmd, shell=True,
  stdout=subprocess.PIPE, stdin=subprocess.PIPE,
  stderr=subprocess.STDOUT, close_fds=True)

while 1:
  time.sleep(1)
  if proc.poll() != None:
    break
  else:
    print "waiting on child..."

print "returncode =", proc.returncode
lines = 0
total = 0
for line in proc.stdout:
    lines += 1
    total += len(line)
print "Received %d lines of %d bytes total" % (lines, total)
#------------------------------------------------------------

So you do need to read stuff from your subprocess, but there isn't a
way in the standard library to do that without potentially blocking.

There are a few solutions

1) use the python expect module (not windows)

  http://pexpect.sourceforge.net/

2) set your file descriptors non blocking.  The following recipe shows
a cross platform module to do it.

  http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554

Or just do it with the fcntl module

3) Use a thread to read stuff from your subprocess and allow it to
block on proc.stdout.read()

Here is an example of 2)

#------------------------------------------------------------
import subprocess, time, os
from fcntl import fcntl, F_GETFL, F_SETFL
from errno import EAGAIN

cmd = """
for i in `seq 100000`; do
  echo $i
done
exit 42
"""

proc = subprocess.Popen(args=cmd, shell=True,
  stdout=subprocess.PIPE, stdin=subprocess.PIPE,
  stderr=subprocess.STDOUT, close_fds=True)

# Set non blocking (unix only)
fcntl(proc.stdout, F_SETFL, fcntl(proc.stdout, F_GETFL) | os.O_NONBLOCK)

def read_all(fd):
    out = ""
    while 1:
        try:
            bytes = fd.read(4096)
        except IOError, e:
            if e[0] != EAGAIN:
                raise
            break
        if not bytes:
            break
        out += bytes
    return out

rx = ""
while 1:
  time.sleep(1)
  if proc.poll() != None:
    break
  else:
    print "waiting on child..."
    rx += read_all(proc.stdout)

rx += read_all(proc.stdout)
print "returncode =", proc.returncode
lines = 0
total = 0
for line in rx.split("\n"):
    lines += 1
    total += len(line)
print "Received %d lines of %d bytes total" % (lines, total)
#------------------------------------------------------------

Which runs like this on my machine

$ python subprocess-shell-nb.py
waiting on child...
waiting on child...
waiting on child...
waiting on child...
waiting on child...
waiting on child...
waiting on child...
waiting on child...
returncode = 42
Received 100001 lines of 488895 bytes total

-- 
Nick Craig-Wood <nick at craig-wood.com> -- http://www.craig-wood.com/nick