spawning a process with subprocess

[MonkeeSage]

On Nov 27, 4:30 am, Nick Craig-Wood <n... at craig-wood.com> wrote:
> MonkeeSage <MonkeeS... at gmail.com> wrote:
> >  Couple of things. You should use poll() on the Popen instance, and
> >  should check it explicitly against None (since a 0 return code,
> >  meaning exit successfully, will be treated as a false condition the
> >  same as None). Also, in your second example, you block the program
> >  when you call readlines on the pipe, since readlines blocks until it
> >  reaches eof (i.e., until pipe closes stdout, i.e., process is
> >  complete). Oh, and you don't have to split the input to the args
> >  option yourself, you can just pass a string.
>
> Though passing an array is good practice if you want to avoid passing
> user data through the shell.

Well, he was setting shell=True, but I guess being explicit (about
that) is better than implicit. ;)

> > So, putting it all together, you want something like:
>
> >  import subprocess, time
>
> >  cmd = "cat somefile"
> >  proc = subprocess.Popen(args=cmd, shell=True,
> >    stdout=subprocess.PIPE, stdin=subprocess.PIPE,
> >    stderr=subprocess.STDOUT, close_fds=True)
>
> >  while 1:
> >    time.sleep(1)
> >    if proc.poll() != None:
> >      break
> >    else:
> >      print "waiting on child..."
>
> >  print "returncode =", proc.returncode
>
> This works fine unless the command generates a lot of output (more
> than 64k on linux) when the output pipe will fill up and the process
> will block until it is emptied.
>
> If you run the below with `seq 10000` then it works fine but as
> written the subprocess will block forever writing its output pipe
> (under linux 2.6.23).
>
> #------------------------------------------------------------
> import subprocess, time
>
> cmd = """
> for i in `seq 20000`; do
>   echo $i
> done
> exit 42
> """
>
> proc = subprocess.Popen(args=cmd, shell=True,
>   stdout=subprocess.PIPE, stdin=subprocess.PIPE,
>   stderr=subprocess.STDOUT, close_fds=True)
>
> while 1:
>   time.sleep(1)
>   if proc.poll() != None:
>     break
>   else:
>     print "waiting on child..."
>
> print "returncode =", proc.returncode
> lines = 0
> total = 0
> for line in proc.stdout:
>     lines += 1
>     total += len(line)
> print "Received %d lines of %d bytes total" % (lines, total)
> #------------------------------------------------------------
>
> So you do need to read stuff from your subprocess, but there isn't a
> way in the standard library to do that without potentially blocking.
>
> There are a few solutions
>
> 1) use the python expect module (not windows)
>
>  http://pexpect.sourceforge.net/
>
> 2) set your file descriptors non blocking.  The following recipe shows
> a cross platform module to do it.
>
>  http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/440554
>
> Or just do it with the fcntl module
>
> 3) Use a thread to read stuff from your subprocess and allow it to
> block on proc.stdout.read()
>
> Here is an example of 2)
>
> #------------------------------------------------------------
> import subprocess, time, os
> from fcntl import fcntl, F_GETFL, F_SETFL
> from errno import EAGAIN
>
> cmd = """
> for i in `seq 100000`; do
>   echo $i
> done
> exit 42
> """
>
> proc = subprocess.Popen(args=cmd, shell=True,
>   stdout=subprocess.PIPE, stdin=subprocess.PIPE,
>   stderr=subprocess.STDOUT, close_fds=True)
>
> # Set non blocking (unix only)
> fcntl(proc.stdout, F_SETFL, fcntl(proc.stdout, F_GETFL) | os.O_NONBLOCK)
>
> def read_all(fd):
>     out = ""
>     while 1:
>         try:
>             bytes = fd.read(4096)
>         except IOError, e:
>             if e[0] != EAGAIN:
>                 raise
>             break
>         if not bytes:
>             break
>         out += bytes
>     return out
>
> rx = ""
> while 1:
>   time.sleep(1)
>   if proc.poll() != None:
>     break
>   else:
>     print "waiting on child..."
>     rx += read_all(proc.stdout)
>
> rx += read_all(proc.stdout)
> print "returncode =", proc.returncode
> lines = 0
> total = 0
> for line in rx.split("\n"):
>     lines += 1
>     total += len(line)
> print "Received %d lines of %d bytes total" % (lines, total)
> #------------------------------------------------------------
>
> Which runs like this on my machine
>
> $ python subprocess-shell-nb.py
> waiting on child...
> waiting on child...
> waiting on child...
> waiting on child...
> waiting on child...
> waiting on child...
> waiting on child...
> waiting on child...
> returncode = 42
> Received 100001 lines of 488895 bytes total
>
> --
> Nick Craig-Wood <n... at craig-wood.com> --http://www.craig-wood.com/nick

Nice. Thanks for the recipe link too.

Regards,
Jordan