[Baypiggies] Subprocess: Common problem/common pattern

[Jeremy Fishman]

As Luca mentioned, there is pexpect.

If you don't want an external library dependency, and the underlying process
produces interactive output (and doesn't explicitly flush) the simplest
solution is to open a pseudo terminal.  There are several ways to interact
with the terminal, documented at http://docs.python.org/library/pty.html,
but here's one way I've coded before:

#!/usr/bin/python
import os
import pty
import subprocess as sp
import sys

# spawn subprocess with input/output connected to a pseudo-terminal
master, slave = pty.openpty()
tstP = sp.Popen(['/usr/bin/python','foo.py'], stdin=slave, stdout=slave,
                stderr=sp.STDOUT, close_fds=True)
# read an write on the terminal file descriptors
sys.stdout.write(os.read(master, 1024))
# we happily execute thanks to the pseudo-terminal
os.write(master, 'bar\n')
sys.stdout.write(os.read(master, 4096))

(foo.py)

#!/usr/bin/python
print "hello world"
foo = raw_input('? ')
print 'foo', foo


 - Jeremy


--
Jeremy R. Fishman
Developer @ Quantcast
jeremy.r.fishman at gmail.com

On Sat, Oct 2, 2010 at 10:14 AM, Glen Jarvis <glen at glenjarvis.com> wrote:

> What is needed is the input as it's being generated.
>
> For example, the C-program counts to 10, with a second delay between each
> count.
>
> The python program, when it calls the C program, will have a 10 second
> delay before it sees anything. Ideally, the python program can be printing
> the results of the C program as they are being generated.
>
>
> Thanks for asking for clarification...
>
>
> Cheers,
>
>
>
> Glen
>
> On Sat, Oct 2, 2010 at 10:05 AM, joshua kaderlan <jkaderlan at yahoo.com>wrote:
>
>> Reading this, it's not entirely clear what the problem is. Is it that
>> subprocess.Popen buffers output, rather than returning it immediately the
>> way os.system() does?
>>
>>
>> *From:* Glen Jarvis <glen at glenjarvis.com>
>> *To:* Baypiggies <baypiggies at python.org>
>> *Sent:* Sat, October 2, 2010 9:44:05 AM
>> *Subject:* [Baypiggies] Subprocess: Common problem/common pattern
>>
>> I've now seen a common problem come up several times. And, I imagine there
>> is a common solution to this problem that I don't know.
>>
>> About a year ago, I wrote an automatic script that would automatically do
>> an 'svn export' of certain branches of a tree depending upon what menu
>> option the customer chose. When I did the svn export, I used
>> subprocess.Popen.
>>
>> The pattern was similar to the following:
>>
>> print """This output is being buffered so I can read the version number.
>>
>>     ....  I'm not stuck, just busy exporting files....
>>
>> """
>>
>> .....
>>
>> process = subprocess.Popen(['svn', 'export', repository,
>> repo_name], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>> stdoutdata, stderrdata = process.communicate()
>>
>>
>> I printed "please wait" and then printed the data given when the process
>> was done (stdoutdata). It wasn't ideal but, it was sufficient for the time.
>> If I were to have gone for the best fix, I would probably have learned the
>> API for subversion to integrate directly into python.
>>
>> However, another BayPIGgie is having the same issue. He is using a command
>> to start a CD burner from the command line and wants to print the output as
>> it is being created from the command line.
>>
>> I can see that to solve this we wouldn't use the communicate() convenience
>> function. A few 'hackish' ways that may solve this, but I'm looking for the
>> common pattern that is used when other pythonista run up against this
>> problem. I also want to ensure that I don't have a 'hack' that causes a
>> deadlock only to discover this much later after I've implemented the pattern
>> a few times.
>>
>> To help keep the conversation more focused, I've created two tiny test
>> programs for a test case:
>> 1) A C command line program that we have no control over changing within
>> python, and
>> 2) A Python program that calls that the c-program (baypiggies_demo):
>>
>> Compile command line so output is same as expected by Python program:
>> gcc -pedantic baypiggies_demo.c -o baypiggies_demo
>>
>>
>> ---- start of c program: File: baypiggies_demo.c ---
>> #include <stdio.h>
>> #include <unistd.h>
>>
>> int
>> main()
>> {
>>     int             i = 0;
>>
>>     printf("Silly output for monitoring...\n");
>>     for (i = 0; i <= 10; i++) {
>>         printf("Counting... %d\n", i);
>>         sleep(1);
>>     }
>> }
>> --- end of c program ---
>>
>>
>>
>> --- start of python program to demonstrate. File baypiggies.py ---
>> import subprocess
>>
>> print "Just waiting...."
>>
>> process = subprocess.Popen(['./baypiggies_demo'],
>>                     stdout=subprocess.PIPE)
>>
>> stdoutdata, stderrdata = process.communicate()
>>
>> print "Well, NOW I get it.. :( "
>> print stdoutdata
>> --- end baypiggies.py --
>>
>>
>>
>> Has anyone else ran into this before? What's the classic pattern used?
>>
>>
>> Thanks in advance,
>>
>>
>> Glen
>> --
>> Whatever you can do or imagine, begin it;
>> boldness has beauty, magic, and power in it.
>>
>> -- Goethe
>>
>>
>>
>
>
> --
> Whatever you can do or imagine, begin it;
> boldness has beauty, magic, and power in it.
>
> -- Goethe
>
> _______________________________________________
> Baypiggies mailing list
> Baypiggies at python.org
> To change your subscription options or unsubscribe:
> http://mail.python.org/mailman/listinfo/baypiggies
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/baypiggies/attachments/20101002/9d164ae1/attachment-0001.html>