[Chicago] capturing output from subprocesses

Noel Thomas Taylor nttaylor at uchicago.edu
Tue Nov 8 18:59:08 CET 2005

Dear ChiPy,

Thank you for your responses. Ian, I looked at the proc_communicate() 
function you pointed me to, and sent it a command to run a short python 
program "ticker", which just prints the word "tick" once a second for 10 

But I couldn't make it capture the output as it was coming out - my only 
choice was to see all of the output at once after the process had 

Did I understand you correctly that proc_communicate() should be able to 
see unbuffered output as the subprocess is running?

I've tried to make this work in my own experiments by using fcntl to set 
the file descriptors on the child's stdout fileobject to non-blocking and 
then using select.select to report when the stdout file object is ready to 
be read from, but I always get a "resource temporarily unavailable" error 
when I try to actually read from stdout.

Martin Maney, who also responded to my question, wrote that such behavior 
is not possible without the use of pseudo-terminals (assuming you cannot 
alter the code of the child process itself to produce unbuffered output). 
In this case my experiments are doomed from the outset. Do you agree?

One more question: capturing the output "live" isn't so important by 
itself. The real issue is that if the child process hangs and needs to be 
aborted, whatever output it produced before it got hung can be preserved. 
Do you guys know if it is possible to get the child process's output out 
of whatever buffer it's in, in the event that the child has to be killed 
by the parent?

with thanks for your help,

Noel Taylor

On Mon, 7 Nov 2005, Ian Bicking wrote:

> Noel Thomas Taylor wrote:
>> I just joined the list and hope to come to the meeting this Thursday. If I
>> can't, however, I wonder if someone could help me with a very tricky
>> problem...
>> A couple of months ago I started trying to find a way to launch a
>> subprocess in python that would allow the parent to capture the stdout
>> and stderr of the child, time the duration of the child, and abort the
>> child if it had failed to produce output after 'n' seconds. Also, the
>> solution should perform consistently across all Unices (Windows doesn't
>> matter) and if the child must be aborted, whatever output it produced up
>> to that point should still be captured.
>> There are several solutions out there, but most do not capture the output
>> of the child in real time. You always have to wait for the child to
>> terminate until you can see any of its output, and consequently if the
>> child must be aborted because it gets stuck, you lose whatever output it
>> generated before you killed it.
> The subprocess module can do this.  It's a little funny, because it
> doesn't allow streaming using proc.communicate(), and I find it
> difficult to do normal piping because of the blocking.
> But if you look at the bottom of this file there's a function
> proc_communicate allows you to communicate using Python file-like objects:
>   http://svn.pythonpaste.org/Paste/trunk/paste/cgiapp.py
> It's basically a slight rewrite of Proc.communicate().  There's already
> some patches filed for this issue, so hopefully subprocess in Python 2.5
> will make this easier.
> -- 
> Ian Bicking  /  ianb at colorstudy.com  /  http://blog.ianbicking.org
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago

More information about the Chicago mailing list