[Chicago] capturing output from subprocesses
Noel Thomas Taylor
nttaylor at uchicago.edu
Tue Nov 8 18:59:08 CET 2005
Thank you for your responses. Ian, I looked at the proc_communicate()
function you pointed me to, and sent it a command to run a short python
program "ticker", which just prints the word "tick" once a second for 10
But I couldn't make it capture the output as it was coming out - my only
choice was to see all of the output at once after the process had
Did I understand you correctly that proc_communicate() should be able to
see unbuffered output as the subprocess is running?
I've tried to make this work in my own experiments by using fcntl to set
the file descriptors on the child's stdout fileobject to non-blocking and
then using select.select to report when the stdout file object is ready to
be read from, but I always get a "resource temporarily unavailable" error
when I try to actually read from stdout.
Martin Maney, who also responded to my question, wrote that such behavior
is not possible without the use of pseudo-terminals (assuming you cannot
alter the code of the child process itself to produce unbuffered output).
In this case my experiments are doomed from the outset. Do you agree?
One more question: capturing the output "live" isn't so important by
itself. The real issue is that if the child process hangs and needs to be
aborted, whatever output it produced before it got hung can be preserved.
Do you guys know if it is possible to get the child process's output out
of whatever buffer it's in, in the event that the child has to be killed
by the parent?
with thanks for your help,
On Mon, 7 Nov 2005, Ian Bicking wrote:
> Noel Thomas Taylor wrote:
>> I just joined the list and hope to come to the meeting this Thursday. If I
>> can't, however, I wonder if someone could help me with a very tricky
>> A couple of months ago I started trying to find a way to launch a
>> subprocess in python that would allow the parent to capture the stdout
>> and stderr of the child, time the duration of the child, and abort the
>> child if it had failed to produce output after 'n' seconds. Also, the
>> solution should perform consistently across all Unices (Windows doesn't
>> matter) and if the child must be aborted, whatever output it produced up
>> to that point should still be captured.
>> There are several solutions out there, but most do not capture the output
>> of the child in real time. You always have to wait for the child to
>> terminate until you can see any of its output, and consequently if the
>> child must be aborted because it gets stuck, you lose whatever output it
>> generated before you killed it.
> The subprocess module can do this. It's a little funny, because it
> doesn't allow streaming using proc.communicate(), and I find it
> difficult to do normal piping because of the blocking.
> But if you look at the bottom of this file there's a function
> proc_communicate allows you to communicate using Python file-like objects:
> It's basically a slight rewrite of Proc.communicate(). There's already
> some patches filed for this issue, so hopefully subprocess in Python 2.5
> will make this easier.
> Ian Bicking / ianb at colorstudy.com / http://blog.ianbicking.org
> Chicago mailing list
> Chicago at python.org
More information about the Chicago