[Chicago] capturing output from subprocesses

[Jess Balint]

On 11/17/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
> Jess, I know you've done a lot of work on this already, but I hope I may
> send this last round of questions about the child processes, and then I
> promise to be done.

No problem, keep asking until you understand. :)

> Inside the main while loop you've got:
>
> (rl, wl, xl) = select([outpi, errpi], [], [outpi, errpi], timeout)
>
> try:
>    # capture the output of the child
>    for f in rl:
>      if f == outpi:
>        o = os.read(f, 100)
>        if o: output += o
>        else: end = 1
>      elif f == errpi:
>        e = os.read(f, 100)
>        if e: error += e
>        else: end = 1
> except os.OSError, e:
>    print "OSError: " + str(e)
>    break
>
> On Linux, execution enters the except block when the child process has
> terminated. For this reason, I'm taking the print statement out of my own
> code, since this seems to be the normal way for the parent to know that
> the child has finished, and so it's not technically an "error". right?

Yes, as far as the program goes, there is no problem. There was a read
error because the child is gone, so we know it's done.

> But I couldn't quite figure out just why this error is raised at all. For
> example, it doesn't seem to happen if I use os.open() and os.read() to
> just read from a text file. Under what circumstances will the return value
> from the "select()" call show that the child's file descriptor is "ready
> to be read" and yet raise an error when you try to read it?

This is answered in the man page:

Quote from man select(2): Those listed in readfds will  be  watched 
to  see  if characters  become available for reading (more precisely,
to see if a read will not block - ***in particular, a file descriptor
is also ready on end-of-file***)

The reason we get the OSError is because it's a pty. pty's are wierd
about eof's because they are a terminal instead of a file or pipe
(etc, etc). So the error stream actually reads the eof (and we set
end). That's what "end" was originally used for when both streams were
pipes. Since we moved to using the pty, we now catch the OSError to
end instead.

> Also, your code has a variable 'end' which you set to 1 if you got
> nothing from the child's stdOut or stdErr, but then nothing happens with
> this variable once it's set. Was this going to be some way out of the
> while loop other than through the except block? I noticed that if I
> add a print statement and a line: "if end: break" like this:
> ...
>
>      elif f == errpi:
>        e = os.read(f, 100)
>        if e: error += e
>        else:
>          end = 1
>          print "got nothing from stderr"
>      if end: break
> except os.OSError, e:
>    print "OSError: " + str(e)
>    break
>
> ...
>
> that a normally terminating child causes the "got nothing from stderr"
> message to be printed, and the while loop exits cleanly without entering
> the "except" block. Did you have something like this in mind for 'end'? If
> that is the case, under what circumstances would execution enter the
> except block? I haven't found anything about os.read() raising an OSError.

Hopefully this makes sense from what is said above. If not, let me know.

> PS Why did you choose 100 for the size of your read buffer? Would 1 be a
> bad choice?

If you are thinking that it will let you "squeeze" more out of the
output of the process, it won't. A 1 byte read is terribly inefficient
because it requires the operating system to handle it. At least using
100 byte reads we get a reasonable amount of work done for the system
call. Also, if less than 100 bytes are available, it will still do the
read. You can add some debugging statements to the program to get a
better idea of what is going on with that.

So based on what was said above, this seems to be the better implementation:

# finished flag
end = 0

for f in rl:
	if f == outpi:
		# we might not be able to read the pty
		# so we trap the error and signal end, we
		# don't break because there might be something
		# able to be read from "error" stream
		try:
			o = read(f, 100)
		except OSError, e:
			end = 1
			continue
		output += o
	elif f == errpi:
		# error stream is a pipe so we get a proper EOF
		# and can signal end from that
		e = read(f, 100)
		if e: error += e
		else: end = 1

# exit the 'while 1' when nothing left to be read
if end:
	break

------------------------------------
Also, it's possible that if you have a very large amount of amount in
one of the buffers that the other one will signal 'end' before you
finish reading. So if you want it to be perfect, you should track
'end' for each (stdout and stderr) buffer individually and adjust the
read's and read fdset accordingly.

Jess