[Chicago] capturing output from subprocesses

[Noel Thomas Taylor]

Hi Jess and the ChiPy users group,

Jess, I know you've done a lot of work on this already, but I hope I may 
send this last round of questions about the child processes, and then I 
promise to be done.

Inside the main while loop you've got:

(rl, wl, xl) = select([outpi, errpi], [], [outpi, errpi], timeout)

try:
   # capture the output of the child
   for f in rl:
     if f == outpi:
       o = os.read(f, 100)
       if o: output += o
       else: end = 1
     elif f == errpi:
       e = os.read(f, 100)
       if e: error += e
       else: end = 1
except os.OSError, e:
   print "OSError: " + str(e)
   break

On Linux, execution enters the except block when the child process has 
terminated. For this reason, I'm taking the print statement out of my own 
code, since this seems to be the normal way for the parent to know that 
the child has finished, and so it's not technically an "error". right?

But I couldn't quite figure out just why this error is raised at all. For 
example, it doesn't seem to happen if I use os.open() and os.read() to 
just read from a text file. Under what circumstances will the return value 
from the "select()" call show that the child's file descriptor is "ready 
to be read" and yet raise an error when you try to read it?

Also, your code has a variable 'end' which you set to 1 if you got 
nothing from the child's stdOut or stdErr, but then nothing happens with 
this variable once it's set. Was this going to be some way out of the 
while loop other than through the except block? I noticed that if I 
add a print statement and a line: "if end: break" like this:
...

     elif f == errpi:
       e = os.read(f, 100)
       if e: error += e
       else:
         end = 1
         print "got nothing from stderr"
     if end: break
except os.OSError, e:
   print "OSError: " + str(e)
   break

...

that a normally terminating child causes the "got nothing from stderr" 
message to be printed, and the while loop exits cleanly without entering 
the "except" block. Did you have something like this in mind for 'end'? If 
that is the case, under what circumstances would execution enter the 
except block? I haven't found anything about os.read() raising an OSError.

with thanks,

Noel Taylor

PS Why did you choose 100 for the size of your read buffer? Would 1 be a 
bad choice?


On Wed, 16 Nov 2005, Jess Balint wrote:

> On 11/16/05, Noel Thomas Taylor <nttaylor at uchicago.edu> wrote:
>> I've tested it too and it works fine all around. This is the greatest.
>> I'll be giving you big props in the final code.
>
> Glad it's working.
>
>> I had a few closing questions. At the beginning of the parent fork, you
>> close the file descriptor for writing to stdErr, since you won't need it;
>> meanwhile in the child fork you close the file descriptor for reading from
>> stdErr, since you won't need it either. This all makes sense, but can't
>> you also close the corresponding file descriptors for writing to and
>> reading from stdOut, just as you did for stdErr? I added these "closes" to
>> the code and noticed no ill effects, but thought I'd ask in case there was
>> some reason I wasn't seeing.
>
> If you don't close the other side of pipes, they won't return EOF (see
> pipe(2) man page). The pty is a different story. You usually only open
> the slave in the child, but since the openpty() does the whole thing,
> we just take it as it is.
>
>> Were you using your own function 'mygetpty()' on platforms / versions of
>> Python where os.getpty() is not supported, like the Python 2.2 you had on
>> your Solaris machine? I'm not sure about just when Python introduced this
>> function, or if it is supported under Solaris. I don't have a Solaris
>> machine here to test, and Python's own documentation pages are not very
>> specific about which platforms support this function.
>
> I just moved the mygetpty() code out to it's own function so I could
> easily replace it with openpty() to see if it worked. As I said Python
> 2.2 (from Solaris companion CD) doesn't have openpty(), but
> ActiveState 2.4 does. So you can make whatever changes you need in
> that direction.
>
>> Finally, I pretty much understand what's happening, but why do you have to
>> make the calls to dup2? I've seen this in several process-spawning codes
>> but have never figured out just why it's necessary.
>
> dup2() is really the key to all this. You should read the man page for
> details, but in a sentence it just redefines file descriptors. This
> allows setting the new stdout and stderr for the child so it can be
> captured.
>
> Jess
> _______________________________________________
> Chicago mailing list
> Chicago at python.org
> http://mail.python.org/mailman/listinfo/chicago
>