what happens to Popen()'s parent-side file descriptors?

[Chris Torek]

In article <b166181a-2aba-4c3a-948d-674755459126 at c10g2000yqh.googlegroups.com>
Roger Davis  <rbd at hawaii.edu> wrote:
>My understanding is that this functionality is best coded via
>subprocess.Popen().

"Best" is always a big question mark. :-)

>I need to read output from these spawned children
>via a pipe from their stdout, hence something like
>
>     p= subprocess.Popen(args, stdout=subprocess.PIPE)
>
>This means that somewhere a pipe file descriptor is opened on the
>parent side to read from the child's stdout. When, if ever, is that
>descriptor closed?

(I am going to tell this tale in a slightly different order than
your question asked, as I think it works out better that way.)

subprocess.Popen() creates the instance variable and any pipes
needed, forks (on a Unix system) the target process, but has not
yet done any I/O with it (except to read a success/fail indicator
for whether the exec worked and/or any exception that occurred
before then, e.g., during the preexec_fn).  It then makes the stdin,
stdout, and/or stderr attributes (p.stdout, for the example above)
using os.fdopen().  Streams not requested in the call are set to
None (so p.stderr, for instance, will be None in this case).

At this point, then, the underlying open pipe is still around.
But your next step is (normally) to use p.communicate(); this is
where most of the magic happens.  The Unix implementation loops,
using select() to read and write from/to whichever pipe(s) are open
to the child process, until *all* data are sent and received.  As
each data stream is finished, it is closed (in this case, via
self.stdout.close()).  Lastly, p.communicate() invokes p.wait() (via
self.wait()), to wait for the child process to exit.

By the time p.communicate() returns, the pipe is closed and the
command has finished.  The entire output text, however large it
is, is returned as the first element of the return-value 2-tuple
(remember that p.communicate() returns both the stdout and the
stderr -- stderr will be the empty string in this case, as stderr
was not redirected in the subprocess.Popen() call).

>Per-process FDs are limited and I am looping
>infinitely so I need to be very careful about not running out of them.
>Are there any other FDs related to this operation that also need to be
>closed?

Only if you (or code you call) have opened them and not set
FD_CLOEXEC.  In this case, you can set close_fds = True in your
call to subprocess.Popen().  That will make the child of fork()
loop over higher-number fd's, calling os.close() on each one.

>Testing with the interpreter (2.6, MacOSX) it appears that p.stdout is
>being closed somehow by someone other than me:
>
>import subprocess
>args= ["echo", "This is a mystery!"]
>i= 0
>while True:
>    p= subprocess.Popen(args, stdout=subprocess.PIPE)
>    for line in p.stdout:
>        print "[%5d] %s" % (i, line.strip())
>    i+= 1
>
>The above code closes nothing but appears to run indefinitely without
>running the parent out of FDs. WTF is going on here?

The above reads from p.stdout -- the os.fdopen() result on the
underlying pipe -- directly.  In the general case (multiple input
and output pipes), this is not safe as you can deadlock with
constipated pipes (hence the existence of p.communicate()).  In
this specific case, there is just one pipe so the deadlock issue
goes away.  Instead, the file descriptor remains open while the
inner loop runs (i.e., while "line in p.stdout" is able to fetch
lines via the file's iterator).  When the loop stops the pipe is
still open in the parent, but the child has finished and is now
exiting (or has exited or will exit soon).  You then reach the
"i+=1" line and resume the loop, calling subprocess.Popen()
anew.

Now we get to the even deeper magic. :-)

What happens to the *old* value in p?  Answer: because p is
reassigned, the (C implementation, interpreted Python bytecode
runtime) reference count drops [%].  Since p was the only live
reference, the count drops from 1 to 0.  This makes the old instance
variable go away, invoking old_p.__del__() as it were.  The deletion
handler cleans up a few things itself, including a a call to
os.waitpid() if needed, and then simply lets the reference to
old_p.stdout go away.  That in turn decrements old_p.stdout's
reference count.  Since that, too, reaches zero, its __del__ is
run ... and *that* closes the underlying file descriptor.

[% This is all simplified -- the Python documentation mentions that
reference counting for local variables is somewhat tricked-out by
the compiler to avoid unnecessary increments and decrements.  The
principles apply, though.]

Running the above code fragment in a different implementation, in
which garbage collection is deferred, would *not* close the file
descriptor, and the system would potentially run out (depending on
when a gc occurred, and/or whether the system would attempt gc on
running out of file descriptors, in the hope that the gc would free
some up).

The subprocess module does go through a bunch of extra work to make
sure that any as-yet-uncollected fork()ed processes are eventually
waitpid()-ed for.

>Can anyone explain the treatment of the pipe FDs opened in the parent
>by Popen() to me or point me to some documentation?

The best documentation seems generally to be the source.  Fortunately
subprocess.py is written in Python.  (Inspecting C modules is less
straightforward. :-) )

>Also, does Popen.returncode contain only the child's exit code or is
>does it also contain signal info like the return of os.wait()?
>Documentation on this is also unclear to me.

"A negative value -N indicates that the child was terminated by
signal N (Unix only)."  Again, the Python source is handy:

        def _handle_exitstatus(self, sts):
            if os.WIFSIGNALED(sts):
                self.returncode = -os.WTERMSIG(sts)
            elif os.WIFEXITED(sts):
                self.returncode = os.WEXITSTATUS(sts)
            else:
                # Should never happen
                raise RuntimeError("Unknown child exit status!")

The only things left out are the core-dump flag, and stopped/suspended.
The latter should never occur as os.waitpid() is called with only
os.WNOHANG, not os.WUNTRACED (of course a process being traced,
stopping at a breakpoint, would mess this up, but subprocess.Popen
is not a debugger :-) ).

It might be nice to capture os.WCOREDUMPED(sts), though.

Also, while I was writing this, I discovered that appears to be a
buglet in _cleanup(), with regard to "abandoned" Unix processes that
terminate due to a signal.  Note that _handle_exitstatus() will
set self.returncode to (e.g.) -1 if the child exits due to SIGHUP.
The _cleanup() function, however, does this in part:

        if inst.poll(_deadstate=sys.maxint) >= 0:
            try:
                _active.remove(inst)

The Unix-specific poll() routine, however, reads:

            if self.returncode is None:
                try:
                    pid, sts = os.waitpid(self.pid, os.WNOHANG)
                    if pid == self.pid:
                        self._handle_exitstatus(sts)
                except os.error:
                    if _deadstate is not None:
                        self.returncode = _deadstate
            return self.returncode

Hence if pid 12345 is abandoned (and thus on _active), and we
os.waitpid(12345, os.WNOHANG) and get a status that has a termination
signal, we set self.returncode to -N, and return that.  Hence
inst.poll returns (e.g.) -1 and we never attempt to remove it from
_active.  Now that its returncode is not None, though, every later
poll() will continue to return -1.  It seems it would be better to
have _cleanup() read:

        if inst.poll(_deadstate=sys.maxint) is not None:

(Note, this is python 2.5, which is what I have installed on my
Mac laptop, where I am writing this at the moment).
-- 
In-Real-Life: Chris Torek, Wind River Systems
Salt Lake City, UT, USA (40°39.22'N, 111°50.29'W)  +1 801 277 2603
email: gmail (figure it out)      http://web.torek.net/torek/index.html