[Python-Dev] PEP 324: popen5 - New POSIX process module

[Guido van Rossum]

Peter,

After seeing the discussion that your PEP has sparked (without reading
all of it) I am torn.  I agree that Python's subprocess management
facilities can use a lot of work.  I agree that your code is by and
large an improvement over popen2 and friends.  But I'm not sure that
it is *enough* of an improvement to be accepted into the standard
library.

What I'm looking for in standard library additions is
"category-killers".  IMO good recent examples are optparse and the
logging module.  Both were in active use and had independent
distributions with happy users before they were accepted into the
standard library.

Note that in my definition, a category-killer doesn't have to have all
conceivable features -- it has to strike the right balance between
feature completeness and a set of properties that are usually referred
to by words like elegance, simplicity, ease-of-use, ease-of-learning.
For example, optparse specifically does not support certain styles of
option parsing -- it strives to encourage uniformity in option syntax,
and that rules out certain features.

(I'm not saying that your popen5 has too many features -- I'm just
warning that I'm not asking you to make it a category-killer by adding
tons more features.  I don't want more features, I want the satisfying
feeling that this is the "best" solution, for my very personal
definition of "best".)

I note that your source code at

 http://cvs.lysator.liu.se/viewcvs/viewcvs.cgi/popen5/?cvsroot=python-popen5

has some useful comments that were missing from the PEP, e.g. how to
replace various older APIs with calls to popen5.  It also has some
motivation for the API choices you made that are missing from the PEP
(e.g. why there are options for setting cwd).

So what's missing?  For the PEP, I would say that a lot of explanatory
and motivational text from the source code comments and doc strings
should be added.  (PEPs don't have to be terse formal documents that
describe the feature in as few words as possible; that can be one
section of the PEP, but in general a PEP needs to provide motivation
and insight as well as specification.)

For the code, I think that a Windows version is essential.  It would
be okay if a small amount of code (e.g. making some basic Windows APIs
available, like CreateProcess) had to be coded in C, as long as the
bulk could still be in Python.  I really like having this kind of code
in Python -- it helps understanding the fine details, and it allows
subclassing more easily.

I also wonder if more support for managing a whole flock of
subprocesses might not be a useful addition; this is a common need in
some applications.  And with this, I'd like to see explicit support
(somehow -- maybe through a separate class) for managing "daemon"
processes.  This certainly is a common need!

I would like this to replace all other "high-level" subprocess
management facilities, in particular os.system() and os.popen() and
the popen2 module, on all platforms.  Oh, and also os.spawn*().  But
not os.fork() and os.exec*(), since those are the building blocks (on
Unix, anyway).

--Guido van Rossum (home page: http://www.python.org/~guido/)