[Python-Dev] PEP 324: popen5 - New POSIX process module

[Peter Astrand]

On Sun, 4 Jan 2004, Martin v. Loewis wrote:

> > "Fixing popen2" would mean a break old applications; exceptions will 
> > happen, which apps are not prepared of. 
> 
> I find that an acceptable incompatibility, and it will likely break
> no existing application. 

Not true. There are lots or apps out there that uses fallback commands:  
tries to execute one, and if it doesn't exist, tries another one. (One
example is jakarta-gump, see
http://cvs.apache.org/viewcvs.cgi/jakarta-gump/python/gump/utils/launcher.py?rev=1.6&view=auto)  
With the current API, you do this by checking if the return code is 127.
No-one is prepared for an exception.

The return code stuff is also very problematic, and is another reason why 
make a new module and not "enhance" the old ones. With the current API 
(which calls /bin/sh most of the time), some returncodes are overloaded by 
the shell. The shell uses these return codes:

126: the command was found but is not executable
127: the command was not found
128+n: the command was terminated by signal n

This means that it is currently impossible to use these return codes for 
programs launched via the current API, since you cannot tell the 
difference between a 127 generated by a successful call to your command, 
and a 127 generated by the shell. 

I don't see how this can be solved by "enhancing" the current functions, 
without breaking old applications. 


>Applications usually expect that the program
> they start actually exists; it is a good thing that they now can
> detect the error that the missing/non-executable application.

There are lots of other errors as well, not just missing/non-executable 
programs. 


> > There is no one flexible enough. The 
> > case for redirecting only stderr is just one example; this is simple not 
> > possible with the current API. 
> 
> Can you elaborate? What is the specific problem, how does your preexec
> function look like, and how is it used with popen5. I can then show you
> how it could be used with popen2, if that was enhanced appropriately.

Yes, the preexec function feature could possiby be added popen2. This is 
not the problem. 


> >>Sounds like a new function on the popen2 module.
> > 
> > 
> > To support all combinations, 12 different functions are necessary. Who 
> > will remember what popen2.popen11() means? 
> 
> Why is that? Just add a single function, with arguments
> stdin/stdout/stderr. No need for 12 functions. Then explain the existing
> functions in terms of your new function (if possible).

Just like popen5.Popen? Yes, that could be done. We would still have the 
problem with returncode incompatibilites, exceptions and such. 


> > With popen5, you can do it *without* using the shell. 
> 
> Why is that a good thing?

1) Performance. No need for parsing .bashrc on every call...

2) Security. You can do pipes without having to deal with all the quoting
   issues.

3) Getting rid of the shells overloading of return codes

It's also much more elegant, IMHO. 


> I think this is the core problem of your approach: You throw away all
> past history, and imply that you can do better than all prior
> contributors could. Honestly, this is doubtful. 

In a discussion like this, I think it's important to separate the new API 
from the new implementation:

1) The new API. If you look at the popen5 implementation and PEP, it's 
obvious that I haven't throwed away the history. I have tried to take all 
the good parts from the various existing functions. The documentation 
contains 140 lines describing how to migrate from the earlier functions. 

Much of the current API has really never been designed. The API for the
functions os.popen, os.system, os.popen2 comes from the old POSIX
functions. These were never intended to be flexible, cross-platform on 
anything like that. So, it's not hard to do better than these. 


2) The new implementation. 

When I wrote popen5, I took some good ideas out of popen2. The rest of the 
code is written from scratch. 


>The current code
> is so complicated because implementing pipes is complicated.

Let's keep the POSIX stuff separated from the Windows stuff. popen2.py 
does not depend on posixmodule.c on POSIX systems, and popen2.py is not 
complicated at all. 

The popen* stuff for Windows (and OS2 etc) in posixmodule.c is complicated 
because:

1) It's written in low-level C

2) It contains lots of old DOS stuff

3) It tries to launch the program through the shell (which is always a 
pain). 


> > Well, I don't see how this could be done easily: The current API is not 
> > flexible enough, and some things (like cross-process exceptions) breaks 
> > compatibility. 
> 
> I never said it would be easy. However, introducing a new popen module
> is a major change, and there must be strong indications that the current
> API cannot be enhanced before throwing it away.

I wouldn't say that introducing a new module is a "major change". 

Of course, we don't want to end up writing "popen6" in two years, because
we've realized that "popen5" is too limited. That's why we should try to
get it exactly right this time. I think it would be more useful it we put
our energy into trying to accomplish that. 


> As for breaking compatibility: This is what the PEP should study in
> detail. It is sometimes acceptable to break compatibility, if
> applications are likely to be improved by the change. *Any* change
> can, in principle, break compatibility. Suppose I had an application
> that did
> 
> from popen5 import open
> 
> This application might break if your proposed change is implemented,
> as a new module is added. So you can't claim "I will break no programs".

Isn't this quite a silly example? 


-- 
/Peter Åstrand <astrand at lysator.liu.se>