[Python-Dev] PEP 324: popen5 - New POSIX process module

[Martin v. Loewis]

Peter Astrand wrote:
> I don't agree. I have used all of the existing mechanism in lots of apps,
> and it's just a pain. There are lots of functions to choose between, but
> none does what you really want. 

So enhance them, instead of replacing them.

>>>    - Cross-process exceptions: Exceptions happening in the child
>>>      before the new process has started to execute are re-raised in
>>>      the parent.
 >>>
>>This is a bug in popen2, IMO. Fixing it is a good thing, but does not
>>require a new module.
> 
> "Fixing popen2" would mean a break old applications; exceptions will 
> happen, which apps are not prepared of. 

I find that an acceptable incompatibility, and it will likely break
no existing application. Applications usually expect that the program
they start actually exists; it is a good thing that they now can
detect the error that the missing/non-executable application.

Errors should never pass silently.

>>>    - A hook for executing custom code between fork and exec.  This
>>>      can be used for, for example, changing uid.
>>
>>Such a hook could be merged as a keyword argument into the existing
>>API.
> 
> 
> Into which module/method/function? 

For example, popen2.popen2, as argument preexec_fn.


> There is no one flexible enough. The 
> case for redirecting only stderr is just one example; this is simple not 
> possible with the current API. 

Can you elaborate? What is the specific problem, how does your preexec
function look like, and how is it used with popen5. I can then show you
how it could be used with popen2, if that was enhanced appropriately.

>>>    - All combinations of file descriptor redirection is possible.
>>>      For example, the "python-dialog" [2] needs to spawn a process
>>>      and redirect stderr, but not stdout.  This is not possible with
>>>      current functions, without using temporary files.
>>
>>Sounds like a new function on the popen2 module.
> 
> 
> To support all combinations, 12 different functions are necessary. Who 
> will remember what popen2.popen11() means? 

Why is that? Just add a single function, with arguments
stdin/stdout/stderr. No need for 12 functions. Then explain the existing
functions in terms of your new function (if possible).

>>>    - Support for connecting several subprocesses (shell "pipe").
>>
>>Isn't this available already, as the shell supports pipe creation,
>>anyway?
> 
> 
> With popen5, you can do it *without* using the shell. 

Why is that a good thing?

> There's already a bug about this; bug 788035. This is what one of the 
> comment says:
> 
> "But this whole popen{,2,3,4} section of posixmodule.c is so fiendishly
> complicated with all the platform special cases that I'm loath to touch
> it..."
> 
> I haven't checked if this is really true, though. 

You really should work with the existing code base. Ignoring it is a
guarantee that your PEP will be rejected. (Studying it, and then
providing educated comments about it, might get you through)

I think this is the core problem of your approach: You throw away all
past history, and imply that you can do better than all prior
contributors could. Honestly, this is doubtful. The current code
is so complicated because implementing pipes is complicated.

> Well, I don't see how this could be done easily: The current API is not 
> flexible enough, and some things (like cross-process exceptions) breaks 
> compatibility. 

I never said it would be easy. However, introducing a new popen module
is a major change, and there must be strong indications that the current
API cannot be enhanced before throwing it away.

There should be one-- and preferably only one --obvious way to do it.

As for breaking compatibility: This is what the PEP should study in
detail. It is sometimes acceptable to break compatibility, if
applications are likely to be improved by the change. *Any* change
can, in principle, break compatibility. Suppose I had an application
that did

from popen5 import open

This application might break if your proposed change is implemented,
as a new module is added. So you can't claim "I will break no programs".

> Writing a good popen module is hard. Providing cross-platform support (for 
> Windows, for example) is even harder. Trying to retrofit a good popen 
> implementation into an old API without breaking compatibility seems 
> impossible to me. I'm not prepared to try. 

So I continue to be -1 with your PEP.

Regards,
Martin