Hi, I will try to answer to your worries. Tell me if I should complete the PEP with these answers. 2013/8/23 Charles-François Natali :

Why does dup2() create inheritable FD, and not dup()?

Ah yes, there were as section explaining it. In a previous version of the PEP (and its implementation), os.dup() and os.dup2() created non-inheritable FD, but inheritable FD for standard steams (fd 0, 1, 2). I did a research on http://code.ohloh.net/ to see how os.dup2() is used in Python projects. 99.4% (169 projects on 170) uses os.dup2() to replace standard streams: file descriptors 0, 1, 2 (stdin, stdout, stderr); sometimes only stdout or stderr. I only found a short demo script using dup2() with arbitrary file descriptor numbers (10 and 11) to keep a copy of stdout and stderr before replacing them (also with dup2). I didn't find use cases of dup() to inherit file descriptors in the Python standard library. It's the opposite: when os.dup() (or the C function dup() is used), the FD must not be inherited. For example, os.listdir(fd) duplicates fd: a child process must not inherit the duplicated file descriptor, it may lead a security vulnerability (ex: parent process running as root, whereas the child process is running as a different used and not allowed to open the directory).

For example, a lot of code uses the guarantee that dup()/open()... returns the lowest numbered file descriptor available, so code like this:

r, w = os.pipe() if os.fork() == 0: # child os.close(r) os.close(1) dup(w)

*will break*

And that's a lot of code (e.g. that's what _posixsubprocess.c uses, but since it's implemented in C it's wouldn't be affected).

Yes, it will break. As I wrote in my previous email, we cannot solve all issues listed in the Rationale section of the PEP without breaking applications (or at least breaking backward compatibility). It is even explicitly said in the "Backward Compatibility" section: http://www.python.org/dev/peps/pep-0446/#backward-compatibility "This PEP break applications relying on inheritance of file descriptors." But I also added a hint to fix applications: "Developers are encouraged to reuse the high-level Python module subprocess which handles the inheritance of file descriptors in a portable way." If you don't want to use subprocess, yes, you will have to add "os.set_inheritable(w)" in the child process. About your example: I'm not sure that it is reliable/portable. I saw daemon libraries closing *all* file descriptors and then expecting new file descriptors to become 0, 1 and 2. Your example is different because w is still open. On Windows, I have seen cases with only fd 0, 1, 2 open, and the next open() call gives the fd 10 or 13... I'm optimistic and I expect that most Python applications and libraries already use the subprocess module. The subprocess module closes all file descriptors (except 0, 1, 2) since Python 3.2. Developers relying on the FD inheritance and using the subprocess with Python 3.2 or later already had to use the pass_fds parameter.

Furthermore, many people use Python for system programming, and this change would be highly surprising.

Yes, it is a voluntary design choice (of the PEP). It is also said explicitly in the "Backward Compatibility" section: "Python does no more conform to POSIX, since file descriptors are now made non-inheritable by default. Python was not designed to conform to POSIX, but was designed to develop portable applications."

So no matter what the final decision on this PEP is, it must be kept in mind.

The purpose of the PEP is to explain correctly the context and the consequences of the changes, so Guido van Rossum can uses the PEP to make its final decision.

...

The programming languages Go, Perl and Ruby make newly created file descriptors non-inheritable by default: since Go 1.0 (2009), Perl 1.0 (1987) and Ruby 2.0 (2013).

OK, but do they expose OS file descriptors?

Yes: - Perl: fileno() function - Ruby: fileno() method of a file object - Go: fd() method of a file object

Last time, I said that to me, the FD inheritance issue is solved on POSIX by the subprocess module which passes close_fds. In my own code, I use subprocess, which is the "official", portable and safe way to create child processes in Python. Someone using fork() + exec() should know what he's doing, and be able to deal with the consequences: I'm not only talking about FD inheritance, but also about async-signal/multi-threaded safety ;-)

The subprocess module has still a (minor?) race condition in the child process. Another C thread can create a new file descriptor after the subprocess module closed all file descriptors and before exec(). I hope that it is very unlikely, but it can happen. It's also explained in the PEP (see "Closing All Open File Descriptors"). I suppose that the race condition explains why Linux still has no closefrom() or nextfd() system calls. IMO the kernel is the best place to decide which FD should be kept, and which must not be inherited (must be closed), in the child process. I like the close-on-exec flag (and HANDLE_FLAG_INHERIT on Windows).

As for Windows, since it doesn't have fork(), it would make sense to make its FD non heritable by default.

As said in the "Inheritance of File Descriptors on Windows" section, Python gets inheritable handles and file descriptors because it does not use the Windows native API. Applications developed for Windows using the native API only create non-inheritable handles, and so don't have all these annoying inheritance issues. Windows does not have a fork() function, but handles and file descriptors can be inherited using CreateProcess() and spawn(), see the table in the "Status of Python 3.3" section.

And then use what you describe here to selectively inherit FDs (i.e. implement keep_fds): """ Since Windows Vista, CreateProcess() supports an extension of the STARTUPINFO struture: the STARTUPINFOEX structure. Using this new structure, it is possible to specify a list of handles to inherit: PROC_THREAD_ATTRIBUTE_HANDLE_LIST. Read Programmatically controlling which handles are inherited by new processes in Win32 (Raymond Chen, Dec 2011) for more information. """

This feature can only be used to inherit *handles*, and it does not need of an intermediate program. If you want to inherit only some file descriptors, you need an intermediate program which will recreates them and then use spawn() (or directly the reserved fields of the STARTUPINFO structure). Using an intermediate program has unexpected consequences, so I prefer to use this option. The feature is also specific to Windows 7, a recent Windows version, whereas there are still millions of Windows XP installations (and I read that Python 3.4 will still supported Windows XP!). We need more feedback from users to know their use cases before chosing the best implementation of pass_handles/pass_fds on Windows. Richard agreed that this point can be deferred. Victor

On Fri, Aug 23, 2013 at 1:30 PM, Charles-François Natali wrote:

...

About your example: I'm not sure that it is reliable/portable. I sa daemon libraries closing *all* file descriptors and then expecting new file descriptors to become 0, 1 and 2. Your example is different because w is still open. On Windows, I have seen cases with only fd 0, 1, 2 open, and the next open() call gives the fd 10 or 13...

Well, my example uses fork(), so obviously doesn't apply to Windows. It's perfectly safe on Unix.

But relying on this in UNIX has also been discouraged ever since the dup2() system call was introduced. (I can't easily find a reference about its history but IIRC it is probably as old as UNIX v7 or otherwise BSD 4.x.)

...

I'm optimistic and I expect that most Python applications and libraries already use the subprocess module. The subprocess module closes all file descriptors (except 0, 1, 2) since Python 3.2. Developers relying on the FD inheritance and using the subprocess with Python 3.2 or later already had to use the pass_fds parameter.

As long as the PEP makes it clear that this breaks backward compatibility, that's fine. IMO the risk of breakage outweights the modicum benefit.

I know this will break code. But it is for the good of mankind. (I will now review the full PEP, finally.) -- --Guido van Rossum (python.org/~guido)

2013/8/26 Guido van Rossum :

I have reviewed the PEP and I think it is good. Thank you so much for pushing this topic and for your very thorough review of all the feedback, related issues and so on. It is an exemplary PEP!

Thanks :-) I updated the PEP: http://hg.python.org/peps/rev/edd8250f6893

I've made a bunch of small edits (mostly to improve grammar slightly, hope you don't mind) and committed these to the repo.

Thanks, I'm not a native english speaker, so not problem for such edit.

https://codereview.appspot.com/13240043/diff/3001/pep-0446.txt#newcode437 pep-0446.txt:437: condition. As C-F Natali pointed out, this is not actually a problem, because after fork() only the main thread survives. Maybe just delete this paragraph?

Ok, I didn't know that only one thread survives to fork(). (I read Charles' email, but I forgot to update the PEP.) I simply deleted the paragraph.

https://codereview.appspot.com/13240043/diff/3001/pep-0446.txt#newcode450 pep-0446.txt:450: parameter is a non-empty list of file descriptors. Well, it could pass closefrom() the max of the given list and manually close the rest. This would be useful if the system max is large but none of the FDs given in the list is. (This would be more complex code but it would address the issue for most programs.)

This was related to the multi-thread issue, which does not exist, so I also removed this paragraph. Using closefrom() to optimize subprocess is unrelated to this PEP. (And yes, the maximum file descriptor can be huge!)

https://codereview.appspot.com/13240043/diff/3001/pep-0446.txt#newcode538 pep-0446.txt:538: descriptors). I would say it should not be changed because the default is still better. :-)

(The PEP does not propose to change the default value.) Under Linux, recent versions of the glibc uses non-inheritable FD for internal files. Slowly, more and more libraries and programs will do the same. This PEP is a step in this direction ;-) Victor