[Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

Victor Stinner victor.stinner at gmail.com
Fri Jul 26 14:08:35 CEST 2013


2013/7/24 Guido van Rossum <guido at python.org>:
> But I'm also ready to propose that all this is such a mess that we
> *should* change the default fd/handle inheritance to False, *across
> platforms*, and damn the torpedoes -- i.e. accept breaking all
> existing 3rd party UNIX code for subprocess creation that bypasses the
> subprocess module, as well as breaking uses of os.spawn*() on both
> platforms that depend on FD inheritance beyond stdin/stdout/stderr).
>
> With the new, sane default, all we need instead of PEP 446 is a way to
> make an FD inheritable after it's been created, which can be a single
> os.make_inheritable(fd) call that you must apply to the fileno() of
> the stream or socket object you want inherited (or directly to a FD
> you created otherwise, e.g. with os.pipe()). On Windows, this should
> probably only work with os.spawn*(), since otherwise you need *handle*
> inheritance, not *FD* inheritance, and that's a non-portable concept
> anyway.

After having written 2 PEP on the topic, I slowly agree that make all
file descriptors non-inheritable is the best *compromise*. It solves
most, or all, issues.

The main drawback is the additionnal syscalls: on some platforms, 2
additional syscalls are need to make a file descriptor non-inheritable
for each creation of file descriptor. According to my benchmark on the
implementation of the PEP 433: the overhead of making a file
descriptor non-inheritable is between 1% and 3% (7.8 µs => 7.9 or 8.0
µs) on Linux 3.6.
http://www.python.org/dev/peps/pep-0433/#performances

Having to make a file descriptor inheritable after creating it
non-inheritable is also not optimal. Making it first non-inheritable
requires 0, 1 or 2 extra syscalls, and making it inheritable again
require also 1 or 2 syscalls. So f=open(...);
os.make_inheritable(f.fileno()) can take up to 5 syscalls (1 open + 4
fnctl), whereas it can be done in only 1 syscall (1 open). One of the
motivation of the PEP 433 an 446 is to reduce the number of syscalls,
even if the use case was to make sockets *non-inheritable*.

If we consider that the most common case is to use non-inheritable
file descriptors, having to call os.make_inheritable() may be
acceptable. Windows and recent operating syscalls support creating
file descriptor directly non-inheritable in a single syscalls. ioctl()
can be also be used instead of fcntl() to use 1 syscall instead of 2.

> We can fix multiprocessing any anything else in the stdlib that this
> breaks, I presume.

The CGI code rely on inheritance of file descriptors 0, 1 and 2 which
are pipes. The file descriptors 0, 1 and 2 are replaced with the pipes
using os.dup2().

Victor


More information about the Python-Dev mailing list