[Python-Dev] PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets

Thu Jul 4 13:03:06 CEST 2013

HTML version:
http://www.python.org/dev/peps/pep-0446/

PEP: 446
Title: Add new parameters to configure the inherance of files and for
non-blocking sockets
Version: $Revision$
Last-Modified: $Date$
Author: Victor Stinner <victor.stinner at gmail.com>
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 3-July-2013
Python-Version: 3.4

Abstract
========

This PEP proposes new portable parameters and functions to configure the
inherance of file descriptors and the non-blocking flag of sockets.

Rationale
=========

Inherance of file descriptors
-----------------------------

The inherance of file descriptors in child processes can be configured
on each file descriptor using a *close-on-exec* flag. By default, the
close-on-exec flag is not set.

On Windows, file descriptors are not inherited if the
``bInheritHandles`` parameter of the ``CreateProcess()`` function is
``FALSE``, even if the close-on-exec flag is not set.

On UNIX, file descriptors with the close-and-exec flag set are closed at
the execution of a new program (ex: when calling ``execv()``). The flag
has no effect on ``fork()``, all file descriptors are inherited by the
child process.

Issues of the inherance of file descriptors
-------------------------------------------

Inherance of file descriptors causes issues. For example, closing a file
descriptor in the parent process does not release the resource (file,
socket, ...), because the file descriptor is still open in the child
process.

Leaking file descriptors is also a major security vulnerability. An
untrusted child process can read sensitive data like passwords and take
control of the parent process though leaked file descriptors. It is for
example a known vulnerability to escape from a chroot.

Non-blocking sockets
--------------------

To handle multiple network clients in a single thread, a multiplexing
function like ``select()`` can be used. For best performances, sockets
must be configured as non-blocking. Operations like ``send()`` and
``recv()`` return an ``EAGAIN`` or ``EWOULDBLOCK`` error if the
operation would block.

By default, newly created sockets are blocking. Setting the non-blocking
mode requires additional system calls.

Setting flags at the creation of the file descriptor
----------------------------------------------------

Windows and recent versions of other operating systems like Linux
support setting the close-on-exec flag directly at the creation of file
descriptors, and close-on-exec and blocking flags at the creation of
sockets.

Setting these flags at the creation is atomic and avoids additional
system calls.

Proposal
========

New cloexec And blocking Parameters
-----------------------------------

Add a new optional *cloexec* on functions creating file descriptors:

* ``io.FileIO``
* ``io.open()``
* ``open()``
* ``os.dup()``
* ``os.dup2()``
* ``os.fdopen()``
* ``os.open()``
* ``os.openpty()``
* ``os.pipe()``
* ``select.devpoll()``
* ``select.epoll()``
* ``select.kqueue()``

Add new optional *cloexec* and *blocking* parameters to functions
creating sockets:

* ``asyncore.dispatcher.create_socket()``
* ``socket.socket()``
* ``socket.socket.accept()``
* ``socket.socket.dup()``
* ``socket.socket.fromfd``
* ``socket.socketpair()``

The default value of *cloexec* is ``False`` and the default value of
*blocking* is ``True``.

The atomicity is not guaranteed. If the platform does not support
setting close-on-exec and blocking flags at the creation of the file
descriptor or socket, the flags are set using additional system calls.

New Functions
-------------

Add new functions the get and set the close-on-exec flag of a file
descriptor:

* ``os.get_cloexec(fd:int) -> bool``
* ``os.set_cloexec(fd:int, cloexec: bool)``

Other Changes
-------------

The ``subprocess.Popen`` class must clear the close-on-exec flag of file
descriptors of the ``pass_fds`` parameter.

The close-on-exec flag must also be set on private file descriptors and
sockets in the Python standard library. For example, on UNIX,
os.urandom() opens ``/dev/urandom`` to read some random bytes and the
file descriptor is closed at function exit. The file descriptor is not
expected to be inherited by child processes.

Rejected Alternatives
=====================

PEP 433
-------

The PEP 433 entitled "Easier suppression of file descriptor inheritance"
is a previous attempt proposing various other alternatives, but no
consensus could be reached.

This PEP has a well defined behaviour (the default value of the new
*cloexec* parameter is not configurable), is more conservative (no
backward compatibility issue), and is much simpler.

Add blocking parameter for file descriptors and Windows overlapped I/O
----------------------------------------------------------------------

Windows supports non-blocking operations on files using an extension of
the Windows API called "Overlapped I/O". Using this extension requires
to modify the Python standard library and applications to pass a
``OVERLAPPED`` structure and an event loop to wait for the completion of
operations.

This PEP only tries to expose portable flags on file descriptors and
sockets. Supporting overlapped I/O requires an abstraction providing a
high-level and portable API for asynchronous operations on files and
sockets. Overlapped I/O are out of the scope of this PEP.

UNIX supports non-blocking files, moreover recent versions of operating
systems support setting the non-blocking flag at the creation of a file
descriptor. It would be possible to add a new optional *blocking*
parameter to Python functions creating file descriptors. On Windows,
creating a file descriptor with ``blocking=False``  would raise a
``NotImplementedError``. This behaviour is not acceptable for the ``os``
module which is designed as a thin wrapper on the C functions of the
operating system. If a platform does not support a function, the
function should not be available on the platform. For example,
the ``os.fork()`` function is not available on Windows.

For all these reasons, this alternative was rejected. The PEP 3156
proposes an abstraction for asynchronous I/O supporting non-blocking
files on Windows.

Links
=====

Python issues:

* `#10115: Support accept4() for atomic setting of flags at socket
  creation <http://bugs.python.org/issue10115>`_
* `#12105: open() does not able to set flags, such as O_CLOEXEC
  <http://bugs.python.org/issue12105>`_
* `#12107: TCP listening sockets created without FD_CLOEXEC flag
  <http://bugs.python.org/issue12107>`_
* `#16850: Add "e" mode to open(): close-and-exec
  (O_CLOEXEC) / O_NOINHERIT <http://bugs.python.org/issue16850>`_
* `#16860: Use O_CLOEXEC in the tempfile module
  <http://bugs.python.org/issue16860>`_
* `#16946: subprocess: _close_open_fd_range_safe() does not set
  close-on-exec flag on Linux < 2.6.23 if O_CLOEXEC is defined
  <http://bugs.python.org/issue16946>`_
* `#17070: Use the new cloexec to improve security and avoid bugs
  <http://bugs.python.org/issue17070>`_