PEP 446: Add new parameters to configure the inherance of files and for non-blocking sockets
HTML version: http://www.python.org/dev/peps/pep-0446/ PEP: 446 Title: Add new parameters to configure the inherance of files and for non-blocking sockets Version: $Revision$ Last-Modified: $Date$ Author: Victor Stinner <victor.stinner@gmail.com> Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 3-July-2013 Python-Version: 3.4 Abstract ======== This PEP proposes new portable parameters and functions to configure the inherance of file descriptors and the non-blocking flag of sockets. Rationale ========= Inherance of file descriptors ----------------------------- The inherance of file descriptors in child processes can be configured on each file descriptor using a *close-on-exec* flag. By default, the close-on-exec flag is not set. On Windows, file descriptors are not inherited if the ``bInheritHandles`` parameter of the ``CreateProcess()`` function is ``FALSE``, even if the close-on-exec flag is not set. On UNIX, file descriptors with the close-and-exec flag set are closed at the execution of a new program (ex: when calling ``execv()``). The flag has no effect on ``fork()``, all file descriptors are inherited by the child process. Issues of the inherance of file descriptors ------------------------------------------- Inherance of file descriptors causes issues. For example, closing a file descriptor in the parent process does not release the resource (file, socket, ...), because the file descriptor is still open in the child process. Leaking file descriptors is also a major security vulnerability. An untrusted child process can read sensitive data like passwords and take control of the parent process though leaked file descriptors. It is for example a known vulnerability to escape from a chroot. Non-blocking sockets -------------------- To handle multiple network clients in a single thread, a multiplexing function like ``select()`` can be used. For best performances, sockets must be configured as non-blocking. Operations like ``send()`` and ``recv()`` return an ``EAGAIN`` or ``EWOULDBLOCK`` error if the operation would block. By default, newly created sockets are blocking. Setting the non-blocking mode requires additional system calls. Setting flags at the creation of the file descriptor ---------------------------------------------------- Windows and recent versions of other operating systems like Linux support setting the close-on-exec flag directly at the creation of file descriptors, and close-on-exec and blocking flags at the creation of sockets. Setting these flags at the creation is atomic and avoids additional system calls. Proposal ======== New cloexec And blocking Parameters ----------------------------------- Add a new optional *cloexec* on functions creating file descriptors: * ``io.FileIO`` * ``io.open()`` * ``open()`` * ``os.dup()`` * ``os.dup2()`` * ``os.fdopen()`` * ``os.open()`` * ``os.openpty()`` * ``os.pipe()`` * ``select.devpoll()`` * ``select.epoll()`` * ``select.kqueue()`` Add new optional *cloexec* and *blocking* parameters to functions creating sockets: * ``asyncore.dispatcher.create_socket()`` * ``socket.socket()`` * ``socket.socket.accept()`` * ``socket.socket.dup()`` * ``socket.socket.fromfd`` * ``socket.socketpair()`` The default value of *cloexec* is ``False`` and the default value of *blocking* is ``True``. The atomicity is not guaranteed. If the platform does not support setting close-on-exec and blocking flags at the creation of the file descriptor or socket, the flags are set using additional system calls. New Functions ------------- Add new functions the get and set the close-on-exec flag of a file descriptor: * ``os.get_cloexec(fd:int) -> bool`` * ``os.set_cloexec(fd:int, cloexec: bool)`` Other Changes ------------- The ``subprocess.Popen`` class must clear the close-on-exec flag of file descriptors of the ``pass_fds`` parameter. The close-on-exec flag must also be set on private file descriptors and sockets in the Python standard library. For example, on UNIX, os.urandom() opens ``/dev/urandom`` to read some random bytes and the file descriptor is closed at function exit. The file descriptor is not expected to be inherited by child processes. Rejected Alternatives ===================== PEP 433 ------- The PEP 433 entitled "Easier suppression of file descriptor inheritance" is a previous attempt proposing various other alternatives, but no consensus could be reached. This PEP has a well defined behaviour (the default value of the new *cloexec* parameter is not configurable), is more conservative (no backward compatibility issue), and is much simpler. Add blocking parameter for file descriptors and Windows overlapped I/O ---------------------------------------------------------------------- Windows supports non-blocking operations on files using an extension of the Windows API called "Overlapped I/O". Using this extension requires to modify the Python standard library and applications to pass a ``OVERLAPPED`` structure and an event loop to wait for the completion of operations. This PEP only tries to expose portable flags on file descriptors and sockets. Supporting overlapped I/O requires an abstraction providing a high-level and portable API for asynchronous operations on files and sockets. Overlapped I/O are out of the scope of this PEP. UNIX supports non-blocking files, moreover recent versions of operating systems support setting the non-blocking flag at the creation of a file descriptor. It would be possible to add a new optional *blocking* parameter to Python functions creating file descriptors. On Windows, creating a file descriptor with ``blocking=False`` would raise a ``NotImplementedError``. This behaviour is not acceptable for the ``os`` module which is designed as a thin wrapper on the C functions of the operating system. If a platform does not support a function, the function should not be available on the platform. For example, the ``os.fork()`` function is not available on Windows. For all these reasons, this alternative was rejected. The PEP 3156 proposes an abstraction for asynchronous I/O supporting non-blocking files on Windows. Links ===== Python issues: * `#10115: Support accept4() for atomic setting of flags at socket creation <http://bugs.python.org/issue10115>`_ * `#12105: open() does not able to set flags, such as O_CLOEXEC <http://bugs.python.org/issue12105>`_ * `#12107: TCP listening sockets created without FD_CLOEXEC flag <http://bugs.python.org/issue12107>`_ * `#16850: Add "e" mode to open(): close-and-exec (O_CLOEXEC) / O_NOINHERIT <http://bugs.python.org/issue16850>`_ * `#16860: Use O_CLOEXEC in the tempfile module <http://bugs.python.org/issue16860>`_ * `#16946: subprocess: _close_open_fd_range_safe() does not set close-on-exec flag on Linux < 2.6.23 if O_CLOEXEC is defined <http://bugs.python.org/issue16946>`_ * `#17070: Use the new cloexec to improve security and avoid bugs <http://bugs.python.org/issue17070>`_ Other links: * `Secure File Descriptor Handling <http://udrepper.livejournal.com/20407.html>`_ (Ulrich Drepper, 2008) Copyright ========= This document has been placed into the public domain.
PEP: 446 Title: Add new parameters to configure the inherance of files and for non-blocking sockets (...) Rejected Alternatives =====================
PEP 433 -------
The PEP 433 entitled "Easier suppression of file descriptor inheritance" is a previous attempt proposing various other alternatives, but no consensus could be reached.
This PEP has a well defined behaviour (the default value of the new *cloexec* parameter is not configurable), is more conservative (no backward compatibility issue), and is much simpler.
Even if the PEP 433 was not explicitly rejected, no consensus could be reached. I didn't want to loose all my work on this PEP and so I'm proposing something new which should make everbody agrees :-) The PEP 446 is written to supersed the PEP 433. Even if they are very close, many drawbacks of the PEP 433 (especially the configurable default value of the new cloexec parameter) goes away with the PEP 466, which explains why the PEP 466 is much shorter. I preferred to create a new PEP to keep the historic in the PEP 433. I also chose to not detail the implementation in the PEP 466, because the PEP 433 already contains a lot of information. Victor
2013/7/4 Victor Stinner <victor.stinner@gmail.com>:
Even if the PEP 433 was not explicitly rejected, no consensus could be reached. I didn't want to loose all my work on this PEP and so I'm proposing something new which should make everbody agrees :-)
Thanks Victor, I think this one is perfectly fine! cf
2013/7/4 Victor Stinner <victor.stinner@gmail.com>:
Add a new optional *cloexec* on functions creating file descriptors:
* ``io.FileIO`` * ``io.open()`` * ``open()``
The PEP 433 proposes adding an "e" mode to open in alternatives. I didn't keep this idea because the fopen() function of the GNU libc library has no mode for the O_NONBLOCK flag. IMO it is not interesting to mention it in the PEP 466. Victor
On 4 Jul, 2013, at 13:19, Victor Stinner <victor.stinner@gmail.com> wrote:
2013/7/4 Victor Stinner <victor.stinner@gmail.com>:
Add a new optional *cloexec* on functions creating file descriptors:
* ``io.FileIO`` * ``io.open()`` * ``open()``
The PEP 433 proposes adding an "e" mode to open in alternatives. I didn't keep this idea because the fopen() function of the GNU libc library has no mode for the O_NONBLOCK flag. IMO it is not interesting to mention it in the PEP 466.
I don't understand your reasoning, that is what has GNU libc to do with adding "e" mode to io.open? BTW. I have no particular fondness for an "e" flag, adding a clo_exec flag would be fine and I'm just curious. Ronald
2013/7/4 Ronald Oussoren <ronaldoussoren@mac.com>:
The PEP 433 proposes adding an "e" mode to open in alternatives. I didn't keep this idea because the fopen() function of the GNU libc library has no mode for the O_NONBLOCK flag. IMO it is not interesting to mention it in the PEP 466.
I don't understand your reasoning, that is what has GNU libc to do with adding "e" mode to io.open?
The GNU libc supports fopen(filename, "re") to set O_CLOEXEC flag on the file. I pick the idea into the PEP 433, in alternatives: open(filename, "re") sets O_CLOEXEC flag on the file. For the PEP 466, I only proposed the original API of the PEP 433: open(filename, "r", cloexec=True). Victor
-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 07/04/2013 07:03 AM, Victor Stinner wrote:
Title: Add new parameters to configure the inherance of files and for non-blocking sockets
Not commenting on either the form or the substance (pun intended), but the word you want is "inheritance" -- "inherence" is a valid term[1], but would a good deal stranger notion to apply to a file descriptor. ;) [1] https://en.wikipedia.org/wiki/Inherence Platonic'ly, Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver@palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with undefined - http://www.enigmail.net/ iEYEARECAAYFAlHV0VoACgkQ+gerLs4ltQ4YCQCgp6mFPxEVVoXAXib/jrChjRxu QkAAoLJQIfBCQezj61LCAgmVaE1kwNmM =yiPj -----END PGP SIGNATURE-----
2013/7/4 Tres Seaver <tseaver@palladion.com>:
Not commenting on either the form or the substance (pun intended), but the word you want is "inheritance" -- "inherence" is a valid term[1], but would a good deal stranger notion to apply to a file descriptor. ;)
Oh... I don't know why I wrote "inherance", it was "inheritance" in the PEP 433. Thanks, I fixed the typo in the PEP 466. Victor
First up I broadly like this. You might want to make clear that the "blocking" parameter refers only to the file creation calls (eg socket.socket) and not to the file descriptor itself, and is not to be confused with the UNIX O_NONBLOCK file descriptor flag (and whatever equivalent flag may apply on Windows). This is deducable from your PEP, but I was at first confused, and initially expected get_blocking/set_blocking functions in New Functions. On 04Jul2013 13:03, Victor Stinner <victor.stinner@gmail.com> wrote: | Other Changes | ------------- | The ``subprocess.Popen`` class must clear the close-on-exec flag of file | descriptors of the ``pass_fds`` parameter. I would expect Popen and friends to need to both clear the flag to get the descriptors across the fork() call, and _possibly_ to set the flag again after the fork. Naively, I would expect the the flag to be as it was before the Popen call, after the call. This is not addressed. Cheers, -- Cameron Simpson <cs@zip.com.au> Time is nature's way of keeping everything from happening at once.
On 05Jul2013 10:41, I wrote: | On 04Jul2013 13:03, Victor Stinner <victor.stinner@gmail.com> wrote: | | Other Changes | | ------------- | | The ``subprocess.Popen`` class must clear the close-on-exec flag of file | | descriptors of the ``pass_fds`` parameter. | | I would expect Popen and friends to need to both clear the flag to | get the descriptors across the fork() call, and _possibly_ to set | the flag again after the fork. Naively, I would expect the the flag | to be as it was before the Popen call, after the call. This is harder than I'd thought through. Descriptors go across a fork() anyway, and after an exec() there is nobody to change their state. In order for the parent to restore the prior close-on-exec state (if deemed the right thing to do) it would need to know when the exec was done in the child. You'd need something like an extra descriptor attached to a pipe marked close-on-exec so that the parent could see EOF on the read end after the exec, since the exec would close the write end. tractable, but baroque. So I'd have the PEP take a position on the post-exec() state of the passed file descriptors, and if state is not going to be restored it should outright say that passing an fd clears the close-on-exec flag. Which is it does, but I think the doco should be pretty overt about: - this side effect of clearing close-on-exec on these fds - that the file descriptor and its close-on-exec state is common between the parent and the child - that setting of close-on-exec on the fds in the parent could in principle comprimise the exec in the child if done too soon If you decide state should get restored, the implementation gets a bit fiddlier to detect the exec() in the child. I can't speak for what might be required on non-UNIX. Cheers, -- Cameron Simpson <cs@zip.com.au> Agree, for Law is costly. -- Very good advice to litigious Persons, founded upon Reason and Experience; for many Times the Charges of a Suit exceed the Value of the Thing in Dispute. - Bailey's dictionary, 1736
2013/7/5 Cameron Simpson <cs@zip.com.au>:
You might want to make clear that the "blocking" parameter refers only to the file creation calls (eg socket.socket) and not to the file descriptor itself, and is not to be confused with the UNIX O_NONBLOCK file descriptor flag (and whatever equivalent flag may apply on Windows).
The two following codes are the same: s = socket.socket(..., blocking=False) and s = socket.socket(...) s.setblocking(False) Both set O_NONBLOCK flag (UNIX) or clear HANDLE_FLAG_INHERIT (Windows) on the socket (which is a file descriptor). socket.socket(..., blocking=False) cannot fail with EAGAIN or EWOULDBLOCK (at least on Linux according to the manual page).
This is deducable from your PEP, but I was at first confused, and initially expected get_blocking/set_blocking functions in New Functions.
socket objects already have get_blocking() and set_blocking() methods. If we supported the blocking flag on any file descriptor, it would make sense to add such function to the os module. But I explained in the "Add blocking parameter for file descriptors and Windows overlapped I/O" section why I only want to set/clear the blocking file on sockets.
I would expect Popen and friends to need to both clear the flag to get the descriptors across the fork() call, and _possibly_ to set the flag again after the fork. Naively, I would expect the the flag to be as it was before the Popen call, after the call.
As explained in the "Inheritance of file descriptors" section, the close-on-exec flag has no effect on fork: "The flag has no effect on fork(), all file descriptors are inherited by the child process." The close-on-exec flag is cleared after the fork and before the exec(). Pseudo-code for Popen: pid = os.fork() if pid: return pid # child process for fd in pass_fds: os.set_cloexec(fd, False) os.execv(...) Victor
On 05Jul2013 08:24, Victor Stinner <victor.stinner@gmail.com> wrote: | 2013/7/5 Cameron Simpson <cs@zip.com.au>: | > You might want to make clear that the "blocking" parameter refers | > only to the file creation calls (eg socket.socket) and not to the | > file descriptor itself, and is not to be confused with the UNIX | > O_NONBLOCK file descriptor flag (and whatever equivalent flag may | > apply on Windows). | | The two following codes are the same: | | s = socket.socket(..., blocking=False) | and | s = socket.socket(...) | s.setblocking(False) | | Both set O_NONBLOCK flag (UNIX) Oh, how embarassing. | or clear HANDLE_FLAG_INHERIT (Windows) | on the socket (which is a file descriptor). | | socket.socket(..., blocking=False) cannot fail with EAGAIN or | EWOULDBLOCK (at least on Linux according to the manual page). Fair enough. | > This is deducable from your PEP, but I was at first confused, and | > initially expected get_blocking/set_blocking functions in New | > Functions. | | socket objects already have get_blocking() and set_blocking() methods. Ah, ok then. As far as it goes. Shouldn't there be a general purpose os.get_blocking() and os.set_blocking() functions that operate on any file descriptor, paralleling os.get_cloexec() and os.set_cloexec()? They're not socket specific operations in principle, merely commonly used with sockets. | If we supported the blocking flag on any file descriptor, it would | make sense to add such function to the os module. But I explained in | the "Add blocking parameter for file descriptors and Windows | overlapped I/O" section why I only want to set/clear the blocking file | on sockets. But WHY? I think socket file decriptors should be treated little differently from other file descriptors. Certainly a stream socket connection is outstanding like other unseekable file descriptors in many respects. Just "only wanting it on sockets" seems a bit special purpose. If you're going address O_NONBLOCK in this proposal in addition to the close-on-exec flag, it should be general purpose. Otherwise, I think it should be separated out into a separate proposal if you're proposing it just for sockets; make this proposal just close-on-exec and forget the blocking stuff for this specific PEP. | > I would expect Popen and friends to need to both clear the flag to | > get the descriptors across the fork() call, and _possibly_ to set | > the flag again after the fork. Naively, I would expect the the flag | > to be as it was before the Popen call, after the call. | | As explained in the "Inheritance of file descriptors" section, the | close-on-exec flag has no effect on fork: | | "The flag has no effect on fork(), all file descriptors are inherited | by the child process." Agreed, see my second post where I explained I'd misthought it. However, as stated there, I think the side effects should be fairly overtly stated in the docuemntation. | The close-on-exec flag is cleared after the fork and before the | exec(). Pseudo-code for Popen: | | pid = os.fork() | if pid: | return pid | # child process | for fd in pass_fds: os.set_cloexec(fd, False) | os.execv(...) Fine. No state restore is fine with me. I think it should be as clear as possible in the doco. Cheers, -- Cameron Simpson <cs@zip.com.au> Experience is what you have got after you needed it.
2013/7/5 Cameron Simpson <cs@zip.com.au>:
| Both set O_NONBLOCK flag (UNIX)
Oh, how embarassing.
You said that the PEP is not cristal clear. Do you have a suggestion to make it more clear? Should I mention that the close-on-exec flag is O_CLOEXEC on UNIX, and HANDLE_FLAG_INHERIT on Windows? (except that HANDLE_FLAG_INHERIT set means inheritable, whereas O_CLOEXEC set means *not* inheritable)
| > This is deducable from your PEP, but I was at first confused, and | > initially expected get_blocking/set_blocking functions in New | > Functions. | | socket objects already have get_blocking() and set_blocking() methods.
Ah, ok then. As far as it goes. Shouldn't there be a general purpose os.get_blocking() and os.set_blocking() functions that operate on any file descriptor, paralleling os.get_cloexec() and os.set_cloexec()?
I didn't propose to add these two functions, because I'm not sure that they are really useful. We can add os.get_blocking() and os.set_blocking() on UNIX, but these functions would not be available on Windows. On Windows, os.set_blocking() only makes sense for sockets, and sockets already have a set_blocking() method. On UNIX, it makes sense because set_blocking(fd, True) can be implemented as a single syscall on some platforms (using ioctl), whereas most developers implement it using two syscalls (fcntl to get current flags, and fcntl to set the O_NONBLOCK flag) maybe because it is more portable. But we cannot add a new "blocking" parameter to all functions creating file descriptors, like open(), because of Windows. Or do you like the "raise NotImplementedError on Windows" option?
But WHY? I think socket file decriptors should be treated little differently from other file descriptors.
Because of Windows. On Windows, sockets and files are two different things. sockets have no "file descriptor", they have a HANDLE. For example, sockobj.fileno() on Windows returns a huge number, not something like 3 or 10. The C type HANDLE is larger than a file descriptor on Windows 64-bit (sizeof(void*) > sizeof(int)) and so functions must be modified to use a wider type (to parse their argument), and must support the HANDLE type (_open_osfhandle() can be used to have a single version of the code).
Otherwise, I think it should be separated out into a separate proposal if you're proposing it just for sockets; make this proposal just close-on-exec and forget the blocking stuff for this specific PEP.
The reason for addressing close-on-exec and blocking parameters in the same PEP is that new versions of operating systems support setting the two flags at the creation a new file descriptor and a new socket. Read the article linked at the end of the PEP for the background: http://udrepper.livejournal.com/20407.html The Python call socket.socket(..., cloexec=True, blocking=False) only calls one syscall on Linux >= 2.6.27.
| > I would expect Popen and friends to need to both clear the flag to | > get the descriptors across the fork() call, and _possibly_ to set | > the flag again after the fork. Naively, I would expect the the flag | > to be as it was before the Popen call, after the call. | | As explained in the "Inheritance of file descriptors" section, the | close-on-exec flag has no effect on fork: | | "The flag has no effect on fork(), all file descriptors are inherited | by the child process."
Agreed, see my second post where I explained I'd misthought it. However, as stated there, I think the side effects should be fairly overtly stated in the docuemntation.
Sorry, I don't understand. Which "side effect"? The close-on-exec flag only affects inheritance of file descriptors at the execv() syscall, not at fork(). And execv() can be called without fork(). Popen must clear close-on-exec flag on files from its pass_fds parameter for convinience. It would be surprising to not inherit a file descriptor passed to Popen in pass_fds, don't you think so? os.execv() has no pass_fds parameter, and it is a thin wrapper on the syscall. Popen is a high-level API, that's why it prepares more things before calling the new program.
| The close-on-exec flag is cleared after the fork and before the | exec(). Pseudo-code for Popen: | | pid = os.fork() | if pid: | return pid | # child process | for fd in pass_fds: os.set_cloexec(fd, False) | os.execv(...)
Fine. No state restore is fine with me. I think it should be as clear as possible in the doco.
I don't understand. I already wrote "The flag has no effect on fork(), all file descriptors are inherited by the child process" in the PEP. It is not enough? Do you have a suggestion to explain it better? Victor
On 05Jul2013 19:03, Victor Stinner <victor.stinner@gmail.com> wrote: | 2013/7/5 Cameron Simpson <cs@zip.com.au>: | > | Both set O_NONBLOCK flag (UNIX) | > | > Oh, how embarassing. | | You said that the PEP is not cristal clear. Do you have a suggestion | to make it more clear? | | Should I mention that the close-on-exec flag is O_CLOEXEC on UNIX, and | HANDLE_FLAG_INHERIT on Windows? (except that HANDLE_FLAG_INHERIT set | means inheritable, whereas O_CLOEXEC set means *not* inheritable) It wouldn't hurt. Otherwise, colour me mostly convinced. I've read your "Rejected Alternatives" more closely and Ulrich Drepper's article, though I think the article also supports adding a blocking (default True) parameter to open() and os.open(). If you try to change that default on a platform where it doesn't work, an exception should be raised. | The close-on-exec flag only affects inheritance of file descriptors at | the execv() syscall, not at fork(). And execv() can be called without | fork(). Yes. Please forget I mentioned fork(); it is only relevant if you were offering some facility to undo the addition of cloexec to a Popen passed file descriptor. Which you are not. | Popen must clear close-on-exec flag on files from its pass_fds | parameter for convinience. It would be surprising to not inherit a | file descriptor passed to Popen in pass_fds, don't you think so? Yes yes yes. | I don't understand. I already wrote "The flag has no effect on fork(), | all file descriptors are inherited by the child process" in the PEP. | It is not enough? Do you have a suggestion to explain it better? My concerns are probably as well bundled with concerns about users not realising the a file descriptor state after a fork is shared, and then being confused if the other process did a seek() etc. I think I'm just concerned about a naive caller using this scenario: # gather up some already open fds to pass fds = (5,6,7) P = Popen(...., pass_fds=fds) # now restore the cloexec state from before for fd in fds: os.set_cloexec(fd) where the cleanup loop in the parent gets to run before the exec() in the child. We could just consider it one of the pitfalls of fork/exec situations in general an expect people being this fiddly to need to add some synchornisation. Please consider your argument won. Cheers, -- Cameron Simpson <cs@zip.com.au>
I've read your "Rejected Alternatives" more closely and Ulrich Drepper's article, though I think the article also supports adding a blocking (default True) parameter to open() and os.open(). If you try to change that default on a platform where it doesn't work, an exception should be raised.
Contrarily to close-on-exec, non-blocking only applies to a limited type of files (e.g. it doesn't work for regular files, which represent 90% of open() use cases). Also, one of the main reasons for exposing close-on-exec in open()/socket() etc is to make it possible to create file descriptors with the close-on-exec flag atomically, to prevent unwanted FD inheritance especially in multi-threaded code. And that's not necessary for the non-blocking parameter. Those are two reasons why IMO "blocking" doesn't have to receive the same treatment as close-on-exec (there's also the Windows issue but I'm not familiar with it). cf
2013/7/6 Charles-François Natali <cf.natali@gmail.com>:
I've read your "Rejected Alternatives" more closely and Ulrich Drepper's article, though I think the article also supports adding a blocking (default True) parameter to open() and os.open(). If you try to change that default on a platform where it doesn't work, an exception should be raised.
Contrarily to close-on-exec, non-blocking only applies to a limited type of files (e.g. it doesn't work for regular files, which represent 90% of open() use cases).
What do you mean by "does not work"? On Linux, O_NONBLOCK flag can be set on regular files, sockets, pipes, etc. Example with a regular file on Linux 3.9: smithers$ python3 Python 3.3.0 (default, Sep 29 2012, 22:07:38)
from fcntl import * import os f=open("/etc/issue", "rb") fd=f.fileno() flags=fcntl(fd, F_GETFL) fcntl(fd, F_SETFL, flags|os.O_NONBLOCK) 0 fcntl(fd, F_GETFL) & os.O_NONBLOCK 2048 f.read(10) b'Fedora rel'
For example, asyncore.file_dispatcher() uses fcntl to set the O_NONBLOCK flag. (The asyncore module is probably the only module of the stdlib which would benefit of a new os.set_blocking() function.)
Also, one of the main reasons for exposing close-on-exec in open()/socket() etc is to make it possible to create file descriptors with the close-on-exec flag atomically, to prevent unwanted FD inheritance especially in multi-threaded code. And that's not necessary for the non-blocking parameter.
In the review of PEP 433 on this mailing list, Martin von Loewis said that the PEP does guarantee the atomicity. The implementation of the PEP falls back on ioctl or fcntl to set the flag on old Linux versions (or if the OS does not support setting O_CLOEXEC flag when creating a file descriptor). The GIL is released during system calls, so another thread can call execv(). That's why I mentionned explicitly in the PEP 446 that it does not offer any atomicity guarantee. But open(filename, blocking=False) on Linux has at least one advantage over f=open(filename); os.set_blocking(f.fileno()): it only uses one syscall (versus 2 or 3 syscalls). Victor
On 6 Jul, 2013, at 14:04, Victor Stinner <victor.stinner@gmail.com> wrote:
2013/7/6 Charles-François Natali <cf.natali@gmail.com>:
I've read your "Rejected Alternatives" more closely and Ulrich Drepper's article, though I think the article also supports adding a blocking (default True) parameter to open() and os.open(). If you try to change that default on a platform where it doesn't work, an exception should be raised.
Contrarily to close-on-exec, non-blocking only applies to a limited type of files (e.g. it doesn't work for regular files, which represent 90% of open() use cases).
What do you mean by "does not work"? On Linux, O_NONBLOCK flag can be set on regular files, sockets, pipes, etc.
I guess he means that O_NONBLOCK doesn't actually do anything for regular files, regular files are always ready for I/O as far as select et. al. are concerned and I/O will block when data has to be read from disk (or in the case of a network filesystem, from another machine). Ronald
On 06Jul2013 11:23, Charles-François Natali <cf.natali@gmail.com> wrote: | > I've read your "Rejected Alternatives" more closely and Ulrich | > Drepper's article, though I think the article also supports adding | > a blocking (default True) parameter to open() and os.open(). If you | > try to change that default on a platform where it doesn't work, an | > exception should be raised. | | Contrarily to close-on-exec, non-blocking only applies to a limited | type of files (e.g. it doesn't work for regular files, which represent | 90% of open() use cases). sockets, pipes, serial devices, ... And you can set it on anything. Just because some things don't block anyway isn't really a counter argument. Cheers, -- Cameron Simpson <cs@zip.com.au> Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law. - Douglas Hosfstadter, Godel, Escher, Bach: an Eternal Golden Braid
2013/7/7 Cameron Simpson <cs@zip.com.au>:
On 06Jul2013 11:23, Charles-François Natali <cf.natali@gmail.com> wrote: | > I've read your "Rejected Alternatives" more closely and Ulrich | > Drepper's article, though I think the article also supports adding | > a blocking (default True) parameter to open() and os.open(). If you | > try to change that default on a platform where it doesn't work, an | > exception should be raised. | | Contrarily to close-on-exec, non-blocking only applies to a limited | type of files (e.g. it doesn't work for regular files, which represent | 90% of open() use cases).
sockets, pipes, serial devices, ...
How do you use open() on a socket (which are already covered by socket(blocking=...)? Also, I said *regular files* - for which O_NONBLOCK doesn't make sense - represent 90% of io.open() use cases, and stand by this claim. Nothing prevents you from setting the FD non-blocking manually.
And you can set it on anything. Just because some things don't block anyway isn't really a counter argument.
Well, it complicates the signature and implementation. If we go the same way, why stop there and not expose O_DSYNC, O_SYNC, O_DIRECT... When using a high-level API like io.open(), I think we should only expose portable flags, which are supported both on all operating systems (like the 'x' O_EXCL flag added in 3.3) and file types. If you want precise control over the open() sementics, os.open() is the way to go (that's also the rationale behind io.open() opener argument, see http://bugs.python.org/issue12105) cf
2013/7/7 Charles-François Natali <cf.natali@gmail.com>:
2013/7/7 Cameron Simpson <cs@zip.com.au>:
On 06Jul2013 11:23, Charles-François Natali <cf.natali@gmail.com> wrote: | > I've read your "Rejected Alternatives" more closely and Ulrich | > Drepper's article, though I think the article also supports adding | > a blocking (default True) parameter to open() and os.open(). If you | > try to change that default on a platform where it doesn't work, an | > exception should be raised. | | Contrarily to close-on-exec, non-blocking only applies to a limited | type of files (e.g. it doesn't work for regular files, which represent | 90% of open() use cases).
sockets, pipes, serial devices, ...
How do you use open() on a socket (which are already covered by socket(blocking=...)? Also, I said *regular files* - for which O_NONBLOCK doesn't make sense - represent 90% of io.open() use cases, and stand by this claim. Nothing prevents you from setting the FD non-blocking manually.
Ok, I think that the best consensus here is to add a new os.set_blocking() function. The function would not be available on Windows, but should be available on all UNIX platforms. The advantage of os.set_blocking() over fcntl is that it may be implemented using ioctl (FIONBIO) to use 1 syscall instead of 2. It is also easier to remember calling "os.set_blocking(fd, True)" instead of "fcntl.fcntl(fd, fcntl.F_SETFL, fcntl.fcntl(fcntl.F_GETFL, fd) | os.O_NONBLOCK)". See the diff: http://hg.python.org/peps/rev/ce61588d244c @Charles-François and Cameron: Do you prefer to new version of the PEP? http://www.python.org/dev/peps/pep-0446/ (will be updated in a few minutes) I'm not sure that many people are concerned by the number of syscalls, but there is at least Peter Portante who opened a thread "Modules/socketmodule.c: avoiding second fcntl() call worth the effort?" on this mailing list, last January: http://mail.python.org/pipermail/python-dev/2013-January/123661.html "OpenStack Swift using the Eventlet module, which sets the accepted socket non-blocking, resulting in twice the number of fcntl() calls. Not a killer on performance, but it seems simple enough to save a system call here." (This specific issue will be solved by the new blocking parameter added to socket constructor ;-)) Victor
Sorry for the delay; work got in the way. On 07Jul2013 15:21, Victor Stinner <victor.stinner@gmail.com> wrote: | Ok, I think that the best consensus here is to add a new | os.set_blocking() function. The function would not be available on | Windows, but should be available on all UNIX platforms. Thanks. | See the diff: | http://hg.python.org/peps/rev/ce61588d244c This sentence: The flag is cleared in the child process before executing the program, the change does not change the flag in the parent process. needs a semicolon, not a comma. | @Charles-François and Cameron: Do you prefer to new version of the PEP? | http://www.python.org/dev/peps/pep-0446/ | (will be updated in a few minutes) I'm happy with it. Thank you, -- Cameron Simpson <cs@zip.com.au>
2013/7/7 Charles-François Natali <cf.natali@gmail.com>:
Well, it complicates the signature and implementation. If we go the same way, why stop there and not expose O_DSYNC, O_SYNC, O_DIRECT...
I added this counter-argument to the PEP.
If you want precise control over the open() sementics, os.open() is the way to go (that's also the rationale behind io.open() opener argument, see http://bugs.python.org/issue12105)
Right. Victor
2013/7/6 Cameron Simpson <cs@zip.com.au>:
Yes. Please forget I mentioned fork(); it is only relevant if you were offering some facility to undo the addition of cloexec to a Popen passed file descriptor. Which you are not.
Oh... gotcha. I now understood your concern. There is a little "trick" here: at fork, file descriptors are duplicated in the child process and almost all properties (open state, flags, offset, etc.) are shared. There is one little exception: file attributes are not shared, and there is only one file attribute: O_CLOEXEC. Setting O_CLOEXEC in a child process does not affect the flag in the parent process ;-) I will add a unit test to check this. I modified the PEP to explain that, and I also mentioned the name of the close-on-exec and blocking flags: http://hg.python.org/peps/rev/425f831fddf7 Victor
On 06Jul2013 14:43, Victor Stinner <victor.stinner@gmail.com> wrote: | 2013/7/6 Cameron Simpson <cs@zip.com.au>: | > Yes. Please forget I mentioned fork(); it is only relevant if you | > were offering some facility to undo the addition of cloexec to a | > Popen passed file descriptor. Which you are not. | | Oh... gotcha. I now understood your concern. | | There is a little "trick" here: at fork, file descriptors are | duplicated in the child process and almost all properties (open state, | flags, offset, etc.) are shared. There is one little exception: file | attributes are not shared, and there is only one file attribute: | O_CLOEXEC. Setting O_CLOEXEC in a child process does not affect the | flag in the parent process ;-) I will add a unit test to check this. That's news to me. Interesting. I can't find UNIX doco to support this, though I haven't finished looking. None of the POSIX, Linux or MacOSX fork() manual pages seem to describe this. Can you point me at doco for this? I thought file descriptors were a direct index into a per-process table of file handle structures, and did not think the latter got part copies, part shared over a fork. Also, is O_CLOEXEC really the only file attribute? So O_NONBLOCK is a flag and not an attribute (guessing the terminology distinguishes these)? Now you mention it, ISTR noticing that there were distinct ioctls to adjust flags and something else. | I modified the PEP to explain that, and I also mentioned the name of | the close-on-exec and blocking flags: | http://hg.python.org/peps/rev/425f831fddf7 Thanks. Your change says: On Windows, the close-on-exec flag is ``HANDLE_FLAG_INHERIT``. I feel it would be clearer to say: On Windows, the close-on-exec flag is the inverse of ``HANDLE_FLAG_INHERIT``. or On Windows, the close-on-exec flag implemented using ``HANDLE_FLAG_INHERIT``. (This latter is fine because the rest of the paragraph explains the meaning of HANDLE_FLAG_INHERIT.) The bare "is" in your current wording suggests to me that the truthiness (ick!) of close-on-exec is the same as for HANDLE_FLAG_INHERIT, which it isn't. -- Cameron Simpson <cs@zip.com.au> I've always been a big Greenaway fan - I've seen and enjoyed "The Falls" for crying out loud. - Peter Alexander Merel <pete@extro.ucc.su.OZ.AU>
2013/7/7 Cameron Simpson <cs@zip.com.au>:
On 06Jul2013 14:43, Victor Stinner <victor.stinner@gmail.com> wrote: | 2013/7/6 Cameron Simpson <cs@zip.com.au>: | > Yes. Please forget I mentioned fork(); it is only relevant if you | > were offering some facility to undo the addition of cloexec to a | > Popen passed file descriptor. Which you are not. | | Oh... gotcha. I now understood your concern. | | There is a little "trick" here: at fork, file descriptors are | duplicated in the child process and almost all properties (open state, | flags, offset, etc.) are shared. There is one little exception: file | attributes are not shared, and there is only one file attribute: | O_CLOEXEC. Setting O_CLOEXEC in a child process does not affect the | flag in the parent process ;-) I will add a unit test to check this.
That's news to me. Interesting.
I can't find UNIX doco to support this, though I haven't finished looking. None of the POSIX, Linux or MacOSX fork() manual pages seem to describe this.
Can you point me at doco for this? I thought file descriptors were a direct index into a per-process table of file handle structures, and did not think the latter got part copies, part shared over a fork.
Sorry, I found this fact my mistake, not in the doc :-) You can verify this behaviour using the follownig script: hg.python.org/peps/file/tip/pep-0466/test_cloexec.py Output on Linux: initial state: fd=3, cloexec=False child process after fork, set cloexec: cloexec=True child process after exec: fd=3, cloexec=<invalid file descriptor> parent process after fork: cloexec=False parent process after exec: fd=3, cloexec=False "parent process after fork: cloexec=False" means that the flag was not modified in the parent process.
Also, is O_CLOEXEC really the only file attribute? So O_NONBLOCK is a flag and not an attribute (guessing the terminology distinguishes these)? Now you mention it, ISTR noticing that there were distinct ioctls to adjust flags and something else.
See the manual page: http://linux.die.net/man/2/fcntl There are "File descriptor flags " and "File status flags", it looks like they are inherited/shared differently between the parent and the child process.
Your change says:
On Windows, the close-on-exec flag is ``HANDLE_FLAG_INHERIT``.
I feel it would be clearer to say:
On Windows, the close-on-exec flag is the inverse of ``HANDLE_FLAG_INHERIT``.
Ok, fixed in the PEP. Victor
participants (5)
-
Cameron Simpson -
Charles-François Natali -
Ronald Oussoren -
Tres Seaver -
Victor Stinner