Hi, I would like add a new flag to open() mode to close the file on exec: "e". This feature exists using different APIs depending on the OS and OS version: O_CLOEXEC, FD_CLOEXEC and O_NOINHERIT. Do you consider that such flag would be interesting? On Linux (glibc), the feature is available as the "e" flag for fopen() (O_CLOEXEC or FD_CLOEXEC depending on the kernel version). With Visual Studio, it's the "N" flag (O_NOINHERIT). I would like to modify open() to be able to benefit of O_CLOEXEC atomicity. The FD_CLOEXEC method has a race condition if another thread does fork() between open() and fcntl() (two calls to fcntl() are required). I created a patch available on the following issue: http://bugs.python.org/issue16850 Drawbacks of my idea/patch: - my patch raises NotImplementedError if the platform doesn't support close-and-exec flag. I tested Linux, OpenBSD, FreeBSD, OpenIndiana (Solaris) and Mac OS X: my patch works on all these platforms (except OpenBSD 4.9 because of an OS bug fixed in OpenBSD 5.2). I don't know platform without this flag. - my patch raises an error if opener argument of open() is used: opener and the "e" mode are exclusive (see the issue for details) --- I first proposed to only expose the atomic close-and-exec option, but only a few platforms (Linux 2.6.23+, FreeBSD 8.3+, Windows, QNX) support it. It's more convinient to have the best-effort method (only atomic if available). This article of Ulrich Drepper explain which kind of problem are solved by O_CLOEXEC: http://udrepper.livejournal.com/20407.html Victor
2013/1/7 Victor Stinner <victor.stinner@gmail.com>:
Hi,
I would like add a new flag to open() mode to close the file on exec: "e". This feature exists using different APIs depending on the OS and OS version: O_CLOEXEC, FD_CLOEXEC and O_NOINHERIT. Do you consider that such flag would be interesting?
I'm not sure it's worth cluttering the open() interface with such a non-portable option. People requiring such control should use the low-level os.open interface. -- Regards, Benjamin
On Mon, Jan 07, 2013 at 06:03:54PM -0600, Benjamin Peterson wrote:
2013/1/7 Victor Stinner <victor.stinner@gmail.com>:
Hi,
I would like add a new flag to open() mode to close the file on exec: "e". This feature exists using different APIs depending on the OS and OS version: O_CLOEXEC, FD_CLOEXEC and O_NOINHERIT. Do you consider that such flag would be interesting? Having close-on-exec makes the code much cleaner, often. Definitely interesting.
I'm not sure it's worth cluttering the open() interface with such a non-portable option. People requiring such control should use the low-level os.open interface. If the best-effort fallback is included, it is quite portable. Definitely all modern and semi-modern systems support either the atomic or the nonatomic methods.
Zbyszek
On Mon, Jan 7, 2013 at 4:03 PM, Benjamin Peterson <benjamin@python.org>wrote:
2013/1/7 Victor Stinner <victor.stinner@gmail.com>:
Hi,
I would like add a new flag to open() mode to close the file on exec: "e". This feature exists using different APIs depending on the OS and OS version: O_CLOEXEC, FD_CLOEXEC and O_NOINHERIT. Do you consider that such flag would be interesting?
I'm not sure it's worth cluttering the open() interface with such a non-portable option. People requiring such control should use the low-level os.open interface.
The ability to supply such flags really belongs on _all_ high or low level file descriptor creating APIs so that things like subprocess_cloexec_pipe() would not be necessary: http://hg.python.org/cpython/file/0afa7b323abb/Modules/_posixsubprocess.c#l7... I'm not excited about raising an exception when it isn't supported; it should attempt to get the same behavior via multiple API calls instead (thus introducing a race condition for threads+fork+exec, but there is no choice at that point and it is what most people would want in that situation regardless). Not being supported will be such an extremely rare situation for any OS someone runs a Python 3.4 program on that that detail is pretty much a non-issue either way as far as I'm concerned. -gps
2013/1/7 Gregory P. Smith <greg@krypto.org>:
On Mon, Jan 7, 2013 at 4:03 PM, Benjamin Peterson <benjamin@python.org> wrote:
2013/1/7 Victor Stinner <victor.stinner@gmail.com>:
Hi,
I would like add a new flag to open() mode to close the file on exec: "e". This feature exists using different APIs depending on the OS and OS version: O_CLOEXEC, FD_CLOEXEC and O_NOINHERIT. Do you consider that such flag would be interesting?
I'm not sure it's worth cluttering the open() interface with such a non-portable option. People requiring such control should use the low-level os.open interface.
The ability to supply such flags really belongs on _all_ high or low level file descriptor creating APIs so that things like subprocess_cloexec_pipe() would not be necessary: http://hg.python.org/cpython/file/0afa7b323abb/Modules/_posixsubprocess.c#l7...
I think the open() interface should have consistent and non-conditional support features to the maximum extent possible. The recent addition of "x" is a good example I think. The myriad cloexec APIs between different platforms suggests to me that using this features requires understanding its various quirks on different platforms. -- Regards, Benjamin
Benjamin wrote:
I'm not sure it's worth cluttering the open() interface with such a non-portable option. Zbyszek wrote: If the best-effort fallback is included, it is quite portable. Definitely all modern and semi-modern systems support either the atomic or the nonatomic methods. Gregory wrote: I'm not excited about raising an exception when it isn't supported; it should attempt to get the same behavior via multiple API calls instead
Yes, I'm proposing the best-effort approach: use O_CLOEXEC/O_NOINHERIT if available, or fcntl()+FD_CLOEXEC otherwise. My patch requires fcntl() + FD_CLOEXEC flag or open() + O_NOINHERIT flag. I failed to find an OS where none of these flag/function is present. Usually, when I would like to test the portability, I test Linux, Windows and Mac OS X. Then I test FreeBSD, OpenBSD and OpenIndiana (Solaris). I don't test other OS because I don't know them and they are not installed on my PC :-) (I have many VM.) Should I try other platforms? Benjamin wrote:
People requiring such control should use the low-level os.open interface.
os.open() is not very convinient because you have to chose the flag and the functions depending on the OS. If the open() flag is rejected, we should at least provide an helper for os.open() + fcntl(). The myriad cloexec APIs between different platforms suggests to me that using this features requires understanding its various quirks on different platforms. Victor 2013/1/8 Benjamin Peterson <benjamin@python.org>:
2013/1/7 Gregory P. Smith <greg@krypto.org>:
On Mon, Jan 7, 2013 at 4:03 PM, Benjamin Peterson <benjamin@python.org> wrote:
2013/1/7 Victor Stinner <victor.stinner@gmail.com>:
Hi,
I would like add a new flag to open() mode to close the file on exec: "e". This feature exists using different APIs depending on the OS and OS version: O_CLOEXEC, FD_CLOEXEC and O_NOINHERIT. Do you consider that such flag would be interesting?
I'm not sure it's worth cluttering the open() interface with such a non-portable option. People requiring such control should use the low-level os.open interface.
The ability to supply such flags really belongs on _all_ high or low level file descriptor creating APIs so that things like subprocess_cloexec_pipe() would not be necessary: http://hg.python.org/cpython/file/0afa7b323abb/Modules/_posixsubprocess.c#l7...
I think the open() interface should have consistent and non-conditional support features to the maximum extent possible. The recent addition of "x" is a good example I think. The myriad cloexec APIs between different platforms suggests to me that using this features requires understanding its various quirks on different platforms.
-- Regards, Benjamin
Oops, I sent my email too early by mistake (it was not finished).
The myriad cloexec APIs between different platforms suggests to me that using this features requires understanding its various quirks on different platforms.
Sorry, I don't understand. What do you mean by "various quirks". The "close-on-exec" feature is implemented differently depending on the platform, but it always have the same meaning. It closes the file when a subprocess is created. Running a subprocess is also implemented differently depending on the OS, there are two mains approaches: fork()+exec() on UNIX, <something else> on Windows (I don't know how it works on Windows). Extract of fcntl() manual page on Linux: "If the FD_CLOEXEC bit is 0, the file descriptor will remain open across an execve(2), otherwise it will be closed." I would like to expose the OS feature using a portable API to hide the "The myriad cloexec APIs". Victor 2013/1/8 Victor Stinner <victor.stinner@gmail.com>:
Benjamin wrote:
I'm not sure it's worth cluttering the open() interface with such a non-portable option. Zbyszek wrote: If the best-effort fallback is included, it is quite portable. Definitely all modern and semi-modern systems support either the atomic or the nonatomic methods. Gregory wrote: I'm not excited about raising an exception when it isn't supported; it should attempt to get the same behavior via multiple API calls instead
Yes, I'm proposing the best-effort approach: use O_CLOEXEC/O_NOINHERIT if available, or fcntl()+FD_CLOEXEC otherwise.
My patch requires fcntl() + FD_CLOEXEC flag or open() + O_NOINHERIT flag. I failed to find an OS where none of these flag/function is present.
Usually, when I would like to test the portability, I test Linux, Windows and Mac OS X. Then I test FreeBSD, OpenBSD and OpenIndiana (Solaris). I don't test other OS because I don't know them and they are not installed on my PC :-) (I have many VM.)
Should I try other platforms?
Benjamin wrote:
People requiring such control should use the low-level os.open interface.
os.open() is not very convinient because you have to chose the flag and the functions depending on the OS. If the open() flag is rejected, we should at least provide an helper for os.open() + fcntl().
The myriad cloexec APIs between different platforms suggests to me that using this features requires understanding its various quirks on different platforms.
Victor
2013/1/8 Benjamin Peterson <benjamin@python.org>:
2013/1/7 Gregory P. Smith <greg@krypto.org>:
On Mon, Jan 7, 2013 at 4:03 PM, Benjamin Peterson <benjamin@python.org> wrote:
2013/1/7 Victor Stinner <victor.stinner@gmail.com>:
Hi,
I would like add a new flag to open() mode to close the file on exec: "e". This feature exists using different APIs depending on the OS and OS version: O_CLOEXEC, FD_CLOEXEC and O_NOINHERIT. Do you consider that such flag would be interesting?
I'm not sure it's worth cluttering the open() interface with such a non-portable option. People requiring such control should use the low-level os.open interface.
The ability to supply such flags really belongs on _all_ high or low level file descriptor creating APIs so that things like subprocess_cloexec_pipe() would not be necessary: http://hg.python.org/cpython/file/0afa7b323abb/Modules/_posixsubprocess.c#l7...
I think the open() interface should have consistent and non-conditional support features to the maximum extent possible. The recent addition of "x" is a good example I think. The myriad cloexec APIs between different platforms suggests to me that using this features requires understanding its various quirks on different platforms.
-- Regards, Benjamin
2013/1/8 Victor Stinner <victor.stinner@gmail.com>:
Oops, I sent my email too early by mistake (it was not finished).
The myriad cloexec APIs between different platforms suggests to me that using this features requires understanding its various quirks on different platforms.
Sorry, I don't understand. What do you mean by "various quirks". The "close-on-exec" feature is implemented differently depending on the platform, but it always have the same meaning. It closes the file when a subprocess is created. Running a subprocess is also implemented differently depending on the OS, there are two mains approaches: fork()+exec() on UNIX, <something else> on Windows (I don't know how it works on Windows).
Extract of fcntl() manual page on Linux: "If the FD_CLOEXEC bit is 0, the file descriptor will remain open across an execve(2), otherwise it will be closed."
I would like to expose the OS feature using a portable API to hide the "The myriad cloexec APIs".
Okay, fair enough, but I really would like it not to ever raise NotImplementedError. Then you would end up having different codepaths for various oses anyway. -- Regards, Benjamin
2013/1/8 Victor Stinner <victor.stinner@gmail.com>:
2013/1/8 Benjamin Peterson <benjamin@python.org>:
Okay, fair enough, but I really would like it not to ever raise NotImplementedError. Then you would end up having different codepaths for various oses anyway.
So what do you suggest?
If the only systems it doesn't work on is ancient RedHat, that's probably okay. -- Regards, Benjamin
2013/1/8 Benjamin Peterson <benjamin@python.org>:
2013/1/8 Victor Stinner <victor.stinner@gmail.com>:
2013/1/8 Benjamin Peterson <benjamin@python.org>:
Okay, fair enough, but I really would like it not to ever raise NotImplementedError. Then you would end up having different codepaths for various oses anyway.
So what do you suggest?
If the only systems it doesn't work on is ancient RedHat, that's probably okay.
What do you mean? NotIlmplementedError is acceptable if only rare and/or old OS raise such issue?
According to the following email, fcntl.FD_CLOEXEC was not available in Python 2.2 on Red Hat 7.3 (in 2003): http://communities.mentor.com/community/cs/archives/qmtest/msg00501.html
This issue looks like http://bugs.python.org/issue496171 So it looks like the problem was just that the constant was not exposed properly whereas the OS supports the feature. I guess that FD_CLOEXEC always worked on RedHat. Victor
2013/1/8 Victor Stinner <victor.stinner@gmail.com>:
2013/1/8 Benjamin Peterson <benjamin@python.org>:
2013/1/8 Victor Stinner <victor.stinner@gmail.com>:
2013/1/8 Benjamin Peterson <benjamin@python.org>:
Okay, fair enough, but I really would like it not to ever raise NotImplementedError. Then you would end up having different codepaths for various oses anyway.
So what do you suggest?
If the only systems it doesn't work on is ancient RedHat, that's probably okay.
What do you mean? NotIlmplementedError is acceptable if only rare and/or old OS raise such issue?
We have to draw the line somewhere. People writing Python to run on such systems will already have to be aware of such issues. -- Regards, Benjamin
2013/1/8 Victor Stinner <victor.stinner@gmail.com>:
I don't know platform without this flag.
According to the following email, fcntl.FD_CLOEXEC was not available in Python 2.2 on Red Hat 7.3 (in 2003): http://communities.mentor.com/community/cs/archives/qmtest/msg00501.html I don't know if the constant was not defined in fcntl.h, or if the constant was just not exposed in Python 2.2? Does anyone have such old version of RedHat to test if fcntl.FD_CLOEXEC is available (on a recent version of Python)? -- In the Python issue #12107, I can read: "I realize this bugreport cannot fix 35 years of a bad design decision in linux." http://bugs.python.org/issue12107 Well... Ruby made a brave choice :-) Ruby (2.0?) does set close-on-exec flag on *ALL file descriptors (except 0, 1, 2) *by default*: http://bugs.ruby-lang.org/issues/5041 This change solves the problem of having to close all file descriptor after a fork to run a new program (see closed Python issues #11284 and #8052)... if you are not using C extensions creating file descriptors? Ruby applications relying on passing FD to child processes have to explicitly disable close-on-exec the flag: it was done in Unicorn for example. -- See also the issue discussing the usage of O_CLOEXEC and SOCK_CLOEXEC in the libapr: https://issues.apache.org/bugzilla/show_bug.cgi?id=46425 Victor
participants (4)
-
Benjamin Peterson -
Gregory P. Smith -
Victor Stinner -
Zbigniew Jędrzejewski-Szmek