[Python-Dev] Proposal for a new function "open_noinherit" to avoid problems with subprocesses and security risks
Henning von Bargen
henning.vonbargen at arcor.de
Fri Jun 22 23:40:04 CEST 2007
I'd like to propose a new function "open_noinherit"
or maybe even a new mode flag "n" for the builtin "open"
(see footnote for the names).
The new function should work exactly like the builtin "open", with one
The open file is not inherited to any child processes
(whereas files opened with "open" will be inherited).
The new function can be implemented (basically) using
os.O_NOINHERIT on MS Windows
resp. fcntl / FD_CLOEXEC on Posix.
I will post a working Python implementation next week.
There are five reasons for the proposal:
1) The builtin "open" causes unexpected problems in conjunction with
in particular in multi-threaded programs.
It can cause file permission errors in the subprocess or in the current
On Microsoft Windows, some of the possible file permission errors are
documented by Microsoft (thus very few programs written for Windows
2) Inheriting open file handles to subprocesses is a security risk.
3) For the developer, finding "cause and effect" is *very* hard, in
multi-threaded programs, when the errors occur only in race-conditions.
4) The problems arise in some of the standard library modules as well,
5) Very few developers are aware of the possible problems.
As a work-around, one can replace open with
os.fdopen (os.open (..., + os.O_NOINHERIT), ... )
on Windows, but that's really ugly, hard to read,
may raise a different exception than open (IOError instead of OSError),
and needs careful work to take platform-specific code into account
Here is a single-threaded example to demonstrate the effect:
outf = open ("blah.tmp", "wt")
subprocess.Popen("notepad.exe") # or whatever program you like, but
# It must be a program that does not exit immediately!
# Now the subprocess has inherited the open file handle
# We can still write:
outf.write ("Hello world!\n")
# But we can not rename the file (at least on Windows)
os.rename ("blah.tmp", "blah.txt")
# this fails with OSError: [Errno 13] Permission denied
# Similar problems with other file operations on non-Windows platforms.
Ok, in this little program one can see what is going wrong easily.
But what if the subprocess exits very quickly?
Then perhaps you see the OSError, perhaps not - depending on the process
of your operation system.
In a commercial multi-theaded daemon application, the error only occured
under heavy load and was hard to reproduce - and it was even harder to find
That's because cause and effect were in two different threads in two
parts of the program:
- Thread A opens a file and starts to write data
- Thread B starts a subprocess (which inherits the file handle from thread
- Thread A continues writing to the file and closes it.
- And now it's a race condition:
- a) Thread A wants to rename the file - b) the subprocess exits.
If a) is first: Error, if b) is first: no error.
To make things more complicated, even two subprocesses can disturb each
The new function should be implemented in C ideally, because the GIL could
prevent a thread-switch between os.open and the fcntl.F_SETFD call.
Note that the problem described here arises not only for files, but for
See bug 1222790: SimpleXMLRPCServer does not set FD_CLOEXEC
Once there is an easy-to-use, platform-independent, documented builtin
"open_noinherit" (or a new mode flag for "open"), the standard library
be considered. For each occurence of "open" or "file", it should be
if it necessary to inherit the file to subprocesses. If not, it should be
One example is shutil.filecopy, where open_noiherit should be used instead
The socket module is another candidate, I think - but I'm not sure about
A nice effect of using "open_noinherit" is that - in many cases - one no
needs to speficy close_fds = True when calling subprocess.Popen.
[Note that close_fds is *terribly* slow if MAX_OPEN_FILES is "big", e.g.
see bug 1663329]
While writing this mail, at least 3 times I typed "nonherit" instead of
So maybe someone can propose a better name?
Or a new mode flag character could be "p" (like "private" or "protected").
More information about the Python-Dev