[Python-Dev] Inherance of file descriptor and handles on Windows (PEP 446)

[Victor Stinner]

The multiprocessing module is an example of use case relying on
inherance of handles. It calls CreateProcess() with
bInheritHandles=TRUE to share a pipe between the manager (parent) and
the worker (child process).

Note: subprocess and multiprocess have their own function to set the
HANDLE_FLAG_INHERIT flag: they use DuplicateHandle(), whereas
SetHandleInformation() could be used (to reuse the existing handle
instead of creating a new handle).

2013/7/24 Victor Stinner <victor.stinner at gmail.com>:
> Python functions open(), os.open() and os.dup() create file
> descriptors with the HANDLE_FLAG_INHERIT flag set (cloexec=False),
> whereas os.pipe() creates 2 file descriptors with the
> HANDLE_FLAG_INHERIT flag unset (cloexec=False, see also issue #4708).
> (...)
> If close_fds=False, handles with the HANDLE_FLAG_INHERIT flag set are
> inherited, but all file descriptors are still closed except 0, 1 and
> 2.

Leaking handles in child processes is also an issue on Windows. Random examples.

http://bugs.python.org/issue17634
"Win32: shutil.copy leaks file handles to child processes"
"Win32's native CopyFile API call doesn't leak file handles to child processes."

http://ghc.haskell.org/trac/ghc/ticket/2650
"The case in which I originally ran into this was
System.Directory.copyFile intermittently reporting a "permission
denied" error for a temp file it was using. I think it was trying to
delete it, but failing because a child process somewhere was hanging
on to the Handle."
According to the issue, GHC calls CreateProcess with
bInheritHandles=TRUE (as Python did until Python 3.2).

http://support.microsoft.com/kb/315939
"This behavior can occur if two threads simultaneously create child
processes and redirect the STD handles through pipes. In this
scenario, there is a race condition during the creation of the pipes
and processes, in which it is possible for one child to inherit file
handles intended for the other child."
=> Python looks to be correct, it uses the (StartupInfo)
"STARTF_USESTDHANDLES" flag

http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6428742
Java is still calling CreateProcess() with bInheritHandles=TRUE which
causes issues like "6347873: (so) Ports opened with
ServerSocketChannel blocks when using Runtime.exec". Interesting
comment: "Submitter has a point.  Very risky to fix." :-/


For the record, the default value of close_fds parameter was also set
to True on Windows by the following changeset:

changeset:   66889:59b43dc34158
user:        Gregory P. Smith <greg at mad-scientist.com>
date:        Tue Dec 14 13:43:30 2010 +0000
files:       Doc/library/subprocess.rst Lib/subprocess.py
Lib/test/test_subprocess.py Misc/NEWS Modules/_posixsubprocess.c
description:
Issue #6559: fix the subprocess.Popen pass_fds implementation. Add a unittest.
Issue #7213: Change the close_fds default on Windows to better match the new
default on POSIX.  True when possible (False if stdin/stdout/stderr are
supplied).

Update the documentation to reflect all of the above.

The changeset was written to fix http://bugs.python.org/issue7213 ;
another example of leak of handles.


> The PEP 466 allows you to control which handles are inherited to child
> process when you use subprocess with close_fds=False. (The subprocess
> parameter should be called "close_handles" on Windows to avoid
> confusion.)

Another advantage of the PEP 446 is that most of the time, the
HANDLE_FLAG_INHERIT flag value can be set atomatically at the creation
of the file descriptor (creation of the handle). It is a nice
enhancement to fight against race conditions with threads ;-) "Most of
the time": for example, socket are inherited by default,
WSA_FLAG_NO_HANDLE_INHERIT flag was only added to Windows Vista.

Victor