[New-bugs-announce] [issue28906] Can't inherit sockets with multiprocessing on Windows

[Preston Landers]

New submission from Preston Landers:

I'm porting a Python 2.6 based application to Python 3.6. This app uses a customized version of the "flup" package to do FastCGI services on Windows using the multiprocessing package. This requires a few sockets to be inherited across processes - the main FastCGI protocol socket plus a control socket.

In Python 2.6, the `socket.from_fd` function was not available on Windows. However I patched Python's socketmodule.c to provide that function using DuplicateHandle. In Python 2.6's version of multiprocessing it spawned a process with CreateProcess and bInheritHandles=True. This worked well for me.

Now I'm trying to get this working after moving from Python 2.6 to 3.6 (currently using 3.6.0b4). Fortunately, the socket module now has a working `socket.from_fd` on Windows so I no longer have to patch that. Unfortunately for me, though, the multiprocessing module now calls CreateProcess with bInheritHandles=False. This causes my scenario to fail.

Here's a short test script which illustrates this problem:
https://gist.github.com/Preston-Landers/712fee10fb557cf0b5592b57561a7c08

If you run with an unpatched multiprocessing, it will fail with an error like:

OSError: [WinError 10038] An operation was attempted on something that is not a socket

If you patch multiprocessing to set bInheritHandles=True this now works. (Change is in popen_spawn_win32.py where it does _winapi.CreateProcess.)

I'm sure there's a good reason for that change in multiprocessing, whether for security or for unrelated/undesired file handles being passed.
https://www.python.org/dev/peps/pep-0446/#inheritance-of-file-descriptors-on-windows

However it does break my scenario and I don't see a way to tell multiprocessing to allow certain handles to be inherited. The docs for multiprocessing say "In particular, unnecessary file descriptors and handles from the parent process will not be inherited." It would be nice to have a way to tell it that my sockets are "necessary." You would think that calling socket.set_inheritable(True) would do it. In fact you must do that, but you must also pass bInheritHandles=True to CreateProcess for it to actually work. There doesn't seem to be a way to pass through an argument to multiprocessing to tell it to set this flag.

I do realize I could be going about this completely wrong, though. But right now it looks like my immediate options are:

a) Go ahead and patch my copy of popen_spawn_win32.py to allow inherited handles despite other possible risks.

b) Try to rewrite things to not use multiprocessing at all and directly spawn my processes instead. That's not attractive because multiprocessing otherwise does what I need to do.

Are there any other options I'm missing? Maybe some way to duplicate the socket on the other end without relying on CreateProcess with bInheritHandles=True?

Otherwise, I guess I'm asking for an option to be made available in multiprocessing to allow handles to be inherited on Windows.

----------
components: Library (Lib)
messages: 282723
nosy: planders
priority: normal
severity: normal
status: open
title: Can't inherit sockets with multiprocessing on Windows
type: behavior
versions: Python 3.6

_______________________________________
Python tracker <report at bugs.python.org>
<http://bugs.python.org/issue28906>
_______________________________________