On 4/20/2020 7:33 PM, Nathaniel Smith wrote:
On Mon, Apr 20, 2020 at 4:26 PM Edwin Zimmerman
On 4/20/2020 6:30 PM, Nathaniel Smith wrote:
We already have robust support for threads for low-isolation and subprocesses for high-isolation. Can you name some use cases where neither of these are appropriate and you instead want an in-between isolation – like subprocesses, but more fragile and with odd edge cases where state leaks between them? I don't know if this has been mentioned before or not, but I'll bring it up now: massively concurrent networking code on Windows. Socket connections could be passed off from the main interpreter to sub-interpreters for concurrent processing that simply isn't possible with the global GIL (provided the GIL actually becomes per-interpreter). On *nix you can fork, this would give CPython on Windows similar capabilities. Both Windows and Unix have APIs for passing sockets between related or unrelated processes -- no fork needed. On Windows, it's exposed as the socket.share method: https://docs.python.org/3/library/socket.html#socket.socket.share
The APIs for managing and communicating between processes are definitely not the most obvious or simplest to use, but they're very mature and powerful, and it's a lot easier to wrap them up in a high-level API than it is to effectively reimplement process separation from scratch inside CPython.
-n +1 on not being most obvious or simplest to use. Not only that, but to use it you have to write Windows-specific code. PEP 554 would provide a uniform, cross-platform capability that I would choose any day over a random pile of os-specific hacks. --Edwin