On Mon, Apr 20, 2020 at 5:36 PM Edwin Zimmerman
On 4/20/2020 7:33 PM, Nathaniel Smith wrote:
On Mon, Apr 20, 2020 at 4:26 PM Edwin Zimmerman
On 4/20/2020 6:30 PM, Nathaniel Smith wrote:
We already have robust support for threads for low-isolation and subprocesses for high-isolation. Can you name some use cases where neither of these are appropriate and you instead want an in-between isolation – like subprocesses, but more fragile and with odd edge cases where state leaks between them? I don't know if this has been mentioned before or not, but I'll bring it up now: massively concurrent networking code on Windows. Socket connections could be passed off from the main interpreter to sub-interpreters for concurrent processing that simply isn't possible with the global GIL (provided the GIL actually becomes per-interpreter). On *nix you can fork, this would give CPython on Windows similar capabilities. Both Windows and Unix have APIs for passing sockets between related or unrelated processes -- no fork needed. On Windows, it's exposed as the socket.share method: https://docs.python.org/3/library/socket.html#socket.socket.share
The APIs for managing and communicating between processes are definitely not the most obvious or simplest to use, but they're very mature and powerful, and it's a lot easier to wrap them up in a high-level API than it is to effectively reimplement process separation from scratch inside CPython.
-n +1 on not being most obvious or simplest to use. Not only that, but to use it you have to write Windows-specific code. PEP 554 would provide a uniform, cross-platform capability that I would choose any day over a random pile of os-specific hacks.
I mean, sure, if you've decided to build one piece of hypothetical software well and another badly, then the good one will be better than the bad one, but that doesn't really say much, does it? In real life, I don't see how it's possible to get PEP 554's implementation to the point where it works reliably and robustly – i.e., I just don't think the promises the PEP makes can actually be fulfilled. And even if you did, it would still be several orders of magnitude easier to build a uniform, robust, cross-platform API on top of tools like socket.share than it would be to force changes on every C extension. PEP 554 is hugely expensive; you can afford a *lot* of careful systems engineering while still coming in way under that budget. -n -- Nathaniel J. Smith -- https://vorpus.org