On Mon, 20 Apr 2020 14:22:03 -0600 Eric Snow <ericsnowcurrently@gmail.com> wrote:
The appeal of the PEP for experimentations is multiple: 1) ability to concurrently run independent execution environments without spawning child processes (which on some platforms and in some situations may not be very desirable: for example on Windows where the cost of spawning is rather high; also, child processes may crash, and sometimes it is not easy for the parent to recover, especially if a synchronization primitive is left in an unexpected state) 2) the potential for parallelizing CPU-bound pure Python code in a single process, if a per-interpreter GIL is finally implemented 3) easier support for sharing large data between separate execution environments, without the hassle of setting up shared memory or the fragility of relying on fork() semantics
(and as I said, I hope people find other applications)
These are covered in the PEP, though not together in the rationale, etc. Should I add explicit mention of experimentation as a motivation in the abstract or rationale sections? Would you like me to add a dedicated paragraph/section covering experimentation?
I was mostly exposing my thought process here :-) IOW, you don't have to do anything, except if you think that would be helpful.
As for the argument that we already have asyncio and several other packages, I actually think that combining these different concurrency mechanisms would be interesting complex applications (such as distributed systems). For that, however, I think the PEP as currently written is a bit lacking, see below.
Yeah, that would be interesting. What in particular will help make subinterpreters and asyncio more cooperative?
Readiness callbacks would help wrangle any kind of asynchronous / event-driven framework around subinterpreters.
* In the same vein, I think channels should allow adding readiness callbacks (that are called whenever a channel becomes ready for sending or receiving, respectively). This would make it easy to plug them into an event loop or other concurrency systems (such as Future-based concurrency). Note that each interpreter "associated" with a channel should be able to set its own readiness callback: so one callback per Python object representing the channel, but potentially multiple callbacks for the underlying channel primitive.
Would this be as useful if we have buffered channels? It sounds like you wanted one or the other but not both.
Both are useful at somewhat different levels (though as Greg said, if you have readiness callbacks, you can probably cook up a buffering layer using them). Especially, readiness callbacks (or some other form of push notification) are desirable for reasonable interaction with an event loop.
Of course, I hope these are all actionable before beta1 :-) If not, here is my preferential priority list:
* High priority: fix association timing * High priority: either buffering /or/ readiness callbacks * Middle priority: get_main() /or/ is_main()
These should be doable for beta1 since they re either trivial or already done. :)
Great :-) Best regards Antoine.