On Sat, Jun 20, 2015 at 11:31 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:
For inter-interpreter communication, the worst case scenario is having to rely on a memcpy based message passing system (which would still be faster than multiprocessing's serialisation + IPC overhead), but there don't appear to be any insurmountable barriers to setting up an object ownership based system instead (code that accesses PyObject_HEAD fields directly rather than through the relevant macros and functions seems to be the most likely culprit for breaking, but I think "don't do that" is a reasonable answer there).
The comparison is unfair -- if you can share between subinterpreters using memcpy, then you can share between processes using just a socket write, and multiprocessing becomes nearly just as fast.
Eric's proposal essentially amounts to three things:
1. Filing off enough of the rough edges of the subinterpreter support that we're comfortable giving them a public Python level API that other interpreter implementations can reasonably support 2. Providing the primitives needed for safe and efficient message passing between subinterpreters 3. Allowing subinterpreters to truly execute in parallel on multicore machines
All 3 of those are useful enhancements in their own right, which offers the prospect of being able to make incremental progress towards the ultimate goal of native Python level support for distributing across multiple cores within a single process.
Why is that the goal? Whatever faults processes have, those are the problems, surely not processes in and of themselves, right? e.g. if the reason we don't like multiprocessed python is extra memory use, it's memory use we're opposed to. A solution that gives us parallel threads, but doesn't decrease memory consumption, doesn't solve anything. The solution has threads that are remarkably like processes, so I think it's really important to be careful about the differences and why this solution has the advantage. I'm not seeing that. And remember that we *do* have many examples of people using parallelized Python code in production. Are you sure you're satisfying their concerns, or whose concerns are you trying to satisfy? -- Devin