[Python-ideas] solving multi-core Python

[Devin Jeanpierre]

On Sat, Jun 20, 2015 at 11:31 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> For inter-interpreter communication, the worst case scenario is having
> to rely on a memcpy based message passing system (which would still be
> faster than multiprocessing's serialisation + IPC overhead), but there
> don't appear to be any insurmountable barriers to setting up an object
> ownership based system instead (code that accesses PyObject_HEAD
> fields directly rather than through the relevant macros and functions
> seems to be the most likely culprit for breaking, but I think "don't
> do that" is a reasonable answer there).

The comparison is unfair -- if you can share between subinterpreters
using memcpy, then you can share between processes using just a socket
write, and multiprocessing becomes nearly just as fast.

> Eric's proposal essentially amounts to three things:
>
> 1. Filing off enough of the rough edges of the subinterpreter support
> that we're comfortable giving them a public Python level API that
> other interpreter implementations can reasonably support
> 2. Providing the primitives needed for safe and efficient message
> passing between subinterpreters
> 3. Allowing subinterpreters to truly execute in parallel on multicore machines
>
> All 3 of those are useful enhancements in their own right, which
> offers the prospect of being able to make incremental progress towards
> the ultimate goal of native Python level support for distributing
> across multiple cores within a single process.

Why is that the goal? Whatever faults processes have, those are the
problems, surely not processes in and of themselves, right?

e.g. if the reason we don't like multiprocessed python is extra memory
use, it's memory use we're opposed to. A solution that gives us
parallel threads, but doesn't decrease memory consumption, doesn't
solve anything. The solution has threads that are remarkably like
processes, so I think it's really important to be careful about the
differences and why this solution has the advantage. I'm not seeing
that.

And remember that we *do* have many examples of people using
parallelized Python code in production. Are you sure you're satisfying
their concerns, or whose concerns are you trying to satisfy?

-- Devin