[Python-ideas] solving multi-core Python

[Devin Jeanpierre]

It's worthwhile to consider fork as an alternative.  IMO we'd get a
lot out of making forking safer, easier, and more efficient. (e.g.
respectively: adding an atfork registration mechanism; separating out
the bits of multiprocessing that use pickle from those that don't;
moving the refcount to a separate page, or allowing it to be frozen
prior to a fork.)

It sounds to me like this approach would use more memory than either
regular threaded code or forking, so its main advantages are being
cross-platform and less bug-prone. Is that right?


Note: I don't count the IPC cost of forking, because at least on
linux, any way to efficiently share objects between independent
interpreters in separate threads can also be ported to independent
interpreters in forked subprocesses, and *should* be.

See also: multiprocessing.Value/Array. This is probably a good
opportunity for that unification you mentioned. :)

On Sat, Jun 20, 2015 at 3:04 PM, Yury Selivanov <yselivanov.ml at gmail.com> wrote:
> On 2015-06-20 5:42 PM, Eric Snow wrote:
>> * only allow immutable objects to be shared between subinterpreters
>
> Even if this is the only thing we have -- an efficient way
> for sharing immutable objects (such as bytes, strings, ints,
> and, stretching the definition of immutable, FDs) that will
> allow us to do a lot.

+1, this has a lot of utility, and can be extended naturally to other
types and circumstances.

-- Devin