[Python-ideas] solving multi-core Python

[Nick Coghlan]

On 23 June 2015 at 10:03, Chris Angelico <rosuav at gmail.com> wrote:
> On Tue, Jun 23, 2015 at 9:59 AM, Nathaniel Smith <njs at pobox.com> wrote:
>> One possibility would be for subinterpreters to copy modules from the
>> main interpreter -- I guess your average module is mostly dicts,
>> strings, type objects, and functions; strings and functions are
>> already immutable and could be shared without copying, and I guess
>> copying the dicts and type objects into the subinterpreter is much
>> cheaper than hitting the disk etc. to do a real import. (Though
>> certainly not free.)
>
> FWIW, functions aren't immutable, but code objects are.

Anything we come up with for optimised data sharing via channels could
be applied to passing a prebuilt sys.modules dictionary through to
subinterpreters.

The key for me is to start from a well-defined "shared nothing"
semantic model, but then look for ways to exploit the fact that we
actually *are* running in the same address space to avoid copy
objects.

The current reference-counts-embedded-in-the-object-structs memory
layout also plays havoc with the all-or-nothing page level
copy-on-write semantics used by the fork() syscall at the operating
system layer, so some of the ideas we've been considering
(specifically, those related to moving the reference counter
bookkeeping out of the object structs themselves) would potentially
help with that as well (but would also have other hard to predict
performance consequences).

There's a reason Eric announced this as the *start* of a research
project, rather than as a finished proposal - while it seems
conceptually sound overall, there are a vast number of details to be
considered that will no doubt hold a great many devils :)

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia