[Python-ideas] solving multi-core Python

[Eric Snow]

On Mon, Jun 22, 2015 at 9:52 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 23 June 2015 at 10:03, Chris Angelico <rosuav at gmail.com> wrote:
>> On Tue, Jun 23, 2015 at 9:59 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>> One possibility would be for subinterpreters to copy modules from the
>>> main interpreter -- I guess your average module is mostly dicts,
>>> strings, type objects, and functions; strings and functions are
>>> already immutable and could be shared without copying, and I guess
>>> copying the dicts and type objects into the subinterpreter is much
>>> cheaper than hitting the disk etc. to do a real import. (Though
>>> certainly not free.)
>>
>> FWIW, functions aren't immutable, but code objects are.
>
> Anything we come up with for optimised data sharing via channels could
> be applied to passing a prebuilt sys.modules dictionary through to
> subinterpreters.
>
> The key for me is to start from a well-defined "shared nothing"
> semantic model, but then look for ways to exploit the fact that we
> actually *are* running in the same address space to avoid copy
> objects.

Exactly.

>
> The current reference-counts-embedded-in-the-object-structs memory
> layout also plays havoc with the all-or-nothing page level
> copy-on-write semantics used by the fork() syscall at the operating
> system layer, so some of the ideas we've been considering
> (specifically, those related to moving the reference counter
> bookkeeping out of the object structs themselves) would potentially
> help with that as well (but would also have other hard to predict
> performance consequences).
>
> There's a reason Eric announced this as the *start* of a research
> project, rather than as a finished proposal - while it seems
> conceptually sound overall, there are a vast number of details to be
> considered that will no doubt hold a great many devils :)

And they keep multiplying! :)

-eric