[Python-ideas] solving multi-core Python

Eric Snow ericsnowcurrently at gmail.com
Wed Jun 24 08:12:37 CEST 2015

On Mon, Jun 22, 2015 at 9:52 PM, Nick Coghlan <ncoghlan at gmail.com> wrote:
> On 23 June 2015 at 10:03, Chris Angelico <rosuav at gmail.com> wrote:
>> On Tue, Jun 23, 2015 at 9:59 AM, Nathaniel Smith <njs at pobox.com> wrote:
>>> One possibility would be for subinterpreters to copy modules from the
>>> main interpreter -- I guess your average module is mostly dicts,
>>> strings, type objects, and functions; strings and functions are
>>> already immutable and could be shared without copying, and I guess
>>> copying the dicts and type objects into the subinterpreter is much
>>> cheaper than hitting the disk etc. to do a real import. (Though
>>> certainly not free.)
>> FWIW, functions aren't immutable, but code objects are.
> Anything we come up with for optimised data sharing via channels could
> be applied to passing a prebuilt sys.modules dictionary through to
> subinterpreters.
> The key for me is to start from a well-defined "shared nothing"
> semantic model, but then look for ways to exploit the fact that we
> actually *are* running in the same address space to avoid copy
> objects.


> The current reference-counts-embedded-in-the-object-structs memory
> layout also plays havoc with the all-or-nothing page level
> copy-on-write semantics used by the fork() syscall at the operating
> system layer, so some of the ideas we've been considering
> (specifically, those related to moving the reference counter
> bookkeeping out of the object structs themselves) would potentially
> help with that as well (but would also have other hard to predict
> performance consequences).
> There's a reason Eric announced this as the *start* of a research
> project, rather than as a finished proposal - while it seems
> conceptually sound overall, there are a vast number of details to be
> considered that will no doubt hold a great many devils :)

And they keep multiplying! :)


More information about the Python-ideas mailing list