[Python-ideas] solving multi-core Python

[Nick Coghlan]

On 21 June 2015 at 10:41, Chris Angelico <rosuav at gmail.com> wrote:
> On Sun, Jun 21, 2015 at 7:42 AM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
>> * disallow forking within subinterpreters
>
> I love the idea as a whole (if only because the detractors can be told
> "Just use subinterpreters, then you get concurrency"), but this seems
> like a tricky restriction. That means no subprocess.Popen, no shelling
> out to other applications. And I don't know what of other restrictions
> might limit any given program. Will it feel like subinterpreters are
> "write your code according to these tight restrictions and it'll
> work", or will it be more of "most programs will run in parallel just
> fine, but there are a few things to be careful of"?

To calibrate expectations appropriately, it's worth thinking about the
concept of Python level subinterpreter support as being broadly
comparable to the JavaScript concept of web worker threads. mod_wsgi's
use of the existing CPython specific subinterpreter support when
embedding CPython in Apache httpd means we already know
subinterpreters largely "just work" in the absence of low level C
shenanigans in extension modules, but we also know keeping
subinterpreters clearly subordinate to the main interpreter simplifies
a number of design and implementation aspects (just as having a main
thread simplified various aspects of the threading implementation),
and that there will likely be things the main interpreter can do that
subinterpreters can't.

A couple of possible examples:

* as Eric noted, we don't know yet if we'll be able to safely let
subinterpreters launch subprocesses (especially via fork)
* there may be restrictions on some extension modules that limit them
to "main interpreter only" (e.g. if the extension module itself isn't
thread-safe, then it will need to remain fully protected by the GIL)

The analogous example with web workers is the fact that they don't
have any access to the window object, document object or parent object
in the browser DOM.

Cheers,
Nick.

-- 
Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia