[Python-ideas] PEP 554: Stdlib Module to Support Multiple Interpreters in Python Code

[Nathaniel Smith]

On Thu, Sep 7, 2017 at 11:26 AM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> Hi all,
>
> As part of the multi-core work I'm proposing the addition of the
> "interpreters" module to the stdlib.  This will expose the existing
> subinterpreters C-API to Python code.  I've purposefully kept the API
> simple.  Please let me know what you think.

My concern about this is the same as it was last time -- the work
looks neat, but right now, almost no-one uses subinterpreters
(basically it's Jep and mod_wsgi and that's it?), and therefore many
packages get away with ignoring subinterpreters. Numpy is the one I'm
most familiar with: when we get subinterpreter bugs we close them
wontfix, because supporting subinterpreters properly would require
non-trivial auditing, add overhead for non-subinterpreter use cases,
and benefit a tiny tiny fraction of our users.

If we add a friendly python-level API like this, then we're committing
to this being a part of Python for the long term and encouraging
people to use it, which puts pressure on downstream packages to do
that work... but it's still not clear whether any benefits will
actually materialize.

I've actually argued with the PyPy devs to try to convince them to add
subinterpreter support as part of their experiments with GIL-removal,
because I think the semantics would genuinely be nicer to work with
than raw threads, but they're convinced that it's impossible to make
this work. Or more precisely, they think you could make it work in
theory, but that it would be impossible to make it meaningfully more
efficient than using multiple processes. I want them to be wrong, but
I have to admit I can't see a way to make it work either...

If this is being justified by the multicore use case, and specifically
by the theory that having two interpreters in the same process will
allow for more efficient communication than two interpreters in two
different processes, then... why should we believe that that's
actually possible? I want your project to succeed, but if it's going
to fail then it seems better if it fails before we commit to exposing
new APIs.

-n

-- 
Nathaniel J. Smith -- https://vorpus.org