[Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

[Eric Snow]

On Wed, May 24, 2017 at 8:30 PM, Guido van Rossum <guido at python.org> wrote:
> Hm... Curiously, I've heard a few people at PyCon

I'd love to get in touch with them and discuss the situation.  I've
spoken with Graham Dumpleton on several occasions about
subinterpreters and what needs to be fixed.

> mention they thought subinterpreters were broken

There are a number of related long-standing bugs plus a few that I
created in the last year or two.  I'm motivated to get these resolved
so that the multi-core Python project can take full advantage of
subinterpreters without worry.

As well, there are known limitations to using extension modules in
subinterpreters.  However, only extension modules that rely on process
globals (rather than leveraging PEP 384, etc.) are affected, and we
can control for that more carefully using the protocol introduced by
PEP 489.

There isn't anything broken about the concept or design of
subinterpreters in CPython that I'm aware of.

> and not useful (and they share the GIL anyways)

I'll argue that their usefulness has been limited by lack of exposure
in the stdlib. :)  Furthermore, I'm finding them extremely useful as
the vehicle for the multi-core Python project.

> and should be taken out. So we should at least have clarity on which
> direction we want to take...

I'd definitely appreciate a firm commitment that they are not getting
removed as I don't want to spend a ton of time on the project just to
have the effort made irrelevant. :)  Also, I'd be surprised if there
were sufficient merit to removing support for subinterpreters, since
there is very little machinery just for that feature.  Instead, it
leverages components of CPython that are there for other valid
reasons.  So I do not consider subinterpreters to currently add any
significant burden to maintenance or development of the code base.
Regardless, exposing the low-level _subinterpreters module should help
us iron out bugs and API, as Nick pointed out.

-eric