[Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

Nathaniel Smith njs at pobox.com
Fri May 26 03:04:41 EDT 2017

On Thu, May 25, 2017 at 12:01 PM, Eric Snow <ericsnowcurrently at gmail.com> wrote:
> More significantly, I genuinely believe that isolated
> interpreters in the same process is a tool that many people will find
> extremely useful and will help the Python community.  Consequently,
> exposing subinterpreters in the stdlib would result in a stronger
> incentive for folks to fix the known bugs and find a solution to the
> challenges for extension modules.

I feel like the most effective incentive would be to demonstrate how
useful they are first? If we do it in the other order, then there's a
risk that cpython does provide an incentive, but it's of the form
"this thing doesn't actually accomplish anything useful yet, but it
got mentioned in whats-new-in-3.7 and now angry people are yelling at
me in my bug tracker for not 'fixing' my package, so I have to do a
bunch of pointless work to shut them up". This tends to leave bad
feelings all around.

I do get that this is a tricky chicken-and-egg situation: currently
subinterpreters don't work very well, so no-one writes cool
applications using them, so no-one bothers to make them work better.
And I share the general intuition that this is a powerful tool that
probably has some kind of useful applications. But I can't immediately
name any such applications, which makes me nervous :-). The obvious
application is your multi-core Python idea, and I think that would
convince a lot of people; in general I'm enthusiastic about the idea
of extending Python's semantics to enable better parallelism. But I
just re-read your blog post and some of the linked thread, and it's
not at all clear to me how you plan to solve the refcounting and
garbage collection problems that will arise once you have objects that
are shared between multiple subinterpreters and no GIL. Which makes it
hard for me to make a case to the other numpy devs that it's worth
spending energy on this now, to support a feature that might or might
not happen in the future, especially if angry shouty people start
joining the conversation.

Does that make sense? I want the project to succeed, and if one of the
conditions for that is getting buy-in from the community of C
extension developers then it seems important to have a good plan for
navigating the incentives tightrope.


Nathaniel J. Smith -- https://vorpus.org

More information about the Python-ideas mailing list