On Thu, May 25, 2017 at 12:01 PM, Eric Snow email@example.com wrote:
More significantly, I genuinely believe that isolated interpreters in the same process is a tool that many people will find extremely useful and will help the Python community. Consequently, exposing subinterpreters in the stdlib would result in a stronger incentive for folks to fix the known bugs and find a solution to the challenges for extension modules.
I feel like the most effective incentive would be to demonstrate how useful they are first? If we do it in the other order, then there's a risk that cpython does provide an incentive, but it's of the form "this thing doesn't actually accomplish anything useful yet, but it got mentioned in whats-new-in-3.7 and now angry people are yelling at me in my bug tracker for not 'fixing' my package, so I have to do a bunch of pointless work to shut them up". This tends to leave bad feelings all around.
I do get that this is a tricky chicken-and-egg situation: currently subinterpreters don't work very well, so no-one writes cool applications using them, so no-one bothers to make them work better. And I share the general intuition that this is a powerful tool that probably has some kind of useful applications. But I can't immediately name any such applications, which makes me nervous :-). The obvious application is your multi-core Python idea, and I think that would convince a lot of people; in general I'm enthusiastic about the idea of extending Python's semantics to enable better parallelism. But I just re-read your blog post and some of the linked thread, and it's not at all clear to me how you plan to solve the refcounting and garbage collection problems that will arise once you have objects that are shared between multiple subinterpreters and no GIL. Which makes it hard for me to make a case to the other numpy devs that it's worth spending energy on this now, to support a feature that might or might not happen in the future, especially if angry shouty people start joining the conversation.
Does that make sense? I want the project to succeed, and if one of the conditions for that is getting buy-in from the community of C extension developers then it seems important to have a good plan for navigating the incentives tightrope.