[Python-ideas] Exposing CPython's subinterpreter C-API in the stdlib.

[Petr Viktorin]

On 05/25/2017 09:01 PM, Eric Snow wrote:
> On Thu, May 25, 2017 at 11:19 AM, Nathaniel Smith <njs at pobox.com> wrote:
>> My impression is that the code to support them inside CPython is fine, but
>> they're broken and not very useful in the sense that lots of C extensions
>> don't really support them, so in practice you can't reliably use them to run
>> arbitrary code. Numpy for example definitely has lots of
>> subinterpreter-related bugs, and when they get reported we close them as
>> WONTFIX.
>>
>> Based on conversations at last year's pycon, my impression is that numpy
>> probably *could* support subinterpreters (i.e. the required apis exist), but
>> none of us really understand the details, it's the kind of problem that
>> requires a careful whole-codebase audit, and a naive approach might make
>> numpy's code slower and more complicated for everyone. (For example, there
>> are lots of places where numpy keeps a little global cache that I guess
>> should instead be per-subinterpreter caches, which would mean adding an
>> extra lookup operation to fast paths.)
> 
> Thanks for pointing this out.  You've clearly described probably the
> biggest challenge for folks that try to use subinterpreters.  PEP 384
> was meant to help with this, but seems to have fallen short.  PEP 489
> can help identify modules that profess subinterpreter support, as well
> as facilitating future extension module helpers to deal with global
> state.  However, I agree that *right now* getting extension modules to
> reliably work with subinterpreters is not easy enough.  Furthermore,
> that won't change unless there is sufficient benefit tied to
> subinterpreters, as you point out below.

PEP 489 was a first step; the work is not finished. The next step is 
solving a major reason people are using global state in extension 
modules: per-module state isn't accessible from all the places it should 
be, namely in methods of classes. In other words, I don't think Python 
is ready for big projects like Numpy to start properly supporting 
subinterpreters.

The work on fixing this has stalled, but it looks like I'll be getting 
back on track.
Discussions about this are on the import-sig list, reach out there if 
you'd like to help.