On May 24, 2017 20:31, "Guido van Rossum" <guido@python.org> wrote:
Hm... Curiously, I've heard a few people at PyCon mention they thought subinterpreters were broken and not useful (and they share the GIL anyways) and should be taken out. So we should at least have clarity on which direction we want to take...

My impression is that the code to support them inside CPython is fine, but they're broken and not very useful in the sense that lots of C extensions don't really support them, so in practice you can't reliably use them to run arbitrary code. Numpy for example definitely has lots of subinterpreter-related bugs, and when they get reported we close them as WONTFIX.

Based on conversations at last year's pycon, my impression is that numpy probably *could* support subinterpreters (i.e. the required apis exist), but none of us really understand the details, it's the kind of problem that requires a careful whole-codebase audit, and a naive approach might make numpy's code slower and more complicated for everyone. (For example, there are lots of places where numpy keeps a little global cache that I guess should instead be per-subinterpreter caches, which would mean adding an extra lookup operation to fast paths.)

Or maybe it'd be fine, but no one is motivated to figure it out, because the other side of the cost/benefit analysis is that almost nobody actually uses subinterpreters. I think the only two projects that do are mod_wsgi and jep [1].

So yeah, the status quo is broken. But there are two possible ways to fix it: IMHO either subinterpreters should be removed *or* they should have some compelling features added to make them actually worth the effort of fixing c extensions to support them. If Eric can pull off this multi-core idea then that would be pretty compelling :-). (And my impression is that the things that break under subinterpreters are essentially the same as would break under any GIL-removal plan.)

The problem is that we don't actually know yet whether the multi-core idea will work, so it seems like a bad time to double down on committing to subinterpreter support and pressuring C extensions to keep up. Eric- do you have a plan written down somewhere? I'm wondering what ‚Äčthe critical path from here to a multi-core proof of concept looks like.


[1] https://github.com/mrj0/jep/wiki/How-Jep-Works