The stable ABI (ergo the limited API) is consistently problematic when it comes to isolating subinterpreters. For example, there are ~120 PyObject exposed by the limited API, which is difficult to deal with in terms of multiple interpreters. (See https://bugs.python.org/issue43503.) After struggling with this problem for a while I had an idea (which I'm pretty sure is just me remembering something someone suggested at some point):
Let's mandate that extensions targeting the limited API cannot be imported in subinterpreters. If we have that constraint in place then nearly all the complex problems go away, since we have substantially more flexibility to adapt the public API. (We'd still be careful about API compatibility but wouldn't have to worry about ABI compatibility.) Note that such extensions could still *create* subinterpreters.
## Concerns
There are two concerns I have that counter the benefits of the idea:
- there may be existing extensions using the limited API *and* subinterpreters
- what if we want to allow "limited" extensions in subinterpreters in the future?
As to the first one, I don't think it's a problem. Given the small sets of people using either of the features, I expect the intersection of those sets will be empty. That said, it would be worth looking at all the extensions we know use subinterpreters, just to be sure.
That leaves the possibility of future support for importing limited API extensions in subinterpreters. At the point we allow them, all the complex problems come back and would have to be dealt with. Here are some possible solutions:
- deal with the problems directly (if possible); the status quo
- introduce a "limited API 2.0"
- specify that a subset of the limited API that extensions must target in order to support for subinterpreters (i.e. "Py_LIMITED_PLUS_SUBINTERPRETERS_API")
Any of these could be done now, later, or only once the support is needed (or never if it never comes up). My preference is to not worry about it for now.
Maybe the limited API + subinterpreters would never be an issue (so doing any of them now would be a waste of effort). I don't know. I bring it up because I recall talk of Cython supporting/targeting the limited API. Also, I hope subinterpreters become a widely used feature and extensions that use the limited API would want to run there. :)
## What Would Need to be Done?
- add the restriction to the docs
- update the import machinery to raise ImportError if an extension
targets the limited API
- how can the import machinery identify such modules?
- we may end up doing this relative to PEP 489 anyway
Anyway, your thoughts on the proposal would be helpful.
-eric