Can we explicitly specify that the limited API can't be used in subinterpreters?
The stable ABI (ergo the limited API) is consistently problematic when it comes to isolating subinterpreters. For example, there are ~120 PyObject exposed by the limited API, which is difficult to deal with in terms of multiple interpreters. (See https://bugs.python.org/issue43503.) After struggling with this problem for a while I had an idea (which I'm pretty sure is just me remembering something someone suggested at some point):
Let's mandate that extensions targeting the limited API cannot be imported in subinterpreters. If we have that constraint in place then nearly all the complex problems go away, since we have substantially more flexibility to adapt the public API. (We'd still be careful about API compatibility but wouldn't have to worry about ABI compatibility.) Note that such extensions could still *create* subinterpreters.
## Concerns
There are two concerns I have that counter the benefits of the idea:
- there may be existing extensions using the limited API *and* subinterpreters
- what if we want to allow "limited" extensions in subinterpreters in the future?
As to the first one, I don't think it's a problem. Given the small sets of people using either of the features, I expect the intersection of those sets will be empty. That said, it would be worth looking at all the extensions we know use subinterpreters, just to be sure.
That leaves the possibility of future support for importing limited API extensions in subinterpreters. At the point we allow them, all the complex problems come back and would have to be dealt with. Here are some possible solutions:
- deal with the problems directly (if possible); the status quo
- introduce a "limited API 2.0"
- specify that a subset of the limited API that extensions must target in order to support for subinterpreters (i.e. "Py_LIMITED_PLUS_SUBINTERPRETERS_API")
Any of these could be done now, later, or only once the support is needed (or never if it never comes up). My preference is to not worry about it for now.
Maybe the limited API + subinterpreters would never be an issue (so doing any of them now would be a waste of effort). I don't know. I bring it up because I recall talk of Cython supporting/targeting the limited API. Also, I hope subinterpreters become a widely used feature and extensions that use the limited API would want to run there. :)
## What Would Need to be Done?
- add the restriction to the docs
- update the import machinery to raise ImportError if an extension
targets the limited API
- how can the import machinery identify such modules?
- we may end up doing this relative to PEP 489 anyway
Anyway, your thoughts on the proposal would be helpful.
-eric
On 16. 03. 21 17:38, Eric Snow wrote:
The stable ABI (ergo the limited API) is consistently problematic when it comes to isolating subinterpreters. For example, there are ~120 PyObject exposed by the limited API, which is difficult to deal with in terms of multiple interpreters. (See https://bugs.python.org/issue43503.) After struggling with this problem for a while I had an idea (which I'm pretty sure is just me remembering something someone suggested at some point):
Let's mandate that extensions targeting the limited API cannot be imported in subinterpreters. If we have that constraint in place then nearly all the complex problems go away, since we have substantially more flexibility to adapt the public API. (We'd still be careful about API compatibility but wouldn't have to worry about ABI compatibility.) Note that such extensions could still *create* subinterpreters.
Hi, The limited API is a subset of the full C-API. I don't understand how you want to limit the limited API but allow the full API.
Also, extensions using the limited API can currently be loaded in subinterpreters. Disallowing this would break backwards compatibility.
## Concerns
There are two concerns I have that counter the benefits of the idea:
- there may be existing extensions using the limited API *and* subinterpreters
- what if we want to allow "limited" extensions in subinterpreters in the future?
As to the first one, I don't think it's a problem. Given the small sets of people using either of the features, I expect the intersection of those sets will be empty. That said, it would be worth looking at all the extensions we know use subinterpreters, just to be sure.
That leaves the possibility of future support for importing limited API extensions in subinterpreters. At the point we allow them, all the complex problems come back and would have to be dealt with. Here are some possible solutions:
- deal with the problems directly (if possible); the status quo
- introduce a "limited API 2.0"
Yes, please. I would very much prefer this (except in my head it's not 2.0 but "Stable ABI 4.0", as the current versions follow Python's).
- specify that a subset of the limited API that extensions must target in order to support for subinterpreters (i.e. "Py_LIMITED_PLUS_SUBINTERPRETERS_API")
Right, and let's call it "Stable ABI 4.0" :)
Before that, we can even deprecate (and eventually remove) problematic API from the limited set -- as long as we don't break ABI before 4.0. For details see PEP 652 (currently waiting for SC approval).
Any of these could be done now, later, or only once the support is needed (or never if it never comes up). My preference is to not worry about it for now.
Maybe the limited API + subinterpreters would never be an issue (so doing any of them now would be a waste of effort). I don't know. I bring it up because I recall talk of Cython supporting/targeting the limited API. Also, I hope subinterpreters become a widely used feature and extensions that use the limited API would want to run there. :)
## What Would Need to be Done?
- add the restriction to the docs
- update the import machinery to raise ImportError if an extension targets the limited API
- how can the import machinery identify such modules?
- we may end up doing this relative to PEP 489 anyway
Anyway, your thoughts on the proposal would be helpful.
Please also look at my idea here: https://github.com/ericsnowcurrently/multi-core-python/issues/71
This topic is also discussed at: https://bugs.python.org/issue43503
- specify that a subset of the limited API that extensions must target in order to support for subinterpreters (i.e. "Py_LIMITED_PLUS_SUBINTERPRETERS_API")
Right, and let's call it "Stable ABI 4.0" :)
Since it would be a subset of the API, I understand it "just" would a subset of the ABI and so that the current stable ABI can be used. It's just that subinterpreter would not use some symbols like "PyLong_Type" (static types).
Victor
Night gathers, and now my watch begins. It shall not end until my death.
On Tue, Mar 16, 2021 at 1:27 PM Petr Viktorin <encukou@gmail.com> wrote:
On 16. 03. 21 17:38, Eric Snow wrote:
...
The limited API is a subset of the full C-API. I don't understand how you want to limit the limited API but allow the full API.
Right. This would require that the limited API and the full API diverge. The limited API would stay as-is. In the public API we would introduce a per-interpreter lookup function (and perhaps replace the public PyObject variables with macros that call lookup functions, more or less).
Again, this is viable only if limited API extensions are only allowed to be imported in the main interpreter. The benefit is that we can move forward toward a per-interpreter GIL while preserving the limited API without a disruptive/broad change in the repo and without breaking existing extensions. The main downside is that the limited API diverges, which I consider okay in this limited and focused case.
Also, extensions using the limited API can currently be loaded in subinterpreters. Disallowing this would break backwards compatibility.
The big question is if there are extensions out there that both define Py_LIMITED_API *and* are used in subinterpreters. It's worth checking but I expect the answer is that there are none.
If that's the case then we're fine. Otherwise there's a viable solution if we want to move ahead with the restriction: add a "legacy" mode for subinterpreters that preserves the current status quo (much like you suggest in https://github.com/ericsnowcurrently/multi-core-python/issues/71).
- introduce a "limited API 2.0"
Yes, please. I would very much prefer this (except in my head it's not 2.0 but "Stable ABI 4.0", as the current versions follow Python's).
- specify that a subset of the limited API that extensions must target in order to support for subinterpreters (i.e. "Py_LIMITED_PLUS_SUBINTERPRETERS_API")
Right, and let's call it "Stable ABI 4.0" :)
Before that, we can even deprecate (and eventually remove) problematic API from the limited set -- as long as we don't break ABI before 4.0. For details see PEP 652 (currently waiting for SC approval).
There are two distinct things involved here: ABI compatibility and API compatibility. I've started another thread about how those relate to the limited API. [1] Basically, I think such an updated limited API should not be tied to the stable ABI like the current one is (and deserves a different name than Py_LIMITED_API). See the thread for more discussion.
(I'm also wondering if we even need the stable ABI any more. [2] Many points in PEP 652 would still apply though.)
Anyway, your thoughts on the proposal would be helpful.
Please also look at my idea here: https://github.com/ericsnowcurrently/multi-core-python/issues/71
Ah, I missed this. I agree (though have a different idea on some of the details).
-eric
[1] https://mail.python.org/archives/list/capi-sig@python.org/thread/DX2HNPMJLIF... [2] https://mail.python.org/archives/list/capi-sig@python.org/thread/ZMNNLKK5NUW...
On Wed, Mar 17, 2021 at 12:11 PM Eric Snow <ericsnowcurrently@gmail.com> wrote:
If that's the case then we're fine. Otherwise there's a viable solution if we want to move ahead with the restriction: add a "legacy" mode for subinterpreters that preserves the current status quo (much like you suggest in https://github.com/ericsnowcurrently/multi-core-python/issues/71).
To elaborate:
- introduce a field to PyConfig (or maybe PyInterpreterState directly)
that indicates the interpreter should operate in "legacy" mode
- this could be a boolean or an int flag, a la PyTypeObject.tp_flags (or maybe an enum field, with members INTERP_LEGACY and INTERP_RETRICTED)
- if on PyConfig it would surface as PyInterpreterState.config.legacy (or "mode" or whatever)
- legacy mode would preserve the current status quo:
- each subinterpreter in legacy mode would use the main interpreter's GIL instead of its own
- extensions built with the limited API would be allowed in subinterpreters
- without legacy mode, importing a limited API extension would result in an ImportError
- there would be a new _PyInterpreterState_NewWIthConfig(PyConfig *) (or similar; internal-only for now) that would allow setting the mode
- the existing PyInterpreterState_New() would wrap it, using legacy mode
- the interpreters module (PEP 554) would *not* use legacy mode
This would take care of compatibility with the existing limited API.
FWIW, something like PyInterpreterState.mode has been part of the plan all along. Serendipitously, it happens to fit here. :) Also, restricted mode could include other things, like requiring PEP 489.
-eric
On 17. 03. 21 19:12, Eric Snow wrote:
On Wed, Mar 17, 2021 at 12:11 PM Eric Snow <ericsnowcurrently@gmail.com> wrote:
If that's the case then we're fine. Otherwise there's a viable solution if we want to move ahead with the restriction: add a "legacy" mode for subinterpreters that preserves the current status quo (much like you suggest in https://github.com/ericsnowcurrently/multi-core-python/issues/71).
To elaborate:
- introduce a field to PyConfig (or maybe PyInterpreterState directly) that indicates the interpreter should operate in "legacy" mode
- this could be a boolean or an int flag, a la PyTypeObject.tp_flags (or maybe an enum field, with members INTERP_LEGACY and INTERP_RETRICTED)
- if on PyConfig it would surface as PyInterpreterState.config.legacy (or "mode" or whatever)
- legacy mode would preserve the current status quo:
- each subinterpreter in legacy mode would use the main interpreter's GIL instead of its own
- extensions built with the limited API would be allowed in subinterpreters
- without legacy mode, importing a limited API extension would result in an ImportError
- there would be a new _PyInterpreterState_NewWIthConfig(PyConfig *) (or similar; internal-only for now) that would allow setting the mode
- the existing PyInterpreterState_New() would wrap it, using legacy mode
- the interpreters module (PEP 554) would *not* use legacy mode
This would take care of compatibility with the existing limited API.
FWIW, something like PyInterpreterState.mode has been part of the plan all along. Serendipitously, it happens to fit here. :) Also, restricted mode could include other things, like requiring PEP 489.
That sounds like a great idea!
It would solve the ABI problem, leaving us with only the API problem: PyLong_Type & friends need to become pointers. This will need the extensions' code to change, and IMO that means we need to introduce PyInt_GetType(), deprecate the static objects, and flag extensions that use the static objects as also not usable with subinterpreters.
From the earlier mail:
The big question is if there are extensions out there that both define Py_LIMITED_API *and* are used in subinterpreters. It's worth checking but I expect the answer is that there are none.
Python's big enough that I can tell you with near certainty that yes, there are :)
On Thu, 18 Mar 2021 at 04:11, Eric Snow <ericsnowcurrently@gmail.com> wrote:
The big question is if there are extensions out there that both define Py_LIMITED_API *and* are used in subinterpreters. It's worth checking but I expect the answer is that there are none.
I'd expect the answer to be "Yes, absolutely".
cryptography, for example, releases abi3 wheels for Windows, Linux, and Mac OS X.
I'm sure that module is frequently used under mod_wsgi in Apache, and it wouldn't surprise me to learn that there are Ceph plugins using it as well.
(I'm also wondering if we even need the stable ABI any more. [2] Many points in PEP 652 would still apply though.)
Yes, we still need it. We don't hear people talking about it much *because it works*.
Try to take it away, and you not only break cryptography, and anything that depends on it, you also break all the pyqt extensions built with pysip.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (4)
-
Eric Snow -
Nick Coghlan -
Petr Viktorin -
Victor Stinner