Mailman 3 March 2021 - capi-sig

Can we explicitly specify that the limited API can't be used in subinterpreters?
by Eric Snow 22 Mar '21

22 Mar '21

The stable ABI (ergo the limited API) is consistently problematic when it comes to isolating subinterpreters. For example, there are ~120 PyObject exposed by the limited API, which is difficult to deal with in terms of multiple interpreters. (See https://bugs.python.org/issue43503.) After struggling with this problem for a while I had an idea (which I'm pretty sure is just me remembering something someone suggested at some point): Let's mandate that extensions targeting the limited API cannot be imported in subinterpreters. If we have that constraint in place then nearly all the complex problems go away, since we have substantially more flexibility to adapt the public API. (We'd still be careful about API compatibility but wouldn't have to worry about ABI compatibility.) Note that such extensions could still *create* subinterpreters. ## Concerns There are two concerns I have that counter the benefits of the idea: * there may be existing extensions using the limited API *and* subinterpreters * what if we want to allow "limited" extensions in subinterpreters in the future? As to the first one, I don't think it's a problem. Given the small sets of people using either of the features, I expect the intersection of those sets will be empty. That said, it would be worth looking at all the extensions we know use subinterpreters, just to be sure. That leaves the possibility of future support for importing limited API extensions in subinterpreters. At the point we allow them, all the complex problems come back and would have to be dealt with. Here are some possible solutions: * deal with the problems directly (if possible); the status quo * introduce a "limited API 2.0" * specify that a subset of the limited API that extensions must target in order to support for subinterpreters (i.e. "Py_LIMITED_PLUS_SUBINTERPRETERS_API") Any of these could be done now, later, or only once the support is needed (or never if it never comes up). My preference is to not worry about it for now. Maybe the limited API + subinterpreters would never be an issue (so doing any of them now would be a waste of effort). I don't know. I bring it up because I recall talk of Cython supporting/targeting the limited API. Also, I hope subinterpreters become a widely used feature and extensions that use the limited API would want to run there. :) ## What Would Need to be Done? * add the restriction to the docs * update the import machinery to raise ImportError if an extension targets the limited API * how can the import machinery identify such modules? * we may end up doing this relative to PEP 489 anyway Anyway, your thoughts on the proposal would be helpful. -eric

4 6

Py_LIMITED_API vs. the Stable ABI
by Eric Snow 22 Mar '21

22 Mar '21

In PEP 384 [1] it says: "This PEP proposes to define a stable set of API functions which are guaranteed to be available for the lifetime of Python 3, and which will also remain binary-compatible across versions." Thus there are two parts that the PEP ties together closely: the limited API (AKA Py_LIMITED_API) and the stable ABI. For the most part, the API stability goal was driven by ABI stability. Lately the binary compatibility part has seemed less important (as implied by sprint/list discussions), while the limited API has begun to stand on its own conceptually. On top of that, the limited API has started to take on a second, distinct-but-similar role: "an approximation of the minimal future API" or "the basic API that most extensions should stick to". In that case strict compatibility is desirable but not required (like it is with the stable ABI) or quite so important. Do we want a stand-alone public API that is focused on a minimal set and extra resilient to future changes (but with less strict compatibility guarantees)? If so then we should be deliberate about it: * introduce a different macro (Py_BASIC_API? Py_MINIMAL_API?) * assess which parts of the limited API to keep in this new API (and what to add from the full API) * decide if there are any future-looking gaps we could fill (e.g. with macros) * document the inclusions explicitly * make (relative) compatibility guarantees clear This will provide what folks are after (mostly) without incurring the maintenance cost of binary compatibility. It seems like there's a lot of overlap here with PEP 652, though I'm suggesting a distinct API. Regardless, I think Py_LIMITED_API should preserve its strong association with the stable ABI. Finally, is such a "minimal" API worth it? Is the current full "public" API sufficient (and projects like HPy can be the less-constrained, future-oriented adaptations)? Would a minimal API mean we'd be more free to make breaking changes in the rest of the public API? -eric

3 3

Re: Get Rid of the Stable ABI?
by Eric Snow 18 Mar '21

18 Mar '21

On Wed, Mar 17, 2021 at 12:08 PM Barry Warsaw <barry(a)python.org> wrote: > Perhaps an even deeper and more provocative question is whether extension writers should be writing against *any* C API? > > Decoupling extensions from the concrete C API could give the interpreter more flexibility for internal change. Good point! I'm totally on board. :) I suppose that would be a Python 4.0 event and we could have no stable ABI for 4.0. (PEP 384 explicitly applies to Python 3.) -eric

6 8

Include/README
by Erlend Aasland 17 Mar '21

17 Mar '21

As Victor mentioned earlier today, there is work in progress on documenting the C API hierarchy: https://github.com/python/cpython/pull/24884 I've received valuable reviews from Victor and Guido, but I'd like to have a few more eyes on it. Most of the text is stolen from bpo-43416, bpo-35134, PEP 384, and PEP 652. I've tried to put it together as concise as possible. Any kind of feeback is highly appreciated. Thanks, Erlend

1 0