Hello,
I'm working on a C extension interface of Python. I want to create a new
interpreter by using the function Py_NewInterpreter() in a new thread,
which is created by pthread_create (my test files are in attachment), but
there are always errors when calling Py_NewInterpreter() such as "failed:
object already tracked by the garbage collector".
I would like to ask how to solve the problem and create a new interpreter
in multi thread in Python C extension?
Sincerely,
Kun
Hello,
since it is very relevant to the topic of this mailing list, I'd like to
announce the new HPy blog (and its first post):
https://hpyproject.org/blog/posts/2021/03/hello-hpy/
Any feedback (both on the blog post and on hpy itself) is appreciated!
cheers,
Antonio
Hello list!
After the e-mails from the previous week, I set up a call with Eric to
sync on Limited API/Stable ABI issues.
The higher-bandwidth (and more emotion-friendly) medium worked great for
us, but unfortunately, everyone else was excluded.
Here are some rather unstructued notes from my point of view. They might
be good discussion starters :)
Please read PEP 652 “Maintaining the Stable ABI” for my thoughts, plans
and rationales for the short term.
For the long term, there are some more vague plans and ideas:
There's not a 1:1 mapping between “Limited API”, “Stable ABI” and
“C-API, the Good Parts™”, *but* unless you're deep in this space, it
makes sense to conflate them. I aim to make them converge.
I intend to focus my CPython time on Limited API/Stable ABI for the
next... few years, probably.
The work I'm doing (extension isolation & now stable ABI) is similar to
the subinterpreters effort in several ways:
- They are both incomplete, but useful *today* for some limited cases,
and have lots of potential
- They need long-term effort, but most of the many steps are useful in
their own right.
In addition to the use cases in PEP 652, the stable ABI *should* ideally
be useful for:
- bindings for non-Python software, where shoehorning Python into the
buildsystem is not straightforward and building for several Python
versions at once is not practical. Also, raw speed tends to not matter.
- GUI apps, whose scripting plugins could use any Python the
user/scripter has installed. (Think Blender, if it was a smaller project
that couldn't afford to bundle Python.)
And here's some more concrete stuff. (Remember that these are still
vague ideas):
Static `PyObject` and `PyObject*` are the main thing in the Limited API
that's blocking subinterpreters. We see no other blockers.
A possible way to solve this is to make isolated subinterpreters support
*opt-in* for extension modules:
- Introduce a macro similar to the old PY_SSIZE_T_CLEAN that removes the
problematic items from the headers.
- This macro will give you access to a flag you can use to tell the
runtime the extension is subinterp-safe.
- Python will support both the current, GIL-sharing subinterpreters, and
isolated subinterpreters. (All signs say implementing this will be easy,
relative to making subinterpreters always isolated and breaking existing
code.)
- The macro will affect both the full API and the limited subset.
PEP 384 [1] introduced the limited API (AKA Py_LIMITED_API) and stable
ABI. The main objective:
"Extension modules and applications embedding Python can work with
different feature releases as long as they restrict themselves to this
stable ABI."
This goal is completely reasonable, particularly in 2009.
Furthermore, even though there are things we would have done
differently, in hindsight, the limited API mostly reaches that goal.
However, is it relevant in the current ecosystem? The burden on
extension maintainers to build for each Python release is much lower
now. At the same time, the maintenance burden of the current limited
API (especially ABI compatibility) is relatively high, especially for
our group of volunteers. It also makes some improvements harder. [2]
So the question is: Is the stable ABI still worth it? If not, what
would be the impact of abandoning it?
Note that the limited API is fine on its own, though I'll argue that
we should re-focus it (PEP 652?) and use a new name (to differentiate
from the API tied to the stable ABI). [3]
-eric
[1] https://www.python.org/dev/peps/pep-0384/
[2] https://bugs.python.org/issue43503
[3] https://mail.python.org/archives/list/capi-sig@python.org/thread/DX2HNPMJLI…
(Sorry for all the posts related to the limited API. It's been a pain
point for me lately.)
Would it be feasible to have distinct header files that contain only
the limited API (perhaps in Include/limited)?
The files could be auto-generated to avoid duplicated code (like other
files in the repo), though the nature of the limited API suggests the
only future changes would be opt-in ones, which should be manual
anyway. If there is no auto-generation then there should be a test to
ensure the limited API matches the corresponding public API
definitions exactly. Either way, Include/Python.h would have to be
updated to switch to Include/limited if Py_LIMITED_API is defined.
Also, FWIW, Include/limited would effectively fill the role of the
manifest described in PEP 652. [1]
With Include/limited, would we still need Include/cpython? I suppose
we could keep only public API in Include/*.h and keep
"private-but-not-internal" API in Include/cpython (or rename it to
Include/private or Include/unstable).
Finally, is the separation of headers by access "rings" worth it? On
the one hand it makes the divisions explicit and obvious and helps
avoid API leaking out accidentally. We'd had success in that regard
with Include/internal and Include/cpython. On the other hand, the
divisions mean related API is often spread out across multiple files
(albeit with matching names), where otherwise those would all be in
one file.
Anway, this is something that came to mind as I was writing up my
other posts to this list.
-eric
[1] https://www.python.org/dev/peps/pep-0652/#stable-abi-manifest
The stable ABI (ergo the limited API) is consistently problematic when
it comes to isolating subinterpreters. For example, there are ~120
PyObject exposed by the limited API, which is difficult to deal with
in terms of multiple interpreters. (See
https://bugs.python.org/issue43503.) After struggling with this
problem for a while I had an idea (which I'm pretty sure is just me
remembering something someone suggested at some point):
Let's mandate that extensions targeting the limited API cannot be
imported in subinterpreters. If we have that constraint in place then
nearly all the complex problems go away, since we have substantially
more flexibility to adapt the public API. (We'd still be careful
about API compatibility but wouldn't have to worry about ABI
compatibility.) Note that such extensions could still *create*
subinterpreters.
## Concerns
There are two concerns I have that counter the benefits of the idea:
* there may be existing extensions using the limited API *and* subinterpreters
* what if we want to allow "limited" extensions in subinterpreters in
the future?
As to the first one, I don't think it's a problem. Given the small
sets of people using either of the features, I expect the intersection
of those sets will be empty. That said, it would be worth looking at
all the extensions we know use subinterpreters, just to be sure.
That leaves the possibility of future support for importing limited
API extensions in subinterpreters. At the point we allow them, all
the complex problems come back and would have to be dealt with. Here
are some possible solutions:
* deal with the problems directly (if possible); the status quo
* introduce a "limited API 2.0"
* specify that a subset of the limited API that extensions must target
in order to support for subinterpreters (i.e.
"Py_LIMITED_PLUS_SUBINTERPRETERS_API")
Any of these could be done now, later, or only once the support is
needed (or never if it never comes up). My preference is to not worry
about it for now.
Maybe the limited API + subinterpreters would never be an issue (so
doing any of them now would be a waste of effort). I don't know. I
bring it up because I recall talk of Cython supporting/targeting the
limited API. Also, I hope subinterpreters become a widely used
feature and extensions that use the limited API would want to run
there. :)
## What Would Need to be Done?
* add the restriction to the docs
* update the import machinery to raise ImportError if an extension
targets the limited API
* how can the import machinery identify such modules?
* we may end up doing this relative to PEP 489 anyway
Anyway, your thoughts on the proposal would be helpful.
-eric
In PEP 384 [1] it says:
"This PEP proposes to define a stable set of API functions which
are guaranteed to be available for the lifetime of Python 3,
and which will also remain binary-compatible across versions."
Thus there are two parts that the PEP ties together closely: the
limited API (AKA Py_LIMITED_API) and the stable ABI. For the most
part, the API stability goal was driven by ABI stability.
Lately the binary compatibility part has seemed less important (as
implied by sprint/list discussions), while the limited API has begun
to stand on its own conceptually. On top of that, the limited API has
started to take on a second, distinct-but-similar role: "an
approximation of the minimal future API" or "the basic API that most
extensions should stick to". In that case strict compatibility is
desirable but not required (like it is with the stable ABI) or quite
so important.
Do we want a stand-alone public API that is focused on a minimal set
and extra resilient to future changes (but with less strict
compatibility guarantees)? If so then we should be deliberate about
it:
* introduce a different macro (Py_BASIC_API? Py_MINIMAL_API?)
* assess which parts of the limited API to keep in this new API (and
what to add from the full API)
* decide if there are any future-looking gaps we could fill (e.g. with macros)
* document the inclusions explicitly
* make (relative) compatibility guarantees clear
This will provide what folks are after (mostly) without incurring the
maintenance cost of binary compatibility. It seems like there's a lot
of overlap here with PEP 652, though I'm suggesting a distinct API.
Regardless, I think Py_LIMITED_API should preserve its strong
association with the stable ABI.
Finally, is such a "minimal" API worth it? Is the current full
"public" API sufficient (and projects like HPy can be the
less-constrained, future-oriented adaptations)? Would a minimal API
mean we'd be more free to make breaking changes in the rest of the
public API?
-eric
On Wed, Mar 17, 2021 at 12:08 PM Barry Warsaw <barry(a)python.org> wrote:
> Perhaps an even deeper and more provocative question is whether extension writers should be writing against *any* C API?
>
> Decoupling extensions from the concrete C API could give the interpreter more flexibility for internal change.
Good point! I'm totally on board. :) I suppose that would be a
Python 4.0 event and we could have no stable ABI for 4.0. (PEP 384
explicitly applies to Python 3.)
-eric
As Victor mentioned earlier today, there is work in progress on documenting the C API hierarchy: https://github.com/python/cpython/pull/24884
I've received valuable reviews from Victor and Guido, but I'd like to have a few more eyes on it.
Most of the text is stolen from bpo-43416, bpo-35134, PEP 384, and PEP 652. I've tried to put it together as concise as possible.
Any kind of feeback is highly appreciated.
Thanks,
Erlend
Hello.
I am writing a PEP to describe how I'd like to "resurrect" and maintain
the Stable ABI & Limited API going forward. I assume this'll be my main
focus for CPython in 2021.
I'm not as far along as I hoped to be at the end of 2020, but I guess
it's time to request comments.
If you have any thoughts, arguments or improvements I'd be happy to hear
them!
I'm including the text below; a rendered and forkable/pull-request-able
version is at https://github.com/encukou/abi3/blob/main/PEP.rst
Have a great start of 2021!
---
PEP: 9999
Title: Maintaining the Stable ABI
Author: Petr Viktorin <encukou(a)gmail.com>
Discussions-To:
Status: Draft
Type: Standards Track
Content-Type: text/x-rst
Created: 08-Dec-2020
Abstract
========
[A short (~200 word) description of the technical issue being addressed.]
XXX: Abstract should be written last
Motivation
==========
:pep:`384` defined a limited API and stable ABI, which allows extenders and
embedders of CPython to compile extension modules that are binary-compatible
with any subsequent version of 3.x.
In theory, this brings many advantages:
* A module can be built only once per platform and support multiple versions
of Python, reducing time, power and maintainer attention needed for
builds.
* Binary wheels using the stable ABI work with new versions of CPython
throughout the pre-release period, and can be tested in environments
where
building from source is not practical.
As a welcome side effect of the stable ABI's hiding of implementation
details
is that it is becoming a viable target for alternate Python implementations
that need to implement (parts of) the C API.
However, in hindsignt, PEP 384 and its implementation has several issues:
* There is no process keep the ABI up to date.
* Contents of the limited API are not listed explicitly, making it unclear
if a particular member (e.g. function, structure) is a part of it.
* There is no way to deprecate parts of the limited API.
This PEP defines the limited API more clearly and introducess process
designed to make the API more useful.
Additionally, PEP 384 defines a *limited API* as a way to build against the
stable ABI.
This PEP defines the limited API more robustly.
Rationale
=========
This PEP contains a lot of clarifications and definitions, but just one big
technical change: the stable ABI will be explicitly listed in
a human-maintained “manifest” file.
There have been efforts to collect such lists automatically, e.g. by
scanning
the symbols exported from Python.
This might seem to be easier to maintain by our volunteer team.
However, designing a future-proof API is not a trivial task.
The cost of updating an explicit manifest is small compared
to the overall work that should go into changing API that will need to
be suppported forever (or until Python 3 reaches
end of life, if that comes sooner).
This PEP proposes automatically generating things *from* the manifest:
initially documentation and DLL contents, with later possibilities
for also automating tests.
Stable ABI vs. Limited API
==========================
:pep:`384` and this document deal with the *Limited API* and the *Stable
ABI*,
two related but distinct concepts.
This section clarifies what they mean and defines some of their semantics
(either pre-existing or newly proposed here).
The word “Extensions” is used as a shorthand for all code that uses the
Python API, e.g. extension modules or software that embeds Python.
Stable ABI
----------
The CPython *Stable ABI* is a promise that extensions built with a specific
Cpython version will be usable with any newer interpreter of the same major
version, on the same platform and with the same compiler & settings.
For example, a extension built with CPython 3.10 Stable ABI will be
usable with
CPython 3.11, 3.12, and so on, but not necessarily with 4.0.
The Stable ABI is not generally forward-compatible: an extension built and
tested with CPython 3.10 will not generally be compatible with CPython 3.9.
.. note::
For example, starting in Python 3.10, the `Py_tp_doc` slot may be set to
`NULL`, while in older versions, a `NULL` value will likely crash the
interpreter.
The Stable ABI trades performance for its stability.
For example, many functions in the stable ABI are available as faster macros
to extensions that are built for a specific CPython version.
Future Python sversions may deprecate some members of the Stable ABI.
Such members will still work, but may suffer from issues like reduced
performance or, in the most extreme cases, memory/resource leaks.
Limited API
-----------
Stable ABI guarantee holds for extensions compiled from code that restricts
itself to the *Limited API*, a subset of CPython's C API.
The limited API is used when preprocessor macro `Py_LIMITED_API` is defined
to either `3` or the current `PYTHON_API_VERSION`.
The Limited API is not guaranteed to be *stable*.
In the future, parts of the limited API may be deprecated.
They may even be removed, as long as the *stable ABI* is kept
stable and Python's general backwards compatibility policy, :pep:`387`,
is followed.
.. note::
For example, a function declaration might be removed from public header
files but kept in the library.
This is currently a possibility for the future; this PEP does not to
propose
a concrete process for deprecations and removals.
The goal is for the limited API to cover everything needed to interact
with the interpreter.
There main reasons to not include a public API in the limited subset
should be that it needs implementation details that change between CPython
versions, like struct memory layouts, for performance reasons.
The limited API is not limited to CPython; other implementations are
encouraged to implement it and help drive its design.
Specification
=============
To make the stable ABI more useful and stable, the following changes
are proposed.
Stable ABI Manifest
-------------------
All members of the stable ABI – functions, typedefs, structs, struct fields,
data values etc. – will be explicitly listed in a single "manifest" file,
along with the Limited API version they were added in.
Struct fields that users of the stable ABI are allowed to access will be
listed explicitly.
Members that are not part of the Limited API, but are part of the Stable ABI
(e.g. ``PyObject.ob_type``, which is accessible by the ``Py_TYPE`` macro),
will be annotated as such.
Notes saying “Part of the stable ABI” will be added to Python's
documentation
automatically, in a way similar to the notes on functions that return
borrowed
references.
Source for the Windows shared library `python3.dll` will be generated
from the
stable ABI definition.
The format of the manifest will be subject to change whenever needed.
It should be consumed only by scripts in the CPython repository.
If a more public list is needed, a script can be added to generate it.
Contents of the Stable ABI
--------------------------
The initial stable ABI manifest will include:
* The Stable ABI specified in :pep:`384`.
* All functions listed in ``PC/python3dll.c``.
* All structs (struct typedefs) which these functions return or take as
arguments. (Fields of such structs will not necessarily be added.)
* New type slots, such as ``Py_am_aiter``.
* The type flags ``Py_TPFLAGS_DEFAULT``, ``Py_TPFLAGS_BASETYPE``,
``Py_TPFLAGS_HAVE_GC``, ``Py_TPFLAGS_METHOD_DESCRIPTOR``.
* The calling conventions ``METH_*`` (except deprecated ones).
* All API needed by macros is the stable ABI (usually annotated as not being
part of the limited API).
Additional items may be aded to the initial manifest according to
the checklist below.
Testing the Stable ABI
----------------------
An automatically generated test module will be added to ensure that all
members
of the stable ABI are available at compile time
For each function in the stable ABI, a test will be added that calls the
function using `ctypes`. (Where calling is not practical, such as with
functions related to intepreter initialization and shutdown, the test will
only look the function up.)
This should prevent regressions when a function is converted to a macro,
which keeps the same API but breaks the ABI.
An check will be added to ensure all functions in the stable ABI are tested
this way.
Changing the Limited API
------------------------
A checklist for changing the limited API, including new members (structs,
functions or values), will be added to the `Devguide`_.
The checklist will 1) mention best practices and common pitfalls in Python
C API design and 2) guide the developer around the files that need
changing and
scripts that need running when the limited API is changed.
Below is the initial proposal for the checklist. After the PEP is accepted,
see the Devguide for the current version.
Note that the checklist applies to new additions; not the existing
limited API.
Design considerations:
* Make sure the change does not break the Stable ABI of any version of
Python
since 3.5.
* Make sure no exposed names are private (i.e. begin with an underscore).
* Make sure the new API is well documented.
* Make sure the types of all parameters and return values of the added
function(s) and all fields of the added struct(s) are be part of the
limited API (or standard C).
* Make sure the new API and its intended use follows standard C, not just
features of currently suppoerted platforms.
* Do not cast a function pointer to ``void*`` (a data pointer) or
vice versa.
* Make sure the new API follows reference counting conventions.
(Following them
makes the API easier to reason about, and easier use in other Python
implementations.)
* Do not return borrowed references from functions.
* Do not steal references to function arguments.
* Make sure the ownership rules and lifetimes of all applicable struct
fields,
arguments and return values are well defined.
* Think about ease of use for the user. (In C, ease of use itself is not
very
important; what *is* important is reducing boilerplate code needed to
use the
API. Bugs like to hide in boiler plates.)
* If a function will be often called with specific value for an argument,
consider making it default (assumed when ``NULL`` is passed in).
* Think about future extensions: for example, if it's possible that future
Python versions will need to add a new field to your struct,
how will that be done?
* Make as few assumptions as possible about details that might change in
future CPython versions or differ across C API implementations:
* The GIL
* Garbage collection
* Layout of PyObject and other structs
If following these guidelines would hurt performance, add a fast function
(or macro) to the non-limited API and a stable equivalent to the limited
API.
If anything is unclear, or you have a good reason to break the guidelines,
consider discussing the change at the `capi-sig`_ mailing list.
.. _capi-sig: https://mail.python.org/mailman3/lists/capi-sig.python.org/
Procedure:
* Move the declaration to a header file directly under ``Include/`` and
``#if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03yy0000``
(with the ``yy`` corresponding to Python version).
* Make an entry in the stable ABI list. (XXX: mention filename)
* For functions, add a test that calls the function using ctypes
(XXX: mention filename).
* Regenerate the autogenerated files. (XXX: specific instructions)
Advice for Extenders and Embedders
----------------------------------
The following notes will be added to documentation.
Extension authors should test with all Python versions they support,
and preferably build with the lowest such version.
Compiling with ``Py_LIMITED_API`` defined is *not* a guarantee that your
code
conforms to the limited API or the stable ABI.
It only covers definitions, but an API also includes other issues,
such as expected semantics.
Examples of issues that ``Py_LIMITED_API`` does not guard against are:
* Calling a function with invalid arguments
* An function that started accepting ``NULL`` values for an argument
in Python 3.9 will fail if ``NULL`` is passed to it under Python 3.8.
Only testing with 3.8 (or lower versions) will uncover this issue.
* Some structs include a few fields that are part of the stable ABI and
other
fields that aren't.
``Py_LIMITED_API`` does not filter out such “private” fields.
* Using something that is not documented as part of the stable ABI,
but exposed even with ``Py_LIMITED_API`` defined.
Despite the team's best efforts, such issues may happen.
Backwards Compatibility
=======================
The PEP aims at full compatibility with the existing stable ABI and limited
API, but defines them terms more explicitly.
It might not be consistent with some interpretations of what the existing
stable ABI/limited API is.
Security Implications
=====================
None known.
How to Teach This
=================
Technical documentation will be provided.
It will be aimed at experienced users familiar with C.
Reference Implementation
========================
None so far.
Rejected Ideas
==============
While this PEP acknowledges that parts of the limited API might be
deprecated
or removed in the future, a process to do this is not in scope, and is left
to a possible future PEP.
Open Issues
===========
None so far.
References
==========
Copyright
=========
This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.
.. _Devguide: https://devguide.python.org/
..
Local Variables:
mode: indented-text
indent-tabs-mode: nil
sentence-end-double-space: t
fill-column: 70
coding: utf-8
End:
