Move the pythoncapi_compat project under the GitHub Python or PSF organization?
#include "pythoncapi_compat.h"< in your project. You don't have to
Hi, Would it make sense to move the pythoncapi_compat project under the GitHub Python or PSF organization to make it more "official" and a little bit more sustainable? "The pythoncapi_compat project can be used to write a C extension supporting a wide range of Python versions with a single code base. It is made of the pythoncapi_compat.h header file and the upgrade_pythoncapi.py script." Documentation: https://pythoncapi-compat.readthedocs.io/en/latest/ GitHub: https://github.com/pythoncapi/pythoncapi_compat In the past, I managed to move my personal pyperf project under the PSF organization. Now other core developers are contributing and using it. It's better than having it as a personal project. pythoncapi_compat respects the PSF requirements to move a project in the GitHub PSF organization: contributors are required to respect the PSF Code of Conduct and the project has 3 maintainers (Dong-hee Na, Erlend E. AAsland and me). --- Some context. Incompatible C API changes in Python 3.10 and 3.11 require adding compatibility code to write C code compatible with old and new Python versions. For example, What's New in Python 3.10 suggests adding the following code to your project to get the Py_SET_REFCNT() function on Python 3.9 and older: #if PY_VERSION_HEX < 0x030900A4 # define Py_SET_REFCNT(obj, refcnt) ((Py_REFCNT(obj) = (refcnt)), (void)0) #endif Python 3.11 requires even more compatibility functions (7) and the implementation of these functions take more lines of code: * Py_SET_TYPE() * Py_SET_SIZE() * PyFrame_GetCode() * PyFrame_GetBack() * PyThreadState_GetFrame() * PyThreadState_EnterTracing() * PyThreadState_LeaveTracing() I added a reference to the pythoncapi_compat project to What's New in Python 3.11: "Or use the pythoncapi_compat project to get these functions on old Python functions." https://docs.python.org/dev/whatsnew/3.11.html#c-api-changes The project provides an upgrade_pythoncapi.py script to automatically add Python 3.10 and 3.11 support without losing support with Python 2.7. Just run the script, maybe copy pythoncapi_compat.h, and you're done! :-) It also provides a "pythoncapi_compat.h" header file which contains all the required compatibility code. Just copy the header file and use maintain this compatibilty code yourself anymore (pythoncapi_compat.h is made of 400 lines of C code). pythoncapi_compat supports Python (2.7 to 3.11), PyPy (2.7, 3.6, 3.7) and C++. It is distributed under the MIT license. My concern is that currently the project lives in my https://github.com/pythoncapi organization which is not really a sustainable option for the future. --- The pythoncapi_compat project is used by more and more Python projects, like Mercurial or mypy: https://pythoncapi-compat.readthedocs.io/en/latest/users.html I already made a similar request in June 2021 (move the project under the PSF organization): https://mail.python.org/archives/list/python-dev@python.org/thread/KHDZGCNOY... What changed since June 2021? Python 3.11 now requires way more compatibility code than Python 3.10 does: 7 functions instead of 1. On Discord, I was asked to ask the Steering Council. I asked the SC who asked me to ask on python-dev, so here I am :-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.
I created a poll on Discourse: https://discuss.python.org/t/move-the-pythoncapi-compat-project-under-the-gi... It will be closed automatically in 10 days. Victor
Results of the poll (which was open for 10 days): * Move pythoncapi_compat: 19 votes (90%) * Don't move pythoncapi_compat: 2 votes (10%) Victor On Fri, Feb 11, 2022 at 12:16 AM Victor Stinner <vstinner@python.org> wrote:
I created a poll on Discourse: https://discuss.python.org/t/move-the-pythoncapi-compat-project-under-the-gi...
It will be closed automatically in 10 days.
Victor
-- Night gathers, and now my watch begins. It shall not end until my death.
On 2022-02-10 2:58 p.m., Victor Stinner wrote:
Would it make sense to move the pythoncapi_compat project under the GitHub Python or PSF organization to make it more "official" and a little bit more sustainable?
I think that makes sense. Extensions typically have this kind of compatibility code built into them, so they can support multiple Python versions. It makes more sense to centralize that code. It will be easier to keep up-to-date and will be better quality. I think having the project more tightly associated with CPython is good too. When a change is made to the CPython extension API, it would be good to consider how pythoncapi_compat will need to be updated. E.g. how can extensions that want to support both old and new versions of the API work? Regards, Neil
On 10. 02. 22 23:58, Victor Stinner wrote:
Hi,
Would it make sense to move the pythoncapi_compat project under the GitHub Python or PSF organization to make it more "official" and a little bit more sustainable?
"The pythoncapi_compat project can be used to write a C extension supporting a wide range of Python versions with a single code base. It is made of the pythoncapi_compat.h header file and the upgrade_pythoncapi.py script."
Documentation: https://pythoncapi-compat.readthedocs.io/en/latest/ GitHub: https://github.com/pythoncapi/pythoncapi_compat
In the past, I managed to move my personal pyperf project under the PSF organization. Now other core developers are contributing and using it. It's better than having it as a personal project.
pythoncapi_compat respects the PSF requirements to move a project in the GitHub PSF organization: contributors are required to respect the PSF Code of Conduct and the project has 3 maintainers (Dong-hee Na, Erlend E. AAsland and me).
---
Some context.
Some additional context: AFAICS, the reason to move it to Python/PSF is to send some kind of signal. The question is, what kind of signal would that be? On the SC issue, Victor said
The signal is that the Python core devs bless and trust this project to be a good solution for the problem that it is trying to solve.
That sounds reasonable. Do we collectively trust the project? Or will this send a message that core devs should co-maintain the project? I personally wouldn't want to maintain it, but it it looks like there are at least 3 maintainers who do.
Incompatible C API changes in Python 3.10 and 3.11 require adding compatibility code to write C code compatible with old and new Python versions.
For example, What's New in Python 3.10 suggests adding the following code to your project to get the Py_SET_REFCNT() function on Python 3.9 and older:
#if PY_VERSION_HEX < 0x030900A4 # define Py_SET_REFCNT(obj, refcnt) ((Py_REFCNT(obj) = (refcnt)), (void)0) #endif
Python 3.11 requires even more compatibility functions (7) and the implementation of these functions take more lines of code:
* Py_SET_TYPE() * Py_SET_SIZE() * PyFrame_GetCode() * PyFrame_GetBack() * PyThreadState_GetFrame() * PyThreadState_EnterTracing() * PyThreadState_LeaveTracing()
Note that PyFrame_* and PyThreadState_* replace API that was previously private (specifically, documented as subject to change at any time). I think pythoncapi_compat is a good tool to run if you used code like that! My opinion on Py_SET_TYPE is that CPython shouldn't make the change necessary in the first place. But that's in PEP 674, so it's another discussion. Note that the upgrade_pythoncapi script uses regular expressions to turn e.g. all occurences of `something->interp` to `PyThreadState_GetInterpreter(something)`. It's fairly simple, but works in practice -- you aren't likely to have another struct with a `interp` member. IOW, you do need to do a thorough review after running it, as with a PR from a human contributor. But I don't think anyone a expects a *fully* automated solution, do they? The question of maintenance brings op questions about the scope and backwards compatibility of pythoncapi_compat itself. Currently there's partial support for Python 2.7, and full for 3.5+. Will the fixes stay in and accumulate indefinitely?
I added a reference to the pythoncapi_compat project to What's New in Python 3.11: "Or use the pythoncapi_compat project to get these functions on old Python functions." https://docs.python.org/dev/whatsnew/3.11.html#c-api-changes
The project provides an upgrade_pythoncapi.py script to automatically add Python 3.10 and 3.11 support without losing support with Python 2.7. Just run the script, maybe copy pythoncapi_compat.h, and you're done! :-)
#include "pythoncapi_compat.h"< in your project. You don't have to
It also provides a "pythoncapi_compat.h" header file which contains all the required compatibility code. Just copy the header file and use maintain this compatibilty code yourself anymore (pythoncapi_compat.h is made of 400 lines of C code).
pythoncapi_compat supports Python (2.7 to 3.11), PyPy (2.7, 3.6, 3.7) and C++. It is distributed under the MIT license.
My concern is that currently the project lives in my https://github.com/pythoncapi organization which is not really a sustainable option for the future.
Why? (That's an honest question.) I have a similar project (just header, not automation) for py2->py3 transition under my *personal* account, and it seems to have been pretty succesful: https://py3c.readthedocs.io/en/latest/
---
The pythoncapi_compat project is used by more and more Python projects, like Mercurial or mypy: https://pythoncapi-compat.readthedocs.io/en/latest/users.html
I already made a similar request in June 2021 (move the project under the PSF organization): https://mail.python.org/archives/list/python-dev@python.org/thread/KHDZGCNOY...
What changed since June 2021? Python 3.11 now requires way more compatibility code than Python 3.10 does: 7 functions instead of 1.
On Discord, I was asked to ask the Steering Council. I asked the SC who asked me to ask on python-dev, so here I am :-)
Note that there was no official communication from the SC, just clarifying questions and discussions from individuals who happen to be on the SC (and who were -- at least in my case -- thinking about what should go into a SC decision). Anyway, it would be great if the SC could just rubber-stamp a consensus here :)
On Fri, Feb 11, 2022 at 12:06 PM Petr Viktorin <encukou@gmail.com> wrote:
Or will this send a message that core devs should co-maintain the project? I personally wouldn't want to maintain it, but it it looks like there are at least 3 maintainers who do.
I think that Neal provided a better answer than me :-) It would be great than once an incompatible C API change is introduced in Python, pythoncapi_compat is updated. Updating pythoncapi_compat (with unit tests) ensures that it *is* possible to write a single code base compatible with old and new Python versions. Sometimes the change can be modified just a little it to make that transition smoother. In the past, that's exactly what I did. When I wrote PyThreadState_EnterTracing(), I started to create a draft PR for pythoncapi_compat. I merged the pythoncapi_compat PR once the API landed in Python. I did that for other new C API functions.
Note that PyFrame_* and PyThreadState_* replace API that was previously private (specifically, documented as subject to change at any time). I think pythoncapi_compat is a good tool to run if you used code like that!
My goal for pythoncapi_compat is to only support new *public* functions. The *implementation* can use the private API. So C extensions can use the new clean API and remains compatible with old Python versions using private structures and private functions. For example, the PyThreadState_EnterTracing() implementation uses PyThreadState.cframe.use_tracing (Python 3.10) or PyThreadState.use_tracing (Python 3.9 and older), and also PyThreadState.cframe.tracing, whereas the PyThreadState structure is considered as "private". // bpo-43760 added PyThreadState_EnterTracing() to Python 3.11.0a2 #if PY_VERSION_HEX < 0x030B00A2 && !defined(PYPY_VERSION) PYCAPI_COMPAT_STATIC_INLINE(void) PyThreadState_EnterTracing(PyThreadState *tstate) { tstate->tracing++; #if PY_VERSION_HEX >= 0x030A00A1 tstate->cframe->use_tracing = 0; #else tstate->use_tracing = 0; #endif } #endif
Note that the upgrade_pythoncapi script uses regular expressions to turn e.g. all occurences of `something->interp` to `PyThreadState_GetInterpreter(something)`. It's fairly simple, but works in practice -- you aren't likely to have another struct with a `interp` member. IOW, you do need to do a thorough review after running it, as with a PR from a human contributor. But I don't think anyone a expects a *fully* automated solution, do they?
Right, currently the script uses regex because it's built in Python: no third party dependency needed. So far, I didn't have to make any further changes after the script updated a C extension. In general, only a few lines number of lines are modified by the script. Less than 50: 5 to 10 lines on most C extensions.
The question of maintenance brings op questions about the scope and backwards compatibility of pythoncapi_compat itself. Currently there's partial support for Python 2.7, and full for 3.5+. Will the fixes stay in and accumulate indefinitely?
Supporting 2.7 and 3.5 is cheap: the code is already written. I plan to continue supporting them until it breaks. Once it will no longer be possible to test these versions, maybe the code can stay, untested, and bugs would only be fixed if someone proposes a fix. It just added Python 3.4 support since it is still possible to get a Python 3.4 in GitHub Actions (Ubuntu 18.04) :-) Python 2.7 support is mostly needed by Mercurial which is already compatible with Python 3. Mercurial plans to drop Python 2 support soon: in Mercurial 6.2, or maybe even Mercurial 6.1 (the latest release is 6.0): https://www.mercurial-scm.org/wiki/Python3 It seems like on Python 3, users target Python 3.6 as the minimum supported version. Some projects already dropped Python 3.6 support, move on to Python 3.7. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 11. 02. 22 13:52, Victor Stinner wrote:
On Fri, Feb 11, 2022 at 12:06 PM Petr Viktorin <encukou@gmail.com> wrote:
Or will this send a message that core devs should co-maintain the project? I personally wouldn't want to maintain it, but it it looks like there are at least 3 maintainers who do.
I think that Neal provided a better answer than me :-) It would be great than once an incompatible C API change is introduced in Python, pythoncapi_compat is updated. Updating pythoncapi_compat (with unit tests) ensures that it *is* possible to write a single code base compatible with old and new Python versions. Sometimes the change can be modified just a little it to make that transition smoother.
In the past, that's exactly what I did. When I wrote PyThreadState_EnterTracing(), I started to create a draft PR for pythoncapi_compat. I merged the pythoncapi_compat PR once the API landed in Python. I did that for other new C API functions.
All sounds good, I just insist on pointing out that this was only an incompatible change in the *internal* API. The functionality of PyThreadState_EnterTracing() was not possible to get with public API, but enough third parties “cheated”** and used private API to make it worth writing an automatic updater. ** Or maybe they read an older version of the docs where this wasn't clear, and then had compatibility expectations changed beneath them. The way we've been drawing lines between public and private in the past few years is not entirely fair, but it's the best we have.
Note that PyFrame_* and PyThreadState_* replace API that was previously private (specifically, documented as subject to change at any time). I think pythoncapi_compat is a good tool to run if you used code like that!
My goal for pythoncapi_compat is to only support new *public* functions. The *implementation* can use the private API. So C extensions can use the new clean API and remains compatible with old Python versions using private structures and private functions.
For example, the PyThreadState_EnterTracing() implementation uses PyThreadState.cframe.use_tracing (Python 3.10) or PyThreadState.use_tracing (Python 3.9 and older), and also PyThreadState.cframe.tracing, whereas the PyThreadState structure is considered as "private".
// bpo-43760 added PyThreadState_EnterTracing() to Python 3.11.0a2 #if PY_VERSION_HEX < 0x030B00A2 && !defined(PYPY_VERSION) PYCAPI_COMPAT_STATIC_INLINE(void) PyThreadState_EnterTracing(PyThreadState *tstate) { tstate->tracing++; #if PY_VERSION_HEX >= 0x030A00A1 tstate->cframe->use_tracing = 0; #else tstate->use_tracing = 0; #endif } #endif
Sounds reasonable, but... The implication of endorsing code like this is that *we cannot change private API even in patch releases*, which I don't think is documented anywhere, and might be a bit controversial. (I'm still planning to document this along with other similar issues, so if anyone wants to help here, please coordinate with me: https://discuss.python.org/t/documenting-python-versioning-and-stability-exp...).
Note that the upgrade_pythoncapi script uses regular expressions to turn e.g. all occurences of `something->interp` to `PyThreadState_GetInterpreter(something)`. It's fairly simple, but works in practice -- you aren't likely to have another struct with a `interp` member. IOW, you do need to do a thorough review after running it, as with a PR from a human contributor. But I don't think anyone a expects a *fully* automated solution, do they?
Right, currently the script uses regex because it's built in Python: no third party dependency needed. So far, I didn't have to make any further changes after the script updated a C extension.
In general, only a few lines number of lines are modified by the script. Less than 50: 5 to 10 lines on most C extensions.
The question of maintenance brings op questions about the scope and backwards compatibility of pythoncapi_compat itself. Currently there's partial support for Python 2.7, and full for 3.5+. Will the fixes stay in and accumulate indefinitely?
Supporting 2.7 and 3.5 is cheap: the code is already written. I plan to continue supporting them until it breaks. Once it will no longer be possible to test these versions, maybe the code can stay, untested, and bugs would only be fixed if someone proposes a fix.
Makes sense! Since you're now inviting the wider community as co-maintainers, could you document this?
It just added Python 3.4 support since it is still possible to get a Python 3.4 in GitHub Actions (Ubuntu 18.04) :-)
Python 2.7 support is mostly needed by Mercurial which is already compatible with Python 3. Mercurial plans to drop Python 2 support soon: in Mercurial 6.2, or maybe even Mercurial 6.1 (the latest release is 6.0): https://www.mercurial-scm.org/wiki/Python3
It seems like on Python 3, users target Python 3.6 as the minimum supported version. Some projects already dropped Python 3.6 support, move on to Python 3.7.
Victor
On Fri, Feb 11, 2022 at 3:14 PM Petr Viktorin <encukou@gmail.com> wrote:
Sounds reasonable, but...
The implication of endorsing code like this is that *we cannot change private API even in patch releases*, which I don't think is documented anywhere, and might be a bit controversial. (I'm still planning to document this along with other similar issues, so if anyone wants to help here, please coordinate with me: https://discuss.python.org/t/documenting-python-versioning-and-stability-exp...).
Maybe... we should not do that :-D The PyGC_Head ABI change was the most controversial, but it was already part of the private API and the following bug was closed as "not a bug": https://bugs.python.org/issue39599 This structure now belings to the *internal* C API.
Since you're now inviting the wider community as co-maintainers, could you document this?
Well, anyone is welcomed to contribute! Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 2022-02-11 06:14, Petr Viktorin wrote:
Sounds reasonable, but...
The implication of endorsing code like this is that *we cannot change private API even in patch releases*, which I don't think is documented anywhere, and might be a bit controversial.
I think we are still allowed to change them. We should be aware of the impact though. If an API is supposed to be private but is actually used by a large number of 3rd party extensions, we need to consider carefully when changing it. I don't have much sympathy for work caused for people using clearly marked private APIs. OTOH, practicality beats purity and we want them to be able to somehow use new versions of Python.
On 11. 02. 22 19:25, Neil Schemenauer wrote:
On 2022-02-11 06:14, Petr Viktorin wrote:
Sounds reasonable, but...
The implication of endorsing code like this is that *we cannot change private API even in patch releases*, which I don't think is documented anywhere, and might be a bit controversial.
I think we are still allowed to change them. We should be aware of the impact though. If an API is supposed to be private but is actually used by a large number of 3rd party extensions, we need to consider carefully when changing it. I don't have much sympathy for work caused for people using clearly marked private APIs. OTOH, practicality beats purity and we want them to be able to somehow use new versions of Python.
I'm all for "practicality beats purity" -- lots of people don't know the complicated backcompat rules, don't read the docs, have read older versions of docs, or don't notice things on reviews. It's bad to break their code without reason. But IMO it's good to set clear rules and stick to them ourselves, even if we're lenient in enforcing them. If we don't have much sympathy for projects that use private API where does that leave pythoncapi_compat? Is it an exception (like Cython? SWIG? Megacorp© Codegen™?) Or would it be hypocritical to adopt it? Or.. hmm, maybe we should retroactively declare the workaround API as stable in the specific CPython versions where pythoncapi_compat uses it? (Usually, these would be the versions before it was removed.) Is that what adopting pythoncapi_compat in the python org would mean?
On Mon, 14 Feb 2022 13:19:00 +0100 Petr Viktorin <encukou@gmail.com> wrote:
If we don't have much sympathy for projects that use private API where does that leave pythoncapi_compat?
If you look at pythoncapi_compat.h, it provides backports for recently-introduced public APIs such as PyObject_CallOneArg(). Regards Antoine.
On 14. 02. 22 13:37, Antoine Pitrou wrote:
On Mon, 14 Feb 2022 13:19:00 +0100 Petr Viktorin <encukou@gmail.com> wrote:
If we don't have much sympathy for projects that use private API where does that leave pythoncapi_compat?
If you look at pythoncapi_compat.h, it provides backports for recently-introduced public APIs such as PyObject_CallOneArg().
Yes. On older Python versions, where the public API wasn't yet available, those backports use private API. If we change the private API in a point release, the backport will break. Is this a proposal for "blessing" these specific workarounds, and making sure they don't break? That sounds nice, actually. It would mean some additional work, though.
On 14 Feb 2022, at 14:07, Petr Viktorin <encukou@gmail.com> wrote:
On 14. 02. 22 13:37, Antoine Pitrou wrote:
On Mon, 14 Feb 2022 13:19:00 +0100 Petr Viktorin <encukou@gmail.com> wrote:
If we don't have much sympathy for projects that use private API where does that leave pythoncapi_compat?
If you look at pythoncapi_compat.h, it provides backports for recently-introduced public APIs such as PyObject_CallOneArg().
Yes. On older Python versions, where the public API wasn't yet available, those backports use private API. If we change the private API in a point release, the backport will break.
Do you have an example of this? On first glance the pythoncapi_compat.h header only uses public APIs, other than (maybe) accessing fields of the thread state directly. BTW. I’m +1 on providing this header, it makes it easier for projects to maintain compatibility with older Python versions. That said, we should continue to be careful and considerate when evolving the public API as migrating a project to a newer API is still work. Ronald — Twitter / micro.blog: @ronaldoussoren Blog: https://blog.ronaldoussoren.net/
On 14. 02. 22 17:26, Ronald Oussoren wrote:
On 14 Feb 2022, at 14:07, Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote:
On 14. 02. 22 13:37, Antoine Pitrou wrote:
On Mon, 14 Feb 2022 13:19:00 +0100 Petr Viktorin <encukou@gmail.com <mailto:encukou@gmail.com>> wrote:
If we don't have much sympathy for projects that use private API where does that leave pythoncapi_compat?
If you look at pythoncapi_compat.h, it provides backports for recently-introduced public APIs such as PyObject_CallOneArg().
Yes. On older Python versions, where the public API wasn't yet available, those backports use private API. If we change the private API in a point release, the backport will break.
Do you have an example of this? On first glance the pythoncapi_compat.h header only uses public APIs, other than (maybe) accessing fields of the thread state directly.
That's my example. Those fields are documented as "subject to change at any time." But I wouldn't be afraid to do this more generally -- if we add a public API for something that needed private API before, freeze the old way in previous versions. Not only add it to pythoncapi_compat, but also to CPython CI, and maybe to the docs.
BTW. I’m +1 on providing this header, it makes it easier for projects to maintain compatibility with older Python versions. That said, we should continue to be careful and considerate when evolving the public API as migrating a project to a newer API is still work.
Yup. I'm strongly against making pythoncapi_compat mandatory.
On Tue, 15 Feb 2022, 2:57 am Petr Viktorin, <encukou@gmail.com> wrote:
Yes. On older Python versions, where the public API wasn't yet available, those backports use private API. If we change the private API in a point release, the backport will break.
Do you have an example of this? On first glance the pythoncapi_compat.h header only uses public APIs, other than (maybe) accessing fields of the thread state directly.
That's my example. Those fields are documented as "subject to change at any time."
But I wouldn't be afraid to do this more generally -- if we add a public API for something that needed private API before, freeze the old way in previous versions. Not only add it to pythoncapi_compat, but also to CPython CI, and maybe to the docs.
Adopting pythoncapi_compat would offer a relatively clean way to test that: if a maintenance branch change breaks pythoncapi_compat, then it's the maintenance branch that's considered broken. Cheers, Nick.
Hi, On Thu, 10 Feb 2022 23:58:31 +0100 Victor Stinner <vstinner@python.org> wrote:
Would it make sense to move the pythoncapi_compat project under the GitHub Python or PSF organization to make it more "official" and a little bit more sustainable?
I'm +1 on this. Many projects maintain their own set of compatibility wrappers and it makes sense to have an actively maintained project that centralizes these. Regards Antoine.
participants (7)
-
Antoine Pitrou -
Neil Schemenauer -
Neil Schemenauer -
Nick Coghlan -
Petr Viktorin -
Ronald Oussoren -
Victor Stinner