How can we ship a header file, or even a dynamic library, to port existing C extension modules to the new incompatible Python C API?
Hi,
As I wrote in a previous email, the C API of CPython is evolving slowly to hide more and more implementation details. Sadly, some changes break the backward compatibility. Sometimes, a new function must be used, except that the function doesn't exist in older Python versions.
"Update on C API changes to hide implementation details" https://mail.python.org/archives/list/capi-sig@python.org/thread/HYFQHLGNCRF...
Would it be possible to ship a header file (a single ".h" file) and/or even a dynamic library to provide new functions to old Python versions? I'm not sure if it's possible to ship a header file as a package on PyPI and make it available for C compilers easily without monkey patching distutils. Another option is to encourage Linux distributions to package such header file to make it easy to install on the system.
Another (worse?) option is to advise authors of each C extension module to vendor a copy of the header file in their project and update it from time to time when needed.
At least in C, we can provide most "missing" functions on older Python versions as macros or static inline functions. Python 3.6 now requires a subset C99 with static inline functions.
The situation is very similar to the Python 2 to Python 3 transition. Benjamin Peterson wrote the "six" which became quickly very popular. First, projects vendored a copy of the "six" module as a submodule of their project (src/project/six.py, imported as "project.six"). Then, slowly, the "six" module was shipped by Linux distributions or installed via "pip install six".
Cython already does that internally. A few examples: https://github.com/cython/cython/blob/57b4bdeb88cc18f8e2f89fe9a7cf506d4ee0f8...
#if PY_VERSION_HEX < 0x030200A4 typedef long Py_hash_t; #define __Pyx_PyInt_FromHash_t PyInt_FromLong #define __Pyx_PyInt_AsHash_t __Pyx_PyIndex_AsHash_t #else #define __Pyx_PyInt_FromHash_t PyInt_FromSsize_t #define __Pyx_PyInt_AsHash_t __Pyx_PyIndex_AsSsize_t #endif
#if CYTHON_COMPILING_IN_LIMITED_API #define __Pyx_PyThreadState_Current PyThreadState_Get() #elif !CYTHON_FAST_THREAD_STATE #define __Pyx_PyThreadState_Current PyThreadState_GET() #elif PY_VERSION_HEX >= 0x03060000 //#elif PY_VERSION_HEX >= 0x03050200 // Actually added in 3.5.2, but compiling against that does not guarantee that we get imported there. #define __Pyx_PyThreadState_Current _PyThreadState_UncheckedGet() #elif PY_VERSION_HEX >= 0x03000000 #define __Pyx_PyThreadState_Current PyThreadState_GET() #else #define __Pyx_PyThreadState_Current _PyThreadState_Current #endif
#if PY_VERSION_HEX >= 0x030900A4 #define __Pyx_SET_REFCNT(obj, refcnt) Py_SET_REFCNT(obj, refcnt) #define __Pyx_SET_SIZE(obj, size) Py_SET_SIZE(obj, size) #else #define __Pyx_SET_REFCNT(obj, refcnt) Py_REFCNT(obj) = (refcnt) #define __Pyx_SET_SIZE(obj, size) Py_SIZE(obj) = (size) #endif
Victor
Night gathers, and now my watch begins. It shall not end until my death.
On Wed, Jun 3, 2020 at 11:30 PM Victor Stinner <vstinner@python.org> wrote:
Hi,
As I wrote in a previous email, the C API of CPython is evolving slowly to hide more and more implementation details. Sadly, some changes break the backward compatibility. Sometimes, a new function must be used, except that the function doesn't exist in older Python versions.
"Update on C API changes to hide implementation details" https://mail.python.org/archives/list/capi-sig@python.org/thread/HYFQHLGNCRF...
Would it be possible to ship a header file (a single ".h" file) and/or even a dynamic library to provide new functions to old Python versions? I'm not sure if it's possible to ship a header file as a package on PyPI and make it available for C compilers easily without monkey patching distutils. Another option is to encourage Linux distributions to package such header file to make it easy to install on the system.
Another (worse?) option is to advise authors of each C extension module to vendor a copy of the header file in their project and update it from time to time when needed.
If you can provide backward compatibility, the obvious question for a C extension developer is WHY ARE YOU CHANGING THE API IN THE FIRST PLACE?
Now I happen to follow the Python dev and CAPI mailing lists, so yes I do know why you want to change the API, you don't have to explain it to me. But I can imagine this being interpreted as the all too common "churn" rather than actual improvement, breaking people's code just because some dev wants to change stuff.
My suggestion is that for the first, or first few, versions of Python with the new API, this backwards compatibility header and any functions be shipped as part of the standard library. Extension developers can either update their code to the new API, or add a new line #include <PythonCompat.h> and everything works but they get a nag message about needing to update.
(Actually, it might be better if an extension module compiled with the backward compat lib gives a runtime nag warning, so users put pressure on the author.)
Making extension developers jump through hoops to get their old code to work will just suck up time and effort that they could have been using to update the code. My suggestion is to have a grace period for deprecated code, then a clean break.
--
cheers,
Hugh FisherHi Hugh,
Le jeu. 4 juin 2020 à 02:15, Hugh Fisher <hugo.fisher@gmail.com> a écrit :
If you can provide backward compatibility, the obvious question for a C extension developer is WHY ARE YOU CHANGING THE API IN THE FIRST PLACE?
It's explain in my draft PEP: https://github.com/vstinner/misc/blob/master/cpython/pep-opaque-c-api.rst
I'm working on the PEP to get it accepted :-) Yeah, it would help to be able to mention a PEP number rather than BPO issues.
Now I happen to follow the Python dev and CAPI mailing lists, so yes I do know why you want to change the API, you don't have to explain it to me. But I can imagine this being interpreted as the all too common "churn" rather than actual improvement, breaking people's code just because some dev wants to change stuff.
I'm not sure where is the best place to communicate on the rationale behind changes.
Currently, the "C API Changes: Porting to Python 3.10" section of What's New in Python 3.10 contains links to the bpo issue which introduced the change: https://github.com/python/cpython/blob/master/Doc/whatsnew/3.10.rst#porting-...
For example, Py_SET_xxx() new macros are linked to https://bugs.python.org/issue39573: the first message of this issue explains the rationale for this incompatible change.
Note: My expectation is that only a few C extensions use Py_TYPE() or Py_SIZE() as l-value. If we consider that these changes are breaking too many C extension modules in the wild during the Python 3.10 dev cycle, we can restrict the change to the limited C API. I did something similar in Python 3.9: we revert incompatible changes which broke too many third party projects. Sadly, there is no tool yet to estimate how many third party projects are broken by incompatible changes.
At Red Hat, we are rebuilding the whole Fedora operating system with a newer Python version, alpha and then beta versions. We discover packages broken by the newer Python. It gives me feedback to take smarter decision about incompatible changes.
I also have a https://github.com/vstinner/pythonci toy project to test a very low number of projects (4 projects: coverage, cython, jinja, numpy).
My suggestion is that for the first, or first few, versions of Python with the new API, this backwards compatibility header and any functions be shipped as part of the standard library. Extension developers can either update their code to the new API, or add a new line #include <PythonCompat.h> and everything works but they get a nag message about needing to update.
I don't see how it is possible to emit a deprecation warning when a Py_TYPE() is used as an l-value, but no warning if it's used as a r-value. Warning on "Py_TYPE(obj) = type;" but no warning on "type = Py_TYPE(obj);". This change is really a corner case.
I added Py_SET_xxx() functions in Python 3.9 on purpose and only introduced the backward incompatible change in Python 3.10. So Python 3.9 is the transition release.
The problem is how to get Py_SET_xxx() functions on Python 3.8 and older?
If Python 3.10 ships a new PythonCompat.h, it will only be available on Python 3.10, not for Python 3.9 and older. Also, I would like to update PythonCompat.h frequently. Old versions of Python, like Python 3.6, only get security fixes. I would prefer to distribute it separately.
Victor
Night gathers, and now my watch begins. It shall not end until my death.
On 2020-06-03 15:29, Victor Stinner wrote:
Hi,
As I wrote in a previous email, the C API of CPython is evolving slowly to hide more and more implementation details. Sadly, some changes break the backward compatibility. Sometimes, a new function must be used, except that the function doesn't exist in older Python versions. > "Update on C API changes to hide implementation details" https://mail.python.org/archives/list/capi-sig@python.org/thread/HYFQHLGNCRF...
Could we do breaking changes within the proper deprecation cycle? And preferably with a PEP?
I see that the current documentation of Py_TYPE, Py_REFCNT ans Py_SIZE suggest that using them as lvalues is perfectly valid. There is no mention that anything changed. I see there was no documentation change even as the breaking code change went in. That's quite unfair to users.
Would it be possible to ship a header file (a single ".h" file) and/or even a dynamic library to provide new functions to old Python versions? I'm not sure if it's possible to ship a header file as a package on PyPI and make it available for C compilers easily without monkey patching distutils. Another option is to encourage Linux distributions to package such header file to make it easy to install on the system.
My project py3c was ment as compatibility headers for the 2→3 transition, but I ended up taking pull requests for newer features as well: https://github.com/encukou/py3c
I haven't figured out how to make the header file easily available from PyPI, however.
Another (worse?) option is to advise authors of each C extension module to vendor a copy of the header file in their project and update it from time to time when needed.
At least in C, we can provide most "missing" functions on older Python versions as macros or static inline functions. Python 3.6 now requires a subset C99 with static inline functions.
The situation is very similar to the Python 2 to Python 3 transition. Benjamin Peterson wrote the "six" which became quickly very popular. First, projects vendored a copy of the "six" module as a submodule of their project (src/project/six.py, imported as "project.six"). Then, slowly, the "six" module was shipped by Linux distributions or installed via "pip install six".
Cython already does that internally. A few examples: https://github.com/cython/cython/blob/57b4bdeb88cc18f8e2f89fe9a7cf506d4ee0f8...
#if PY_VERSION_HEX < 0x030200A4 typedef long Py_hash_t; #define __Pyx_PyInt_FromHash_t PyInt_FromLong #define __Pyx_PyInt_AsHash_t __Pyx_PyIndex_AsHash_t #else #define __Pyx_PyInt_FromHash_t PyInt_FromSsize_t #define __Pyx_PyInt_AsHash_t __Pyx_PyIndex_AsSsize_t #endif
#if CYTHON_COMPILING_IN_LIMITED_API #define __Pyx_PyThreadState_Current PyThreadState_Get() #elif !CYTHON_FAST_THREAD_STATE #define __Pyx_PyThreadState_Current PyThreadState_GET() #elif PY_VERSION_HEX >= 0x03060000 //#elif PY_VERSION_HEX >= 0x03050200 // Actually added in 3.5.2, but compiling against that does not guarantee that we get imported there. #define __Pyx_PyThreadState_Current _PyThreadState_UncheckedGet() #elif PY_VERSION_HEX >= 0x03000000 #define __Pyx_PyThreadState_Current PyThreadState_GET() #else #define __Pyx_PyThreadState_Current _PyThreadState_Current #endif
#if PY_VERSION_HEX >= 0x030900A4 #define __Pyx_SET_REFCNT(obj, refcnt) Py_SET_REFCNT(obj, refcnt) #define __Pyx_SET_SIZE(obj, size) Py_SET_SIZE(obj, size) #else #define __Pyx_SET_REFCNT(obj, refcnt) Py_REFCNT(obj) = (refcnt) #define __Pyx_SET_SIZE(obj, size) Py_SIZE(obj) = (size) #endif
Victor
Hi Petr,
(I already replied to some of your questions in my reply to Hugh.)
Le jeu. 4 juin 2020 à 12:09, Petr Viktorin <encukou@gmail.com> a écrit :
I see that the current documentation of Py_TYPE, Py_REFCNT ans Py_SIZE suggest that using them as lvalues is perfectly valid. There is no mention that anything changed.
Oh, it seems like https://docs.python.org/dev/ is no longer updated for an unknown reason.
The documentation was updated.
If you look into the up to date Git repository, Py_TYPE() got a ".. versionchanged:: 3.10" markup, and the change is mentioned in "Porting to Python 3.10: C API Changes" section of What's New in Python 3.10. https://github.com/python/cpython/blob/master/Doc/c-api/structures.rst
My project py3c was ment as compatibility headers for the 2→3 transition, but I ended up taking pull requests for newer features as well: https://github.com/encukou/py3c
That's an interesting approach, to automate "updating" C extensions to the newer C API.
There is also a similar project for Python code: https://github.com/asottile/pyupgrade (and many others more specific for Python 2 => Python 3 migration).
One option is to copy/paste compatibility macros like Py_SET_SIZE() for old Python versions directly in the C code of these extensions. It avoids depending on something external. It's a variant of having a vendored copy of an hypothetical PythonCompat.h header file.
By the way, it's also how Cython handles backward compatibility.
Victor
Night gathers, and now my watch begins. It shall not end until my death.
On 2020-06-04 15:56, Victor Stinner wrote:
Hi Petr,
(I already replied to some of your questions in my reply to Hugh.)
Le jeu. 4 juin 2020 à 12:09, Petr Viktorin <encukou@gmail.com> a écrit :
I see that the current documentation of Py_TYPE, Py_REFCNT ans Py_SIZE suggest that using them as lvalues is perfectly valid. There is no mention that anything changed.
Oh, it seems like https://docs.python.org/dev/ is no longer updated for an unknown reason.
The documentation was updated.
If you look into the up to date Git repository, Py_TYPE() got a ".. versionchanged:: 3.10" markup, and the change is mentioned in "Porting to Python 3.10: C API Changes" section of What's New in Python 3.10. https://github.com/python/cpython/blob/master/Doc/c-api/structures.rst
AFAIK, the deprecation period should be at least a release. Wouldn't that translate to documenting the new behavior 3.10, but only actually breaking in 3.11?
Compiler warnings for deprecation and deprecation in the documentation are good. But in my experience, even if a deprecation is documented for 10 years, people only start to pay attention when the compilation actually fails.
Victor
Le jeu. 4 juin 2020 à 16:46, Petr Viktorin <encukou@gmail.com> a écrit :
On 2020-06-04 15:56, Victor Stinner wrote:
Hi Petr,
(I already replied to some of your questions in my reply to Hugh.)
Le jeu. 4 juin 2020 à 12:09, Petr Viktorin <encukou@gmail.com> a écrit :
I see that the current documentation of Py_TYPE, Py_REFCNT ans Py_SIZE suggest that using them as lvalues is perfectly valid. There is no mention that anything changed.
Oh, it seems like https://docs.python.org/dev/ is no longer updated for an unknown reason.
The documentation was updated.
If you look into the up to date Git repository, Py_TYPE() got a ".. versionchanged:: 3.10" markup, and the change is mentioned in "Porting to Python 3.10: C API Changes" section of What's New in Python 3.10. https://github.com/python/cpython/blob/master/Doc/c-api/structures.rst
AFAIK, the deprecation period should be at least a release. Wouldn't that translate to documenting the new behavior 3.10, but only actually breaking in 3.11?
-- Night gathers, and now my watch begins. It shall not end until my death.
On Thu, 4 Jun 2020 at 17:37, Victor Stinner <vstinner@python.org> wrote:
Compiler warnings for deprecation and deprecation in the documentation are good. But in my experience, even if a deprecation is documented for 10 years, people only start to pay attention when the compilation actually fails.
Maybe by default allow extensions using the deprecated function to fail compilation, but have an easy escape hatch, e.g. if you add -DPYTHON_ALLOW_DEPRECATED_XXXX to your compiler flags, to acknowledge that you're using a deprecated function, but at least you can move on for now.
Having that -DPYTHON_ALLOW_DEPRECATED_XXXX compiler flag in your build scripts will be a constant reminder that you need to fix it eventually.
And to be effective, I suggest that there should be several relatively fine grained PYTHON_ALLOW_DEPRECATED_XXXX flags, for small groups of functions. Otherwise it's too easy, people will just turn on the -DPYTHON_DEPRECATED_FUNCTIONS once, and forget about it for years.
Victor
Le jeu. 4 juin 2020 à 16:46, Petr Viktorin <encukou@gmail.com> a écrit :
On 2020-06-04 15:56, Victor Stinner wrote:
Hi Petr,
(I already replied to some of your questions in my reply to Hugh.)
Le jeu. 4 juin 2020 à 12:09, Petr Viktorin <encukou@gmail.com> a
écrit :
I see that the current documentation of Py_TYPE, Py_REFCNT ans Py_SIZE suggest that using them as lvalues is perfectly valid. There is no mention that anything changed.
Oh, it seems like https://docs.python.org/dev/ is no longer updated for an unknown reason.
The documentation was updated.
If you look into the up to date Git repository, Py_TYPE() got a ".. versionchanged:: 3.10" markup, and the change is mentioned in "Porting to Python 3.10: C API Changes" section of What's New in Python 3.10. https://github.com/python/cpython/blob/master/Doc/c-api/structures.rst
AFAIK, the deprecation period should be at least a release. Wouldn't that translate to documenting the new behavior 3.10, but only actually breaking in 3.11?
-- Night gathers, and now my watch begins. It shall not end until my death.
capi-sig mailing list -- capi-sig@python.org To unsubscribe send an email to capi-sig-leave@python.org https://mail.python.org/mailman3/lists/capi-sig.python.org/ Member address: gjcarneiro@gmail.com
-- Gustavo J. A. M. Carneiro Gambit Research "The universe is always one step beyond logic." -- Frank Herbert
On 2020-06-04 18:37, Victor Stinner wrote:
Compiler warnings for deprecation and deprecation in the documentation are good. But in my experience, even if a deprecation is documented for 10 years, people only start to pay attention when the compilation actually fails.
Not *all* people.
I don't think you can count the projects that take the warning into account. You only see the projects that ignore the warning until it's too late. If you remove things without a warning, you're not even giving anyone a fair chance to update their code.
Le jeu. 4 juin 2020 à 16:46, Petr Viktorin <encukou@gmail.com> a écrit :
On 2020-06-04 15:56, Victor Stinner wrote:
Hi Petr,
(I already replied to some of your questions in my reply to Hugh.)
Le jeu. 4 juin 2020 à 12:09, Petr Viktorin <encukou@gmail.com> a écrit :
I see that the current documentation of Py_TYPE, Py_REFCNT ans Py_SIZE suggest that using them as lvalues is perfectly valid. There is no mention that anything changed.
Oh, it seems like https://docs.python.org/dev/ is no longer updated for an unknown reason.
The documentation was updated.
If you look into the up to date Git repository, Py_TYPE() got a ".. versionchanged:: 3.10" markup, and the change is mentioned in "Porting to Python 3.10: C API Changes" section of What's New in Python 3.10. https://github.com/python/cpython/blob/master/Doc/c-api/structures.rst
AFAIK, the deprecation period should be at least a release. Wouldn't that translate to documenting the new behavior 3.10, but only actually breaking in 3.11?
I was only talking about the very specific case of Py_TYPE(), Py_REFCNT() and Py_SIZE() used as l-value.
For other removed C functions, we do add deprecation warnings and try to have at least one release with these warnings. For example, all functions using Py_UNICODE are deprecated.
Recently (1 or 2 years ago?), the warnings were modified to also be emitted using Visual Studio C compiler (MSC).
Examples of usage of the Py_DEPRECATED macro:
Py_DEPRECATED(3.3) PyAPI_FUNC(Py_UNICODE) PyUnicode_GetMax(void);
Py_DEPRECATED(3.3) PyAPI_FUNC(PyObject*) PyUnicode_Encode( const Py_UNICODE *s, /* Unicode char buffer */ Py_ssize_t size, /* number of Py_UNICODE chars to encode */ const char *encoding, /* encoding */ const char *errors /* error handling */ );
Victor
Le jeu. 4 juin 2020 à 19:50, Petr Viktorin <encukou@gmail.com> a écrit :
On 2020-06-04 18:37, Victor Stinner wrote:
Compiler warnings for deprecation and deprecation in the documentation are good. But in my experience, even if a deprecation is documented for 10 years, people only start to pay attention when the compilation actually fails.
Not *all* people.
I don't think you can count the projects that take the warning into account. You only see the projects that ignore the warning until it's too late. If you remove things without a warning, you're not even giving anyone a fair chance to update their code.
Le jeu. 4 juin 2020 à 16:46, Petr Viktorin <encukou@gmail.com> a écrit :
On 2020-06-04 15:56, Victor Stinner wrote:
Hi Petr,
(I already replied to some of your questions in my reply to Hugh.)
Le jeu. 4 juin 2020 à 12:09, Petr Viktorin <encukou@gmail.com> a écrit :
I see that the current documentation of Py_TYPE, Py_REFCNT ans Py_SIZE suggest that using them as lvalues is perfectly valid. There is no mention that anything changed.
Oh, it seems like https://docs.python.org/dev/ is no longer updated for an unknown reason.
The documentation was updated.
If you look into the up to date Git repository, Py_TYPE() got a ".. versionchanged:: 3.10" markup, and the change is mentioned in "Porting to Python 3.10: C API Changes" section of What's New in Python 3.10. https://github.com/python/cpython/blob/master/Doc/c-api/structures.rst
AFAIK, the deprecation period should be at least a release. Wouldn't that translate to documenting the new behavior 3.10, but only actually breaking in 3.11?
-- Night gathers, and now my watch begins. It shall not end until my death.
Changing the subject because I think it's worth talking about over arching goals and principles without too much implementation detail. (Although obviously goals must be achievable too.)
I suggest that the backwards compatibility WHATEVER has to be shipped with standard Python, ie the link you click on on the main www.python.org website.
And any code or build setting changes are made by the extension author should likewise be made at the source, and not require any action by distributors or users.
The goal of backwards compatibility is not to break things, to keep existing code working for as long as practical. So the necessary work should be done by as few as people as possible. Someone building a standard OS image for their workplace, or maintaining the local wheel repository for the organisation, should not have to do anything. Even Linux distro packagers won't thank us for adding one more step that has to be done.
--
cheers,
Hugh FisherHaving read my own email about goals, I realise my previous suggestion was all wrong.
The backwards compatibility header file should be Python.h! The new AP is in python_api.h or whatever. Existing code doesn't change and still works fine, code updated to the new API is clearly marked as such.
Python.h goes through two or three stages over the next few releases.
- Compiling with Python.h generates a single warning, the initial reminder that you should update your code sometime. That might be the only change.
- Python.h is implemented largely or entirely using the new API. Warning(s) are louder and more visible.
- Python.h becomes one line #error "UPDATE YOUR CODE"
And a good first step for the new python_api.h will be the backwards compatibility library, so developers can test that the new thing really does provide the capabilities of the old.
--
cheers,
Hugh FisherI'm not sure that we are talking about the same thing. Let's take an example. Python 3.9 added the PyFrame_GetCode() function. In the future, PyFrameObject may become an opaque structure, and so all C extensions accessing PyFrameObject would have to replace "f->f_code" with PyFrame_GetCode() (and fix reference counting, since PyFrame_GetCode() returns a strong reference.
https://github.com/pythoncapi/pythoncapi_compat/blob/master/pythoncapi_compa...
Python.h of Python 3.9 do add the new PyFrame_GetCode() function.
My question is how to provide PyFrame_GetCode() on Python 3.8 and older, so a C extension modified to use PyFrame_GetCode() can now be compiled on Python 3.9 (and newer) *and* Python 3.8 (and older). Python version N cannot depend on Python version N+1.
My proposition is to provide a pythoncapi_compat.h header file separately. PyFrame_GetCode() is implemented as static inline function:
#if PY_VERSION_HEX < 0x030900B1 static inline PyCodeObject* PyFrame_GetCode(PyFrameObject *frame) { assert(frame != NULL); PyCodeObject *code = frame->f_code; assert(code != NULL); Py_INCREF(code); return code; } #endif
https://github.com/pythoncapi/pythoncapi_compat/blob/b5bce9c469b56a3dc9cbc41...
Or you may look at PyObject_GC_IsTracked() function which is simply a new feature, but it's not related to any backward incompatible change. It's also possible to provide PyObject_GC_IsTracked() on Python 3.8 and older by the same way.
Victor
Le ven. 5 juin 2020 à 02:19, Hugh Fisher <hugo.fisher@gmail.com> a écrit :
Having read my own email about goals, I realise my previous suggestion was all wrong.
The backwards compatibility header file should be Python.h! The new AP is in python_api.h or whatever. Existing code doesn't change and still works fine, code updated to the new API is clearly marked as such.
Python.h goes through two or three stages over the next few releases.
- Compiling with Python.h generates a single warning, the initial reminder that you should update your code sometime. That might be the only change.
- Python.h is implemented largely or entirely using the new API. Warning(s) are louder and more visible.
- Python.h becomes one line #error "UPDATE YOUR CODE"
And a good first step for the new python_api.h will be the backwards compatibility library, so developers can test that the new thing really does provide the capabilities of the old.
--
cheers, Hugh Fisher
-- Night gathers, and now my watch begins. It shall not end until my death.
On Fri, Jun 5, 2020 at 10:55 AM Victor Stinner <vstinner@python.org> wrote:
I'm not sure that we are talking about the same thing. Let's take an example. Python 3.9 added the PyFrame_GetCode() function. In the future, PyFrameObject may become an opaque structure, and so all C extensions accessing PyFrameObject would have to replace "f->f_code" with PyFrame_GetCode() (and fix reference counting, since PyFrame_GetCode() returns a strong reference.
https://github.com/pythoncapi/pythoncapi_compat/blob/master/pythoncapi_compa...
Python.h of Python 3.9 do add the new PyFrame_GetCode() function.
My question is how to provide PyFrame_GetCode() on Python 3.8 and older, so a C extension modified to use PyFrame_GetCode() can now be compiled on Python 3.9 (and newer) *and* Python 3.8 (and older). Python version N cannot depend on Python version N+1.
My proposal is that Python.h gets the same treatment as Python 2.7 did, frozen with bug fixes only going forward. After a few years, abandoned.
In this case, if I'd thought of this earlier and it had been adopted, the new PyFrame_GetCode function would be added to python_api.h, not Python.h.
python_api.h itself gets added to previous releases of Python, as far back as possible. For those releases it relies on a backwards compatible library implementing the new API on top of the old.
Developers who don't want to to change over to the new API continue to #include Python.h. There's no pythoncapi_compat.h, as Python.h itself now serves that purpose.
Developers who #include python_api.h are signing up for the new API, and accepting that the transition may be bumpy and with breaking changes for the near future.
--
cheers,
Hugh FisherLe ven. 5 juin 2020 à 06:19, Hugh Fisher <hugo.fisher@gmail.com> a écrit :
My proposal is that Python.h gets the same treatment as Python 2.7 did, frozen with bug fixes only going forward. After a few years, abandoned.
As far as I recall, Python 2 to Python 3 "flag day" like migration using 2to3 was *not* a success. 2to3 was only looking "forward".
The approach which worked was to use six to have a single code base working on Python 2 and Python 3. I'm trying to do the same. The six approach looks "backward" and it was more successful.
Victor
Night gathers, and now my watch begins. It shall not end until my death.
On Fri, Jun 5, 2020 at 7:42 PM Victor Stinner <vstinner@python.org> wrote: [ munch ]
As far as I recall, Python 2 to Python 3 "flag day" like migration using 2to3 was *not* a success. 2to3 was only looking "forward".
The approach which worked was to use six to have a single code base working on Python 2 and Python 3. I'm trying to do the same. The six approach looks "backward" and it was more successful.
The Python 2 to 3 migration was criticised because it *wasn't* backwardly compatible with existing code. As you say, six is popular because then all your existing code still worked.
Unless I have badly misunderstood the discussions going on here for the past few months, the new Python C API *won't* be backward compatible and *will* break existing C extension code. So I think it would be good to have C extension modules that have been updated to the new API clearly identified, by not including Python.h
--
cheers,
Hugh Fisher
Am 4. Juni 2020 15:56:54 MESZ schrieb Victor Stinner:
One option is to copy/paste compatibility macros like Py_SET_SIZE() for old Python versions directly in the C code of these extensions. It avoids depending on something external. It's a variant of having a vendored copy of an hypothetical PythonCompat.h header file.
By the way, it's also how Cython handles backward compatibility.
And IMHO the only really reliable way to do it. Cython is admittedly a bit excessive here (we implement and backport most recent Python features and still support Py2.7), but most user code should have a somewhat small set of forward dependencies like the above. As long as it's clear when these were added, it's easy to copy over code snippets one by one and guard them with PY_VERSION_HEX. Old Python versions don't change anymore, so copying new code into old versions is a very safe thing to do, as long as you then switch to the real thing for the future.
Now, whether this is done via a header file that users can copy over entirely or selectively, or by documenting newly added stuff in a way that makes it easy to find and copy, I don't mind. I personally think it's easier to keep these things under local VC rather than as a dependency, but that might just be me.
I think it could be helpful if CPython maintained such a header file in its repo, with the appropriate version guards in it. Then users can decide whether they want to copy the whole file or just bits from it.
Stefan
I’ve been carefully reading this thread (and the previous threads on the subject), and I’m wondering if we’re not over-engineering the whole backward/forward compatibility thing.
With Python modules there is often dependencies on other modules. Therefore, both backward and forward compatibility is important, otherwise if your module depends on dep1 and dep2, and dep1 has been ported to the new world but dep2 has not you are basically hosed: you cannot take your module to the new world and you cannot remain in the old world. Therefore, things like six were wonderful, because they allowed module creators to have Python 2 compatibility at little developer cost.
But for Python extension modules there is much less interdependency. Actually the only cases I can think of off the top of my head is things in the scipy/numpy area. Because there is much less interdependency it means that the value of backward compatibility is much less. (Note I’m not saying “no value”, only “less value”).
That means that a simple measure, such as supplying a header file that provides backward compatibility back to 3.6 and telling extension module distributors to copy this file into their distribution may be just as good as a more complex solution. And if you want to get fancy you could add a script “genpycompatheader” which creates that include file in future Python distributions.
-- Jack Jansen, <Jack.Jansen@cwi.nl>, http://www.cwi.nl/~jack If I can't dance I don't want to be part of your revolution -- Emma Goldman
On Mon, Jun 8, 2020 at 6:49 AM Jack Jansen <jack.jansen@cwi.nl> wrote: [ munch good points ]
That means that a simple measure, such as supplying a header file that provides backward compatibility back to 3.6 and telling extension module distributors to copy this file into their distribution may be just as good as a more complex solution. And if you want to get fancy you could add a script “genpycompatheader” which creates that include file in future Python distributions.
Something I myself have been guilty of is not specifying *who* backward compatibility is for, because I think there are (at least) two different kinds of developer / user.
To me, backwards compatibility means "I can upgrade to a new version and all my stuff still works". In the case of Python, that should, as far as possible, mean extension modules continue to compile without any changes at all. Nothing to install, no code changes or patches, everything Just Works.
And yes I realise Python has never guaranteed binary compatibility for extension modules from version to version, and sometimes source compatibility has not been guaranteed either. That's why I wrote "as far as possible". And eventually the old C API *will* be broken, and those extension modules that haven't been updated will stop compiling.
My suggestion about freezing Python.h is aimed at this kind of backwards compatibility, not breaking existing stuff for as long as possible.
The other kind of backwards compatibility, as Victor has well explained, is module writers who are updating their code to the new API, but need to still work with the older versions of Python. Even though these developers are by definition willing to make changes, I would still prefer to have everything they need, new headers and functions and whatever else, included in the standard Python source, for both old and new versions.
And my suggestion that the new API be in a renamed header file is to distinguish the first case from the second. It would allow a simple form of dependency analysis: "Has this code been updated to the new API?" becomes "grep Python.h"
--
cheers,
Hugh FisherLe lun. 8 juin 2020 à 05:53, Hugh Fisher <hugo.fisher@gmail.com> a écrit :
And yes I realise Python has never guaranteed binary compatibility for extension modules from version to version, and sometimes source compatibility has not been guaranteed either.
Python provides a stable ABI for 9 years, since Python 3.2 (2011) with the PEP 384 "Defining a Stable ABI". But you have to opt-in for the "limited C API" by defining Py_LIMITED_API macro.
My intent is to slowly move the public C API towards this limited C API, to slowly move all C extensions towards the stable ABI. Using a stable ABI has multiple advantages, explained in my PEP: https://github.com/vstinner/misc/blob/master/cpython/pep-opaque-c-api.rst
Victor
Night gathers, and now my watch begins. It shall not end until my death.
On 2020-06-08 17:04, Victor Stinner wrote:
Le lun. 8 juin 2020 à 05:53, Hugh Fisher <hugo.fisher@gmail.com> a écrit :
And yes I realise Python has never guaranteed binary compatibility for extension modules from version to version, and sometimes source compatibility has not been guaranteed either.
Python provides a stable ABI for 9 years, since Python 3.2 (2011) with the PEP 384 "Defining a Stable ABI". But you have to opt-in for the "limited C API" by defining Py_LIMITED_API macro.
My intent is to slowly move the public C API towards this limited C API, to slowly move all C extensions towards the stable ABI. Using a stable ABI has multiple advantages, explained in my PEP: https://github.com/vstinner/misc/blob/master/cpython/pep-opaque-c-api.rst
Can you make this an actual PEP? Currently it's more of a dollection of notes, not a design document :(
Hi,
Le sam. 6 juin 2020 à 13:50, Stefan Behnel <python_capi@behnel.de> a écrit :
Now, whether this is done via a header file that users can copy over entirely or selectively, or by documenting newly added stuff in a way that makes it easy to find and copy, I don't mind. I personally think it's easier to keep these things under local VC rather than as a dependency, but that might just be me.
I'm fine with people who only copy/paste a few functions that they need, directly in their C code. It's the most convenient option to *install* a C extension. But it may make maintenance harder if more and more compatibility code is needed in the future.
I think it could be helpful if CPython maintained such a header file in its repo, with the appropriate version guards in it. Then users can decide whether they want to copy the whole file or just bits from it.
I propose to start in a separate repository, and we can decide later to move it into the Python repository.
Right now, the scope and goals of this https://github.com/pythoncapi/pythoncapi_compat is still blurry to me :-)
Victor
Night gathers, and now my watch begins. It shall not end until my death.
Hi,
Until we decided how to ship the header file, I created the header file with a short documentation (... README file) and unit tests: https://github.com/pythoncapi/pythoncapi_compat
The header file: https://github.com/pythoncapi/pythoncapi_compat/blob/master/pythoncapi_compa...
The README file currently suggests to copy the header file in your project.
I wrote it for Python 3.6 to Python 3.10. It requires C99 support as defined in PEP 7, especially "static inline functions" and <stdint.h> types like uint64_t.
So far, I only tested it on Linux. I added functions introduced in Python 3.9.
Victor
Le mer. 3 juin 2020 à 15:29, Victor Stinner <vstinner@python.org> a écrit :
Hi,
As I wrote in a previous email, the C API of CPython is evolving slowly to hide more and more implementation details. Sadly, some changes break the backward compatibility. Sometimes, a new function must be used, except that the function doesn't exist in older Python versions.
"Update on C API changes to hide implementation details" https://mail.python.org/archives/list/capi-sig@python.org/thread/HYFQHLGNCRF...
Would it be possible to ship a header file (a single ".h" file) and/or even a dynamic library to provide new functions to old Python versions? I'm not sure if it's possible to ship a header file as a package on PyPI and make it available for C compilers easily without monkey patching distutils. Another option is to encourage Linux distributions to package such header file to make it easy to install on the system.
Another (worse?) option is to advise authors of each C extension module to vendor a copy of the header file in their project and update it from time to time when needed.
At least in C, we can provide most "missing" functions on older Python versions as macros or static inline functions. Python 3.6 now requires a subset C99 with static inline functions.
The situation is very similar to the Python 2 to Python 3 transition. Benjamin Peterson wrote the "six" which became quickly very popular. First, projects vendored a copy of the "six" module as a submodule of their project (src/project/six.py, imported as "project.six"). Then, slowly, the "six" module was shipped by Linux distributions or installed via "pip install six".
Cython already does that internally. A few examples: https://github.com/cython/cython/blob/57b4bdeb88cc18f8e2f89fe9a7cf506d4ee0f8...
#if PY_VERSION_HEX < 0x030200A4 typedef long Py_hash_t; #define __Pyx_PyInt_FromHash_t PyInt_FromLong #define __Pyx_PyInt_AsHash_t __Pyx_PyIndex_AsHash_t #else #define __Pyx_PyInt_FromHash_t PyInt_FromSsize_t #define __Pyx_PyInt_AsHash_t __Pyx_PyIndex_AsSsize_t #endif
#if CYTHON_COMPILING_IN_LIMITED_API #define __Pyx_PyThreadState_Current PyThreadState_Get() #elif !CYTHON_FAST_THREAD_STATE #define __Pyx_PyThreadState_Current PyThreadState_GET() #elif PY_VERSION_HEX >= 0x03060000 //#elif PY_VERSION_HEX >= 0x03050200 // Actually added in 3.5.2, but compiling against that does not guarantee that we get imported there. #define __Pyx_PyThreadState_Current _PyThreadState_UncheckedGet() #elif PY_VERSION_HEX >= 0x03000000 #define __Pyx_PyThreadState_Current PyThreadState_GET() #else #define __Pyx_PyThreadState_Current _PyThreadState_Current #endif
#if PY_VERSION_HEX >= 0x030900A4 #define __Pyx_SET_REFCNT(obj, refcnt) Py_SET_REFCNT(obj, refcnt) #define __Pyx_SET_SIZE(obj, size) Py_SET_SIZE(obj, size) #else #define __Pyx_SET_REFCNT(obj, refcnt) Py_REFCNT(obj) = (refcnt) #define __Pyx_SET_SIZE(obj, size) Py_SIZE(obj) = (size) #endif
Victor
Night gathers, and now my watch begins. It shall not end until my death.
-- Night gathers, and now my watch begins. It shall not end until my death.
participants (6)
-
Gustavo Carneiro -
Hugh Fisher -
Jack Jansen -
Petr Viktorin -
Stefan Behnel -
Victor Stinner