Expected stability of PyCode_New() and types.CodeType() signatures

Hello, PEP 570 (Positional-Only Parameters) changed the signatures of PyCode_New() and types.CodeType(), adding a new argument for "posargcount". Our policy for such changes seems to be fragmented tribal knowledge. I'm writing to check if my understanding is reasonable, so I can apply it and document it explicitly. There is a surprisingly large ecosystem of tools that create code objects. The expectation seems to be that these tools will need to be adapted for each minor version of Python. But that's not the same as saying that if you use types.CodeType(), you're on your own. To me [PR 13271], seems to go too far when it adds:
This kind of "washing our hands" doesn't do justice to tools like Cython, whose use of PyCode_New is (IMO) perfectly justified. As is adapting to Python minor versions. So, we should document the changes as any backwards-incompatible change. Specifically, changes should be (or should have been, in hindsight) mentioned in: * The "API changes" and/or "Porting" sections of the What's New document * Version history of the documentation (e.g. versionchanged blocks) * Any relevant PEPs Also, the expected level of API stability should be documented in the docs. Does that sound right? PEP 570: https://www.python.org/dev/peps/pep-0570/ PyCode_New: https://docs.python.org/3.8/c-api/code.html#c.PyCode_New [PR 13271]: https://github.com/python/cpython/pull/13271/files

On 5/31/2019 4:46 AM, Petr Viktorin wrote:
I summarized the above on the issue https://bugs.python.org/issue36896 and suggested specific revisions on the PR. https://github.com/python/cpython/pull/13271
The second PR for Document changes ... added something to What's New. https://bugs.python.org/issue36886 https://github.com/python/cpython/commit/5d23e282af69d404a3430bb95aefe371112... If you think something more is needed, say so on the issue.
* Version history of the documentation (e.g. versionchanged blocks)
We put version history of the code (but not of the docs) in the docs. https://docs.python.org/3/reference/compound_stmts.html#function-definitions should have a version-changed note for the added '/' syntax.
* Any relevant PEPs
If you mean PEPs other than 570, we don't do that.
Also, the expected level of API stability should be documented in the docs.
Yes. I think that part of PR 13271, rewritten, is good.
-- Terry Jan Reedy

As the maintainer of Genshi, one the libraries affected by the CodeType and similar changes, I thought I could add a users perspective to the discussion: Genshi is a templating engine that parses Python code from inside its templates. It supports Python 2.6+, 3.2+, pypy2 and pypy3. It parses Python using compile(...) and then walks the AST to, for example, translate variable lookups into template context lookups. Pretty much every major release of Python has broken Genshi's Python parsing in some way. While on the one hand this is a bit annoying, I also don't want Python to stop evolving where it makes sense. My requests to core developers are largely pragmatic: * Try not to change things unless there's a good reason to (i.e. it makes Python better). * Don't try declare that these things shouldn't be used (there is not much I can do about that now). * Do warn people that these things evolve with the language. * If changes do happen, try make them visible and give a clear description of what has changed. Also many thanks to the core developers who've submitted patches to update Genshi in the past -- that was awesome of you. The new CodeType.replace will remove some potential sources of breakages in the future, so thank you very much for adding that. Schiavo Simon

On 5/31/2019 5:28 PM, Simon Cross wrote:
We try not to. There is already too much work.
* Don't try declare that these things shouldn't be used (there is not much I can do about that now).
That was removed from the PR before it was merged.
* Do warn people that these things evolve with the language.
That was merged.
* If changes do happen, try make them visible and give a clear description of what has changed.
Another chunk was added to What's New.
-- Terry Jan Reedy

On 2019-05-31, Simon Cross wrote:
Thanks. I think this change to PyCode_New() could have been handled a bit better. Couldn't we introduce a new CPP define that enables the revised API of PyCode_New()? For 3.8 extensions, they would get the backwards compatible API (and a warning) unless they set the define. For 3.9, we would enable the new API by default. That gives 3rd party extensions one release cycle to catch up to the change. Perhaps something similar could be done for CodeType called from within Python code. In this case, it seems that introducing a new API like PyCode_NewEx() is not the way. However, just changing an API like that is not very friendly to 3rd party extensions, even if we don't claim it is a stable API. For me, change to PyCode_New() means that I can't test the 3.8.0b1 because the 3rd party extensions I rely on don't compile with it. Normally I try to test my application code with the latest alpha and beta releases. It would be great if we had a system that did CI testing with the top PyPI modules. E.g. pull the latest versions of the top 100 PyPI modules and test them with the latest CPython branch. With that, at least we would know what the fallout would be from some incompatible CPython change. Setting that system up would be a fair amounnt of work but I suspect the PSF could fund someone who puts together a plan to do it. Such a system would be even more useful if we start moving stuff out of stdlib into PyPI.

Neil Schemenauer schrieb am 08.06.19 um 22:46:
FWIW, travis-ci provides the latest CPython master builds (and some latest dev branches). We use them in Cython for our CI tests. One of the problems is that their images include some widely used libraries like NumPy, some of which in turn depend on Cython these days, so it happened once already that they failed to provide updated images because they were lacking a Cython version that worked with them, and we didn't notice that a change in Cython was needed because the CI builds were continuing to use an outdated CPython master version. :) Ah, circular dependencies… I think they fixed something about that, though. It wasn't a problem this time, at least. We also have the "Cython testbed", which we (irregularly) use before releases to check that we didn't break more than was broken before the release. https://travis-ci.org/cython-testbed It's pretty much what was asked for here, just for Cython, and it turns out to be a considerable amount of work to keep this from breaking arbitrarily for the included projects, even without changing something in Cython along the way. Thus, personally, I would prefer a decentralised CI approach, where interested/important projects test themselves against CPython master (which many of them do already), and have them report back when they notice an undermotivated breakage. Some projects do that with Cython (and CPython) already, and that works quite well so far and seems the least work for everyone. Stefan

31.05.19 11:46, Petr Viktorin пише:
I have a related proposition. Yesterday I have reported two bugs (and Pablo quickly fixed them) related to handling positional-only arguments. These bugs were occurred due to subtle changing the meaning of co_argcount. When we make some existing parameters positional-only, we do not add new arguments, but mark existing parameters. But co_argcount now means the only number of positional-or-keyword parameters. Most code which used co_argcount needs now to be changed to use co_posonlyargcount+co_argcount. I propose to make co_argcount meaning the number of positional parameters (i.e. positional-only + positional-or-keyword). This would remove the need of changing the code that uses co_argcount. As for the code object constructor, I propose to make posonlyargcount an optional parameter (default 0) added after existing parameters. PyCode_New() can be kept unchanged, but we can add new PyCode_New2() or PyCode_NewEx() with different signature.

Serhiy Storchaka schrieb am 01.06.19 um 09:02:
Sounds reasonable to me. The main distinction points are positional arguments vs. keyword arguments vs. local variables. Whether the positional ones are positional or positional-only is irrelevant in many cases.
PyCode_New() can be kept unchanged, but we can add new PyCode_New2() or PyCode_NewEx() with different signature.
It's not a commonly used function, and it's easy for C code to adapt. I don't think it's worth adding a new function to the C-API here, compared to just changing the signature. Very few users would benefit, at the cost of added complexity. Stefan

Update. Le ven. 31 mai 2019 à 10:49, Petr Viktorin <encukou@gmail.com> a écrit :
PEP 570 (Positional-Only Parameters) changed the signatures of PyCode_New() and types.CodeType(), adding a new argument for "posargcount".
Pablo proposed a PR to revert PyCode_New() API to Python 3.7 API: https://github.com/python/cpython/pull/13959 I dislike "PyCode_NewEx" name, it will become worse when another parameter will be added in the future. I prefer "With" naming. I prefer "PyCode_NewWithPosArgs()" name. For types.CodeType constructor, well, I added CodeType.replace() to help projects to be prepared for next CodeType constructor change :-) But *I* am fine with breaking this constructor since very few projects create explicitly code objects.
The doc has been updated: https://bugs.python.org/issue36896 "If you instantiate any of these types, note that signatures may vary between Python versions."
For PyCode_New() change, it seems like (almost?) only projects using Cython are affected, no? To not require Cython when they are installed, many projects include C files generated by Cython in their tarball (and other released files). So yeah, the PyCode_New() change is very annoying in practical, since every single project using Cython requires a new release in practice. For this reason, I'm in favor of reverting PyCode_New() API and add a new function with the extra positional only parameter.
Also, the expected level of API stability should be documented in the docs.
Note: code.h header is excluded from the stable API :-) The whole file is surrounded by "#ifndef Py_LIMITED_API". Victor -- Night gathers, and now my watch begins. It shall not end until my death.

Victor Stinner schrieb am 12.06.19 um 00:09:
So yeah, the PyCode_New() change is very annoying in practical, since every single project using Cython requires a new release in practice.
I think Cython's deal with regard to this is: """ If you use Cython, we will try hard to cover up the adaptations for upcoming (and existing) CPython versions for you. But you'll likely have to rerun Cython on your project before a new CPython major release comes out. """ That's obviously not ideal for projects that end up being unmaintained. But then again, you can't freeze time forever, and /any/ change to a dependency can end up being fatal to them. I personally think that rerunning Cython when a new CPython release comes out is an acceptable price to pay for a project. In the worst case, this can even be done by others, as you suggested as a common Fedora practice elsewhere, Victor. (To be fair, I'll have to add that new Cython versions are also part of those dependencies that might end up introducing incompatible changes and forcing your code to be adapted. The upcoming Cython 3.0 release will be such a case. However, given our compatibility record so far, I still consider Cython a very fair deal.) Stefan

On 6/12/19 7:40 AM, Stefan Behnel wrote:
I hope this is something that improvements in Python's packaging story (specifically, PEP 518) should help with. I see the current practice of including Cython's output in releases as a workaround for the fact that you can't (reasonably) specify Cython as a build dependency. Cython is a much lighter dependency than a C compiler -- though a less common one. When there's a reliable way to specify build-time dependencies, running Cython on each build will hopefully become the obvious way to do it. Or is there something else blocking that future?

Petr Viktorin wrote:
PEP 518 lets you specify Cython as a build dependency, but it's still up to _some_ build tool to specify that the project used Cython and then to know how to run Cython as appropriate. So if you're got setuptools working then listing setuptools, wheel, and cython as build dependencies _should_ work as you want.

I'm not one of the people who ever shipped pre-cythonized code, but I think at this point it should be pretty safe to ship just your .pyx files and not the generated C files - at least if your reason for shipping the C code was "it will be hard for people to pip install things". For /other/ reasons, it's best to ship wheels anyway, and pip will prefer wheels over source distributions, so if you ship a manylinux wheel and a Windows wheel for each of the most popular Python versions, the majority of your users won't even ever hit the source distribution anyway. PEP 518 has been supported in pip since I think version 18.0, and PEP 517/518 support in pip landed in 19.0 (though it's been a rocky few months). I do tend to live a bit ahead of the curve in terms of packaging because of my intimate familiarity with the topic, but I do think that we're at the point where the build-related problems you'll see from just shipping `.pyx` files are going to be pretty rare, if you do it right. On 6/12/19 4:11 AM, Petr Viktorin wrote:

Hi Antoine, In Python 3.4, I added "void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)" to the public C API. Problem: we wanted to add a new "calloc" field to PyMemAllocator structure. I chose to rename the structure to PyMemAllocatorEx to ensure that all C extensions using this function get a compilation error, rather than leaving the new "calloc" field uninitialized. The function became: "void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator)" I'm not sure what will be the new structure name when we will add new fields (like malloc_aligned) :-( For PyCodeOptions: does it contain posonlyargcount? If not, how do you pass posonlyargcount. I'm not sure if you plan to handle the backward compatibility. Note: In the PEP 587, I added a private "_config_version" field to PyConfig structure to prepare the structure to future extension, but keep backward compatibility. The structure will be versionned ;-) Victor Le jeu. 13 juin 2019 à 01:08, Antoine Pitrou <solipsis@pitrou.net> a écrit :
-- Night gathers, and now my watch begins. It shall not end until my death.

On Thu, 13 Jun 2019 02:07:54 +0200 Victor Stinner <vstinner@redhat.com> wrote:
The idiom to use it could be: PyCodeOptions opts = PyCodeOptions_INIT; // set attributes here PyObject* code_obj = PyCode_NewEx(opts); So you can change the PyCodeOptions_INIT macro when some new fields are added. Regards Antoine.

On 5/31/2019 4:46 AM, Petr Viktorin wrote:
I summarized the above on the issue https://bugs.python.org/issue36896 and suggested specific revisions on the PR. https://github.com/python/cpython/pull/13271
The second PR for Document changes ... added something to What's New. https://bugs.python.org/issue36886 https://github.com/python/cpython/commit/5d23e282af69d404a3430bb95aefe371112... If you think something more is needed, say so on the issue.
* Version history of the documentation (e.g. versionchanged blocks)
We put version history of the code (but not of the docs) in the docs. https://docs.python.org/3/reference/compound_stmts.html#function-definitions should have a version-changed note for the added '/' syntax.
* Any relevant PEPs
If you mean PEPs other than 570, we don't do that.
Also, the expected level of API stability should be documented in the docs.
Yes. I think that part of PR 13271, rewritten, is good.
-- Terry Jan Reedy

As the maintainer of Genshi, one the libraries affected by the CodeType and similar changes, I thought I could add a users perspective to the discussion: Genshi is a templating engine that parses Python code from inside its templates. It supports Python 2.6+, 3.2+, pypy2 and pypy3. It parses Python using compile(...) and then walks the AST to, for example, translate variable lookups into template context lookups. Pretty much every major release of Python has broken Genshi's Python parsing in some way. While on the one hand this is a bit annoying, I also don't want Python to stop evolving where it makes sense. My requests to core developers are largely pragmatic: * Try not to change things unless there's a good reason to (i.e. it makes Python better). * Don't try declare that these things shouldn't be used (there is not much I can do about that now). * Do warn people that these things evolve with the language. * If changes do happen, try make them visible and give a clear description of what has changed. Also many thanks to the core developers who've submitted patches to update Genshi in the past -- that was awesome of you. The new CodeType.replace will remove some potential sources of breakages in the future, so thank you very much for adding that. Schiavo Simon

On 5/31/2019 5:28 PM, Simon Cross wrote:
We try not to. There is already too much work.
* Don't try declare that these things shouldn't be used (there is not much I can do about that now).
That was removed from the PR before it was merged.
* Do warn people that these things evolve with the language.
That was merged.
* If changes do happen, try make them visible and give a clear description of what has changed.
Another chunk was added to What's New.
-- Terry Jan Reedy

On 2019-05-31, Simon Cross wrote:
Thanks. I think this change to PyCode_New() could have been handled a bit better. Couldn't we introduce a new CPP define that enables the revised API of PyCode_New()? For 3.8 extensions, they would get the backwards compatible API (and a warning) unless they set the define. For 3.9, we would enable the new API by default. That gives 3rd party extensions one release cycle to catch up to the change. Perhaps something similar could be done for CodeType called from within Python code. In this case, it seems that introducing a new API like PyCode_NewEx() is not the way. However, just changing an API like that is not very friendly to 3rd party extensions, even if we don't claim it is a stable API. For me, change to PyCode_New() means that I can't test the 3.8.0b1 because the 3rd party extensions I rely on don't compile with it. Normally I try to test my application code with the latest alpha and beta releases. It would be great if we had a system that did CI testing with the top PyPI modules. E.g. pull the latest versions of the top 100 PyPI modules and test them with the latest CPython branch. With that, at least we would know what the fallout would be from some incompatible CPython change. Setting that system up would be a fair amounnt of work but I suspect the PSF could fund someone who puts together a plan to do it. Such a system would be even more useful if we start moving stuff out of stdlib into PyPI.

Neil Schemenauer schrieb am 08.06.19 um 22:46:
FWIW, travis-ci provides the latest CPython master builds (and some latest dev branches). We use them in Cython for our CI tests. One of the problems is that their images include some widely used libraries like NumPy, some of which in turn depend on Cython these days, so it happened once already that they failed to provide updated images because they were lacking a Cython version that worked with them, and we didn't notice that a change in Cython was needed because the CI builds were continuing to use an outdated CPython master version. :) Ah, circular dependencies… I think they fixed something about that, though. It wasn't a problem this time, at least. We also have the "Cython testbed", which we (irregularly) use before releases to check that we didn't break more than was broken before the release. https://travis-ci.org/cython-testbed It's pretty much what was asked for here, just for Cython, and it turns out to be a considerable amount of work to keep this from breaking arbitrarily for the included projects, even without changing something in Cython along the way. Thus, personally, I would prefer a decentralised CI approach, where interested/important projects test themselves against CPython master (which many of them do already), and have them report back when they notice an undermotivated breakage. Some projects do that with Cython (and CPython) already, and that works quite well so far and seems the least work for everyone. Stefan

31.05.19 11:46, Petr Viktorin пише:
I have a related proposition. Yesterday I have reported two bugs (and Pablo quickly fixed them) related to handling positional-only arguments. These bugs were occurred due to subtle changing the meaning of co_argcount. When we make some existing parameters positional-only, we do not add new arguments, but mark existing parameters. But co_argcount now means the only number of positional-or-keyword parameters. Most code which used co_argcount needs now to be changed to use co_posonlyargcount+co_argcount. I propose to make co_argcount meaning the number of positional parameters (i.e. positional-only + positional-or-keyword). This would remove the need of changing the code that uses co_argcount. As for the code object constructor, I propose to make posonlyargcount an optional parameter (default 0) added after existing parameters. PyCode_New() can be kept unchanged, but we can add new PyCode_New2() or PyCode_NewEx() with different signature.

Serhiy Storchaka schrieb am 01.06.19 um 09:02:
Sounds reasonable to me. The main distinction points are positional arguments vs. keyword arguments vs. local variables. Whether the positional ones are positional or positional-only is irrelevant in many cases.
PyCode_New() can be kept unchanged, but we can add new PyCode_New2() or PyCode_NewEx() with different signature.
It's not a commonly used function, and it's easy for C code to adapt. I don't think it's worth adding a new function to the C-API here, compared to just changing the signature. Very few users would benefit, at the cost of added complexity. Stefan

Update. Le ven. 31 mai 2019 à 10:49, Petr Viktorin <encukou@gmail.com> a écrit :
PEP 570 (Positional-Only Parameters) changed the signatures of PyCode_New() and types.CodeType(), adding a new argument for "posargcount".
Pablo proposed a PR to revert PyCode_New() API to Python 3.7 API: https://github.com/python/cpython/pull/13959 I dislike "PyCode_NewEx" name, it will become worse when another parameter will be added in the future. I prefer "With" naming. I prefer "PyCode_NewWithPosArgs()" name. For types.CodeType constructor, well, I added CodeType.replace() to help projects to be prepared for next CodeType constructor change :-) But *I* am fine with breaking this constructor since very few projects create explicitly code objects.
The doc has been updated: https://bugs.python.org/issue36896 "If you instantiate any of these types, note that signatures may vary between Python versions."
For PyCode_New() change, it seems like (almost?) only projects using Cython are affected, no? To not require Cython when they are installed, many projects include C files generated by Cython in their tarball (and other released files). So yeah, the PyCode_New() change is very annoying in practical, since every single project using Cython requires a new release in practice. For this reason, I'm in favor of reverting PyCode_New() API and add a new function with the extra positional only parameter.
Also, the expected level of API stability should be documented in the docs.
Note: code.h header is excluded from the stable API :-) The whole file is surrounded by "#ifndef Py_LIMITED_API". Victor -- Night gathers, and now my watch begins. It shall not end until my death.

Victor Stinner schrieb am 12.06.19 um 00:09:
So yeah, the PyCode_New() change is very annoying in practical, since every single project using Cython requires a new release in practice.
I think Cython's deal with regard to this is: """ If you use Cython, we will try hard to cover up the adaptations for upcoming (and existing) CPython versions for you. But you'll likely have to rerun Cython on your project before a new CPython major release comes out. """ That's obviously not ideal for projects that end up being unmaintained. But then again, you can't freeze time forever, and /any/ change to a dependency can end up being fatal to them. I personally think that rerunning Cython when a new CPython release comes out is an acceptable price to pay for a project. In the worst case, this can even be done by others, as you suggested as a common Fedora practice elsewhere, Victor. (To be fair, I'll have to add that new Cython versions are also part of those dependencies that might end up introducing incompatible changes and forcing your code to be adapted. The upcoming Cython 3.0 release will be such a case. However, given our compatibility record so far, I still consider Cython a very fair deal.) Stefan

On 6/12/19 7:40 AM, Stefan Behnel wrote:
I hope this is something that improvements in Python's packaging story (specifically, PEP 518) should help with. I see the current practice of including Cython's output in releases as a workaround for the fact that you can't (reasonably) specify Cython as a build dependency. Cython is a much lighter dependency than a C compiler -- though a less common one. When there's a reliable way to specify build-time dependencies, running Cython on each build will hopefully become the obvious way to do it. Or is there something else blocking that future?

Petr Viktorin wrote:
PEP 518 lets you specify Cython as a build dependency, but it's still up to _some_ build tool to specify that the project used Cython and then to know how to run Cython as appropriate. So if you're got setuptools working then listing setuptools, wheel, and cython as build dependencies _should_ work as you want.

I'm not one of the people who ever shipped pre-cythonized code, but I think at this point it should be pretty safe to ship just your .pyx files and not the generated C files - at least if your reason for shipping the C code was "it will be hard for people to pip install things". For /other/ reasons, it's best to ship wheels anyway, and pip will prefer wheels over source distributions, so if you ship a manylinux wheel and a Windows wheel for each of the most popular Python versions, the majority of your users won't even ever hit the source distribution anyway. PEP 518 has been supported in pip since I think version 18.0, and PEP 517/518 support in pip landed in 19.0 (though it's been a rocky few months). I do tend to live a bit ahead of the curve in terms of packaging because of my intimate familiarity with the topic, but I do think that we're at the point where the build-related problems you'll see from just shipping `.pyx` files are going to be pretty rare, if you do it right. On 6/12/19 4:11 AM, Petr Viktorin wrote:

Hi Antoine, In Python 3.4, I added "void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocator *allocator)" to the public C API. Problem: we wanted to add a new "calloc" field to PyMemAllocator structure. I chose to rename the structure to PyMemAllocatorEx to ensure that all C extensions using this function get a compilation error, rather than leaving the new "calloc" field uninitialized. The function became: "void PyMem_SetAllocator(PyMemAllocatorDomain domain, PyMemAllocatorEx *allocator)" I'm not sure what will be the new structure name when we will add new fields (like malloc_aligned) :-( For PyCodeOptions: does it contain posonlyargcount? If not, how do you pass posonlyargcount. I'm not sure if you plan to handle the backward compatibility. Note: In the PEP 587, I added a private "_config_version" field to PyConfig structure to prepare the structure to future extension, but keep backward compatibility. The structure will be versionned ;-) Victor Le jeu. 13 juin 2019 à 01:08, Antoine Pitrou <solipsis@pitrou.net> a écrit :
-- Night gathers, and now my watch begins. It shall not end until my death.

On Thu, 13 Jun 2019 02:07:54 +0200 Victor Stinner <vstinner@redhat.com> wrote:
The idiom to use it could be: PyCodeOptions opts = PyCodeOptions_INIT; // set attributes here PyObject* code_obj = PyCode_NewEx(opts); So you can change the PyCodeOptions_INIT macro when some new fields are added. Regards Antoine.
participants (10)
-
Antoine Pitrou
-
Brett Cannon
-
Neil Schemenauer
-
Paul Ganssle
-
Petr Viktorin
-
Serhiy Storchaka
-
Simon Cross
-
Stefan Behnel
-
Terry Reedy
-
Victor Stinner