Version 2 of PEP 670 – Convert macros to functions in the Python C API
Hi, Since Erlend and me posted PEP 670 on python-dev last October, we took all feedback (python-dev and Steering Council) in account to clarify the intent of the PEP and to reduce its scope (remove *any* risk of backward compatibility). PEP 670: https://python.github.io/peps/pep-0670/ Changes since the first version: * Stricter policy on not changing argument types and return type. * Better explain why pointer arguments require a cast to not emit new compiler warnings. * Macros which can be used as l-values are no longer modified by the PEP (I wrote PEP 674 just for that). * Macros having multiple return types are no longer modified by the PEP. * Limited C API version 3.11 no longer casts pointer arguments (move towards a cleaner API, but only if you opt in for that!). * No longer remove return values of macros "which should not have a return value" (this rule was complicated and in fact, all macros have already been fixed). * Add "Macros converted to functions since Python 3.8" section to show that macros already converted didn't introduce performance issue nor new compiler warnings. * Add "Benchmark comparing macros and static inline functions" section to prove that there is no significant impact on performance. Note: we already submitted PEP 670 to the Steering Council at the end of November, and the SC 2022 came with feedback, so we modified the PEP: https://mail.python.org/archives/list/python-dev@python.org/message/IJ3IBVY3... Following the SC decision, we already modified PEP 7 to add: "static inline functions should be preferred over macros in new code." https://www.python.org/dev/peps/pep-0007/#c-dialect Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 22. 02. 22 13:41, Victor Stinner wrote:
Hi,
Since Erlend and me posted PEP 670 on python-dev last October, we took all feedback (python-dev and Steering Council) in account to clarify the intent of the PEP and to reduce its scope (remove *any* risk of backward compatibility).
I did realize there's one more issue when converting macros or static inline functions to regular functions. Regular functions' bodies aren't guarded by limited API #ifdefs, so if they are part of the limited API it's easy to forget to think about it when changing them. If a macro in the limited API is converted to a regular function, then a test should be added to ensure the old implementation of the macro (i.e. what's compiled into stable ABI extensions) still works. In the PEP, the "Performance and inlining" section seems unnecessary. It talks about attributes that aren't used in the implementation. Or are they? How does the section relate to the rest of the PEP? The "Benchmark comparing macros and static inline functions" section at the end should be enough to explain the impact. [...]> Following the SC decision, we already modified PEP 7 to add:
"static inline functions should be preferred over macros in new code." https://www.python.org/dev/peps/pep-0007/#c-dialect
Nice! Thanks!
On Wed, Feb 23, 2022 at 7:11 PM Petr Viktorin <encukou@gmail.com> wrote:
In the PEP, the "Performance and inlining" section seems unnecessary. It talks about attributes that aren't used in the implementation. Or are they? How does the section relate to the rest of the PEP? The "Benchmark comparing macros and static inline functions" section at the end should be enough to explain the impact.
I added this section to the PEP because some core devs believe that C compilers don't inline static inline functions sometimes and that it causes performance regressions. This section is an answer to that: I checked that static inline functions *are* inlined as expected in practice. The "Benchmark comparing macros and static inline functions" section uses regular benchmarks to confirm that. Forcing the compiler to inline or asking the compiler to not inline has also been discussed multiple times when it was proposed to convert some macros to static inline functions. So I prefer to put it in the PEP to avoid people having to dig into old discussions to have scattered information about that. You may want to dig into links from the PEP to see old discussions. See for example https://bugs.python.org/issue35059 for the first discussion on forcing inline or not. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On Wed, Feb 23, 2022 at 7:11 PM Petr Viktorin <encukou@gmail.com> wrote:
I did realize there's one more issue when converting macros or static inline functions to regular functions. Regular functions' bodies aren't guarded by limited API #ifdefs, so if they are part of the limited API it's easy to forget to think about it when changing them. If a macro in the limited API is converted to a regular function, then a test should be added to ensure the old implementation of the macro (i.e. what's compiled into stable ABI extensions) still works.
Does it problem really belongs to PEP 670 "Convert macros to functions in the Python C API", or is it more something for PEP 652 "Maintaining the Stable ABI"? I don't think that Python 3.11 should keep a copy of Python 3.10 macros: it would increase the maintenance burden, each function would have 2 implementations (3.11 function and 3.10 macro). Also, there would be no warranty that the copied 3.10 macros would remain exactly the same than 3.10 code if someone changes them by mistake directly or indirectly (by changing code used by this macro, changing a compiler flag, etc). Maybe such stable ABI test belongs to an external project building a C extension with the Python 3.10 limited C API (or an older version) and then test it on Python 3.11. IMO it's the reliable way to test the stable ABI: a functional test. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 23. 02. 22 20:15, Victor Stinner wrote:
On Wed, Feb 23, 2022 at 7:11 PM Petr Viktorin <encukou@gmail.com> wrote:
I did realize there's one more issue when converting macros or static inline functions to regular functions. Regular functions' bodies aren't guarded by limited API #ifdefs, so if they are part of the limited API it's easy to forget to think about it when changing them. If a macro in the limited API is converted to a regular function, then a test should be added to ensure the old implementation of the macro (i.e. what's compiled into stable ABI extensions) still works.
Does it problem really belongs to PEP 670 "Convert macros to functions in the Python C API", or is it more something for PEP 652 "Maintaining the Stable ABI"?
PEP 652 is a historical document for Python 3.10. Maybe this should go in the devguide, in a section on how to convert macros/static functions to regular functions?
I don't think that Python 3.11 should keep a copy of Python 3.10 macros: it would increase the maintenance burden, each function would have 2 implementations (3.11 function and 3.10 macro). Also, there would be no warranty that the copied 3.10 macros would remain exactly the same than 3.10 code if someone changes them by mistake directly or indirectly (by changing code used by this macro, changing a compiler flag, etc).
Maybe such stable ABI test belongs to an external project building a C extension with the Python 3.10 limited C API (or an older version) and then test it on Python 3.11. IMO it's the reliable way to test the stable ABI: a functional test.
Maybe. But before we have that kind of test infrastructure, I'm worried that converting limited API macros to regular functions will make it harder to keep the stable ABI working.
participants (2)
-
Petr Viktorin -
Victor Stinner