Hi, If you got issues with using the Python C API <Python.h> in C++, please speak up! I'm looking for feedback :-) Extending Python by writing C++ code is now easy with the pybind11 project: https://pybind11.readthedocs.io/ It seems like over the last years, building C++ extensions with the Python C API started to emit more C++ compiler warnings. One explanation may be that converting macros to static inline functions (PEP 670) introduce new warnings, even if the old and the new code is exactly the same. I just discover this issue recently. C and C++ compilers treat static inline functions differently. Macros are treated at legacy code which cannot be fixed, like system headers or old C header files, and so many warnings are made quiet. Static inline functions (defined in header files) are treated as regular code and compilers are more eager to emit warnings. I just modified the Python C API to use C++ static_cast<type>(expr) and reinterpret_cast<type>(expr) if it's used with C++. In C, the regular (type)expr cast (called "old-style cast" by C++ compilers ;-)) is still used as before. I'm also working on adding an unit test to make suite that using the Python C API works with a C++ compiler and doesn't emit compiler warnings: * https://github.com/python/cpython/issues/91321 * https://github.com/python/cpython/pull/32175 In terms of C++ version, it was proposed to target C++11. In the pythoncapi-compat project, I got warnings when the NULL constant is used in static inline functions. I modified the pythoncapi_compat.h header file to use nullptr if used with C++ to fix these compiler warnings. So far, I'm unable to reproduce the issue with <Python.h>, and so I didn't try to address this issue in Python. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
In terms of C++ version, it was proposed to target C++11.
GCC 5 has full C++14 support (one library functionality missing), and so does VS2015 onwards as well as Clang 3.4, see https://en.cppreference.com/w/cpp/compiler_support I doubt that any older compilers are in use _anywhere_ in reasonable numbers that this should constrain the development of CPython. While I don't know who proposed C++11 or where, I'd therefore like to propose to move to _at least_ C++14. Note that https://github.com/python/peps/pull/2309 already bumped the required C-standard to C11, and originally defined this as
The C11 subset are features supported by GCC 8.5, clang 8.0, and MSVC of Visual Studio 2017.
If those versions should be regarded as the lower bounds of compiler support (are they - or anything lower - tested on the build bots...?), then C++17 core language support would automatically fall out of this (there are some stragglers for full stdlib support, especially in clang; but that is usually not an issue). Best H.
While I don't know who proposed C++11 or where, I'd therefore like to propose to move to _at least_ C++14.
What benefits does this have for Python development?
Likewise I can ask what benefits choosing C++11 would have? In general, I think standards and compilers need version hygiene like anything else, just with a larger lag. But even in the standard things get deprecated occasionally, and more importantly: new warnings may get issued, and being able to include CPython into C++ warning-free was one of the points that Victor mentioned specifically.
h.vetinari@gmx.com writes:
While I don't know who proposed C++11 or where, I'd therefore like to propose to move to _at least_ C++14.
What benefits does this have for Python development?
Likewise I can ask what benefits choosing C++11 would have?
Not for me to answer, I'm not a proponent of the change. I'm sure if you read past discussions here and on Discourse you'll find answers from the people who studied the problem carefully. I thought you might have something to add to the conversation, but I guess not? Steve
Not for me to answer, I'm not a proponent of the change. I'm sure if you read past discussions here and on Discourse you'll find answers from the people who studied the problem carefully.
The opening mail proposed C++11 without rationale or references. I did search the archives and discourse before, but nothing stood out, and I don't think an encyclopedic knowledge of past python-dev discussions is a reasonable requirement to comment or propose a variation on its merits.
I thought you might have something to add to the conversation, but I guess not?
I find this tone quite out-of-place. I made a proposal (based on compiler support, version hygiene and compatibility with newer standards), and I'd have been more than happy to hear arguments (like Antoine's) or references for the merits of preferring C++11 (though, again, the point became moot since Victor correctly pointed out we can test against several versions). Still, the insinuation (as it arrives on my end) that I shouldn't participate seems really unnecessary. Best, H.
Hi, The C++ version was discussed in the 2nd link that I gave in my first message: https://github.com/python/cpython/issues/91321 Gregory wrote "If we can conditionally test new things based on C++XX version, accumulating modern issue regression tests seems useful. Otherwise 11 at minimum." Another data point is that I mentioned that pybind11 is an important project for Python ecosystem. This project name and its definition refer to C++11: "pybind11: Seamless operability between C++11 and Python". So far, the only C++ code used in Python is the very recent change that I made to use reinterpret_cast<> and static_cast<> in the Python C API (.h header files). The C code base of Python is, well, written only in C. I mentioned nullptr if we get compiler warnings on the NULL constant in static inline functions. nullptr was added by C++11. I also mentioned C++20 "module" keyword and explained that it doesn't affect the Python C API. Having believes and assumptions about C++ compatibility is one thing. My plan is more about actually *testing it* ;-) Victor On Fri, Apr 29, 2022 at 12:36 AM <h.vetinari@gmx.com> wrote:
Not for me to answer, I'm not a proponent of the change. I'm sure if you read past discussions here and on Discourse you'll find answers from the people who studied the problem carefully.
The opening mail proposed C++11 without rationale or references. I did search the archives and discourse before, but nothing stood out, and I don't think an encyclopedic knowledge of past python-dev discussions is a reasonable requirement to comment or propose a variation on its merits.
I thought you might have something to add to the conversation, but I guess not?
I find this tone quite out-of-place. I made a proposal (based on compiler support, version hygiene and compatibility with newer standards), and I'd have been more than happy to hear arguments (like Antoine's) or references for the merits of preferring C++11 (though, again, the point became moot since Victor correctly pointed out we can test against several versions).
Still, the insinuation (as it arrives on my end) that I shouldn't participate seems really unnecessary.
Best, H. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/7SSI53ED... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.
On Thu, 28 Apr 2022 22:03:25 +0900 "Stephen J. Turnbull" <stephenjturnbull@gmail.com> wrote:
h.vetinari@gmx.com writes:
While I don't know who proposed C++11 or where, I'd therefore like to propose to move to _at least_ C++14.
What benefits does this have for Python development?
Let me second that question as well. I work on Apache Arrow, where the C++ parts require C++11 (and we can't go further than this for now because of R compatibility concerns). We could say that enabling the Python bindings switches the required C++ version to C++14, but that would bring complication for no actual again given that you're not likely to benefit from C++14 features in the header files of a *C* project, are you? Regards Antoine.
Recently, a issue about C++20 compatibility was reported: "The Python library will not compile with a C++2020 compiler because the code uses the reserved “module” keyword" https://github.com/python/cpython/issues/83536 In fact, after a long study, Python C API is *not* affected by this issue. Using "module" remains valid in C++20: see the issue for details. Victor On Thu, Apr 28, 2022 at 5:19 PM Antoine Pitrou <antoine@python.org> wrote:
On Thu, 28 Apr 2022 22:03:25 +0900 "Stephen J. Turnbull" <stephenjturnbull@gmail.com> wrote:
h.vetinari@gmx.com writes:
While I don't know who proposed C++11 or where, I'd therefore like to propose to move to _at least_ C++14.
What benefits does this have for Python development?
Let me second that question as well.
I work on Apache Arrow, where the C++ parts require C++11 (and we can't go further than this for now because of R compatibility concerns). We could say that enabling the Python bindings switches the required C++ version to C++14, but that would bring complication for no actual again given that you're not likely to benefit from C++14 features in the header files of a *C* project, are you?
Regards
Antoine.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BIK3SEBQ... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.
On Thu, 28 Apr 2022 17:52:40 +0200 Victor Stinner <vstinner@python.org> wrote:
Recently, a issue about C++20 compatibility was reported:
"The Python library will not compile with a C++2020 compiler because the code uses the reserved “module” keyword" https://github.com/python/cpython/issues/83536
In fact, after a long study, Python C API is *not* affected by this issue. Using "module" remains valid in C++20: see the issue for details.
I'm not surprised. The C++ committee takes compatibility extremely seriously... Regards Antoine.
Victor
On Thu, Apr 28, 2022 at 5:19 PM Antoine Pitrou <antoine@python.org> wrote:
On Thu, 28 Apr 2022 22:03:25 +0900 "Stephen J. Turnbull" <stephenjturnbull@gmail.com> wrote:
h.vetinari@gmx.com writes:
While I don't know who proposed C++11 or where, I'd therefore like to propose to move to _at least_ C++14.
What benefits does this have for Python development?
Let me second that question as well.
I work on Apache Arrow, where the C++ parts require C++11 (and we can't go further than this for now because of R compatibility concerns). We could say that enabling the Python bindings switches the required C++ version to C++14, but that would bring complication for no actual again given that you're not likely to benefit from C++14 features in the header files of a *C* project, are you?
Regards
Antoine.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/BIK3SEBQ... Code of Conduct: http://python.org/psf/codeofconduct/
I work on Apache Arrow, where the C++ parts require C++11 (and we can't go further than this for now because of R compatibility concerns).
Thanks for the datapoint, that's reasonable of course (though I'll note you're using abseil at least through grpc, and abseil is scheduled to remove C++11 support this year: https://abseil.io/blog/20201001-platforms).
We could say that enabling the Python bindings switches the required C++ version to C++14, but that would bring complication for no actual again given that you're not likely to benefit from C++14 features in the header files of a *C* project, are you?
I get your point, and I agree. My argument was to ensure compatibility with more recent standard versions, and Victor already suggested that several versions could be tested. Best H.
Since we are you talking about tests, we can easily run the tests on multiple C++ versions. But we have to start somewhere, so I propose to start with C++11. More C++ versions can be tested later. Victor On Thu, Apr 28, 2022 at 5:54 AM <h.vetinari@gmx.com> wrote:
In terms of C++ version, it was proposed to target C++11.
GCC 5 has full C++14 support (one library functionality missing), and so does VS2015 onwards as well as Clang 3.4, see https://en.cppreference.com/w/cpp/compiler_support
I doubt that any older compilers are in use _anywhere_ in reasonable numbers that this should constrain the development of CPython.
While I don't know who proposed C++11 or where, I'd therefore like to propose to move to _at least_ C++14.
Note that https://github.com/python/peps/pull/2309 already bumped the required C-standard to C11, and originally defined this as
The C11 subset are features supported by GCC 8.5, clang 8.0, and MSVC of Visual Studio 2017.
If those versions should be regarded as the lower bounds of compiler support (are they - or anything lower - tested on the build bots...?), then C++17 core language support would automatically fall out of this (there are some stragglers for full stdlib support, especially in clang; but that is usually not an issue).
Best H. _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/AGUI6B6W... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.
As a data point, I don't remember that recent versions of CPython brought any particular pain for PyArrow, which is a set of bindings written in Cython around some C++ core library code. Regards Antoine. On Wed, 27 Apr 2022 18:31:13 +0200 Victor Stinner <vstinner@python.org> wrote:
Hi,
If you got issues with using the Python C API <Python.h> in C++, please speak up! I'm looking for feedback :-)
Extending Python by writing C++ code is now easy with the pybind11 project: https://pybind11.readthedocs.io/
It seems like over the last years, building C++ extensions with the Python C API started to emit more C++ compiler warnings. One explanation may be that converting macros to static inline functions (PEP 670) introduce new warnings, even if the old and the new code is exactly the same. I just discover this issue recently. C and C++ compilers treat static inline functions differently. Macros are treated at legacy code which cannot be fixed, like system headers or old C header files, and so many warnings are made quiet. Static inline functions (defined in header files) are treated as regular code and compilers are more eager to emit warnings.
I just modified the Python C API to use C++ static_cast<type>(expr) and reinterpret_cast<type>(expr) if it's used with C++. In C, the regular (type)expr cast (called "old-style cast" by C++ compilers ;-)) is still used as before.
I'm also working on adding an unit test to make suite that using the Python C API works with a C++ compiler and doesn't emit compiler warnings:
* https://github.com/python/cpython/issues/91321 * https://github.com/python/cpython/pull/32175
In terms of C++ version, it was proposed to target C++11.
In the pythoncapi-compat project, I got warnings when the NULL constant is used in static inline functions. I modified the pythoncapi_compat.h header file to use nullptr if used with C++ to fix these compiler warnings. So far, I'm unable to reproduce the issue with <Python.h>, and so I didn't try to address this issue in Python.
Victor
Hi, I merged the best basic tests to make sure that using the Python C API in C++ does not emit compiler warnings: * Code: https://github.com/python/cpython/blob/main/Lib/test/_testcppext.cpp * Test: https://github.com/python/cpython/blob/main/Lib/test/test_cppext.py The code is only built to check for compiler warnings. Later, I plan to build this C++ extension to be able to *execute* it at runtime. For now, a C++ compiler is required to run the Python 3.11 test suite. Maybe it should be made optional, but so far I failed to test if distutils has an available C++ compiler. Slowly, more and more tests can be added. For example, this change fixing compiler warnings when passing "const PyObject*" to Python C API functions adds tests to _testcppext.cpp: https://github.com/python/cpython/pull/92138 Victor
On 5/2/2022 2:21 PM, Victor Stinner wrote:
Maybe it should be made optional, but so far I failed to test if distutils has an available C++ compiler.
Considering the current status of distutils, you should probably put that check in configure or the Makefile and build the extension from there. Then the test suite can check whether it was built or not. Cheers, Steve
On Mon, 2 May 2022 15:21:24 +0200 Victor Stinner <vstinner@python.org> wrote:
Slowly, more and more tests can be added. For example, this change fixing compiler warnings when passing "const PyObject*" to Python C API functions adds tests to _testcppext.cpp: https://github.com/python/cpython/pull/92138
Doesn't passing "const PyObject*" leak implementation details, for example that the reference count does not change? It seems to go counter the objective of making the C API more abstract and more stable. (C++ has the "mutable" keyword for such situat but I don't think C has it yet) Regards Antoine.
Hi Antoine, I'm in favor of making the C API more strict. The limited C API of Python 3.11 no longer cast arguments to PyObject* ! The caller must now cast to PyObject*: see PEP 670 "convert macros to functions". Converting macros to static inline functions (PEP 670) is already an important change. I prefer to not change too many things at the same time. Maybe in Python 3.12 (or later), we can consider requiring to not pass "const PyObject*" in C++: remove the cast dropping constness of the argument. My long term goal is to only use opaque "PyObject*" "handles" in the C API. But for practical reasons, changing the C API must be done slowly, step by step. To reduce the maintenance burden in third party C and C+ extensions. -- IMO it's a bad idea to use "const PyObject*" in the C API. Maybe today, a function doesn't modify the memory. But maybe tomorrow, the implementation will deeply change and will need to modify the memory. In C, using "const" (for something other than const char*) is just not convenient. Victor On Tue, May 3, 2022 at 9:22 AM Antoine Pitrou <antoine@python.org> wrote:
On Mon, 2 May 2022 15:21:24 +0200 Victor Stinner <vstinner@python.org> wrote:
Slowly, more and more tests can be added. For example, this change fixing compiler warnings when passing "const PyObject*" to Python C API
functions
adds tests to _testcppext.cpp: https://github.com/python/cpython/pull/92138
Doesn't passing "const PyObject*" leak implementation details, for example that the reference count does not change? It seems to go counter the objective of making the C API more abstract and more stable.
(C++ has the "mutable" keyword for such situat but I don't think C has it yet)
Regards
Antoine.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/YFD3WOOR... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.
participants (5)
-
Antoine Pitrou -
h.vetinari＠gmx.com -
Stephen J. Turnbull -
Steve Dower -
Victor Stinner