Why aren't we allowing the use of C11?
Hi everyone, PEP 7 says that C code should conform to C89 with a subset of C99 allowed. It's 2021 and all the major compilers support C11 (ignoring the optional parts). C11 has support for thread locals, static asserts, and anonymous structs and unions. All useful features. Is there a good reason not to start using C11 now? Cheers, Mark.
On Thu, Jan 28, 2021 at 8:31 AM Mark Shannon <mark@hotpy.org> wrote:
Hi everyone,
PEP 7 says that C code should conform to C89 with a subset of C99 allowed. It's 2021 and all the major compilers support C11 (ignoring the optional parts).
C11 has support for thread locals, static asserts, and anonymous structs and unions. All useful features.
Is there a good reason not to start using C11 now?
Depends on compiler support. I would say that if clang, gcc, and MSVC all support it then we should consider switching, but if any of them are lagging we would need to find the common base line (hence why the subset of C99 when we last talked about this).
On Thu, 28 Jan 2021 16:26:37 +0000 Mark Shannon <mark@hotpy.org> wrote:
Hi everyone,
PEP 7 says that C code should conform to C89 with a subset of C99 allowed. It's 2021 and all the major compilers support C11 (ignoring the optional parts).
I think that CPython is supposed to compile on non-mainstream compilers too (e.g. vendor-specific compilers on embedded platforms). While those are likely to support C99, they may not have caught up with C11.
C11 has support for thread locals, static asserts, and anonymous structs and unions. All useful features.
Native (fast!) thread locals are definitely useful. The other ones you're listing are syntactic sugar. Regards Antoine.
----- Original Message -----
From: "Mark Shannon" <mark@hotpy.org> To: "Python Dev" <python-dev@python.org> Sent: Thursday, January 28, 2021 5:26:37 PM Subject: [Python-Dev] Why aren't we allowing the use of C11?
Hi everyone,
PEP 7 says that C code should conform to C89 with a subset of C99 allowed. It's 2021 and all the major compilers support C11 (ignoring the optional parts).
C11 has support for thread locals, static asserts, and anonymous structs and unions. All useful features.
Is there a good reason not to start using C11 now?
Cheers, Mark.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PLXETSQE... Code of Conduct: http://python.org/psf/codeofconduct/
Depends what platforms the python core developers are willing to support. Currently downstream on e.g. RHEL7 we compile versions of CPython under gcc 4.8.2 which does not support C11. In addition the manylinux2014 base image is also based on CentOS 7, which wouldn't support C11 as well. -- Regards, Charalampos Stratakis Software Engineer Python Maintenance Team, Red Hat
On Thu, Jan 28, 2021 at 10:52 AM Charalampos Stratakis <cstratak@redhat.com> wrote:
----- Original Message -----
From: "Mark Shannon" <mark@hotpy.org> To: "Python Dev" <python-dev@python.org> Sent: Thursday, January 28, 2021 5:26:37 PM Subject: [Python-Dev] Why aren't we allowing the use of C11?
Hi everyone,
PEP 7 says that C code should conform to C89 with a subset of C99 allowed. It's 2021 and all the major compilers support C11 (ignoring the optional parts).
C11 has support for thread locals, static asserts, and anonymous structs and unions. All useful features.
Is there a good reason not to start using C11 now?
Cheers, Mark.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at
https://mail.python.org/archives/list/python-dev@python.org/message/PLXETSQE...
Code of Conduct: http://python.org/psf/codeofconduct/
Depends what platforms the python core developers are willing to support.
Currently downstream on e.g. RHEL7 we compile versions of CPython under gcc 4.8.2 which does not support C11.
In addition the manylinux2014 base image is also based on CentOS 7, which wouldn't support C11 as well.
I *suspect* this is the primary technical reason not to adopt C11 left. But aren't things like manylinux2014 defined by the contents of a centrally maintained docker container? If so (I'm not one who knows how wrong my guess likely is...), can we get those updated to include a more modern compiler so we can move on sooner than the deprecation of manylinux2014? -gps
-- Regards,
Charalampos Stratakis Software Engineer Python Maintenance Team, Red Hat _______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PCOZN5NH... Code of Conduct: http://python.org/psf/codeofconduct/
On Thu, Jan 28, 2021 at 9:03 PM Gregory P. Smith <greg@krypto.org> wrote:
On Thu, Jan 28, 2021 at 10:52 AM Charalampos Stratakis <cstratak@redhat.com> wrote:
----- Original Message -----
From: "Mark Shannon" <mark@hotpy.org> To: "Python Dev" <python-dev@python.org> Sent: Thursday, January 28, 2021 5:26:37 PM Subject: [Python-Dev] Why aren't we allowing the use of C11?
Hi everyone,
PEP 7 says that C code should conform to C89 with a subset of C99 allowed. It's 2021 and all the major compilers support C11 (ignoring the optional parts).
C11 has support for thread locals, static asserts, and anonymous structs and unions. All useful features.
Is there a good reason not to start using C11 now?
Cheers, Mark.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PLXETSQE... Code of Conduct: http://python.org/psf/codeofconduct/
Depends what platforms the python core developers are willing to support.
Currently downstream on e.g. RHEL7 we compile versions of CPython under gcc 4.8.2 which does not support C11.
In addition the manylinux2014 base image is also based on CentOS 7, which wouldn't support C11 as well.
I suspect this is the primary technical reason not to adopt C11 left.
But aren't things like manylinux2014 defined by the contents of a centrally maintained docker container? If so (I'm not one who knows how wrong my guess likely is...), can we get those updated to include a more modern compiler so we can move on sooner than the deprecation of manylinux2014?
RedHat maintains builds of gcc 8.2.1 for CentOS/RHEL 7, that have some clever hacks to guarantee that the resulting binaries will work on CentOS/RHEL 7: https://www.softwarecollections.org/en/scls/rhscl/devtoolset-8/ I'm pretty sure that's what the manylinux2014 image is using. -n -- Nathaniel J. Smith -- https://vorpus.org
----- Original Message -----
From: "Nathaniel Smith" <njs@pobox.com> To: "Gregory P. Smith" <greg@krypto.org> Cc: "Charalampos Stratakis" <cstratak@redhat.com>, "Python Dev" <python-dev@python.org> Sent: Friday, January 29, 2021 6:29:58 AM Subject: Re: [Python-Dev] Re: Why aren't we allowing the use of C11?
On Thu, Jan 28, 2021 at 9:03 PM Gregory P. Smith <greg@krypto.org> wrote:
On Thu, Jan 28, 2021 at 10:52 AM Charalampos Stratakis <cstratak@redhat.com> wrote:
----- Original Message -----
From: "Mark Shannon" <mark@hotpy.org> To: "Python Dev" <python-dev@python.org> Sent: Thursday, January 28, 2021 5:26:37 PM Subject: [Python-Dev] Why aren't we allowing the use of C11?
Hi everyone,
PEP 7 says that C code should conform to C89 with a subset of C99 allowed. It's 2021 and all the major compilers support C11 (ignoring the optional parts).
C11 has support for thread locals, static asserts, and anonymous structs and unions. All useful features.
Is there a good reason not to start using C11 now?
Cheers, Mark.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PLXETSQE... Code of Conduct: http://python.org/psf/codeofconduct/
Depends what platforms the python core developers are willing to support.
Currently downstream on e.g. RHEL7 we compile versions of CPython under gcc 4.8.2 which does not support C11.
In addition the manylinux2014 base image is also based on CentOS 7, which wouldn't support C11 as well.
I suspect this is the primary technical reason not to adopt C11 left.
But aren't things like manylinux2014 defined by the contents of a centrally maintained docker container? If so (I'm not one who knows how wrong my guess likely is...), can we get those updated to include a more modern compiler so we can move on sooner than the deprecation of manylinux2014?
RedHat maintains builds of gcc 8.2.1 for CentOS/RHEL 7, that have some clever hacks to guarantee that the resulting binaries will work on CentOS/RHEL 7: https://www.softwarecollections.org/en/scls/rhscl/devtoolset-8/
I'm pretty sure that's what the manylinux2014 image is using.
-n
-- Nathaniel J. Smith -- https://vorpus.org
That is correct, Software Collections allow that (I'm also one of the maintainers of the Python collection), not sure though if indeed that's the case for manylinux2014. Apart from that though, in general Python is compatible with a wide variety of Platforms and operating systems. Are all those supported platforms compatible with C11 or later? RHEL7 is not (apart from using scl's) for example. If the standard changes that will also render useless a number of buildbots for Python's later branches. -- Regards, Charalampos Stratakis Software Engineer Python Maintenance Team, Red Hat
On 1/28/2021 11:26 AM, Mark Shannon wrote:
PEP 7 says that C code should conform to C89 with a subset of C99 allowed.
As I remember, when the Python C dialect was discussed about 4 years ago, no one seriously proposed moving beyond a subset of C99 addition, because of the state of MSVC.
It's 2021 and all the major compilers support C11 (ignoring the optional parts).
C11 has support for thread locals, static asserts, and anonymous structs and unions. All useful features.
Is there a good reason not to start using C11 now?
Inertia is *a* reason. C11 would require (and allow!) most (nearly all?) people compiling CPython on Windows to upgrade their VS-VC setup. Maybe including some buildbot machines. Judging from the post for https://bugs.python.org/issue42380 this requires VS 16.8 with MSVC 14.28, released last November. Prior distributions of VS 2019 had earlier versions. Last I knew, regular people without a dev license cannot get just MSVC. So we cannot require a compiler most people cannot get. This post otherwise gives additional reasons why upgrading would be good. https://bugs.python.org/issue39952 discusses issues compiling with VS 2019 last spring (compiler not specified. The main blocker was tcl/tk. 3.9 uses 8.6.9; 3.10 is using 8.6.10 and I hope this is upgraded to 3.6.11 before release. The devguide needs updating, as it says to get VS 2017, while https://visualstudio.microsoft.com/ only distributes VS 2019. It appears that C11 would require VS 2019 16.8 or later. https://en.wikipedia.org/wiki/Microsoft_Visual_C%2B%2B#Internal_version_numb... has a table of MSVC versions, _MSC_VERs, and VS 20xx releases. Each of 'VS 2017' and 'VS 2019' are not just one thing but comprise multiple releases/versions, just like 'Windows 10'. -- Terry Jan Reedy
On 2021-01-28 19:20, Terry Reedy wrote:
On 1/28/2021 11:26 AM, Mark Shannon wrote:
PEP 7 says that C code should conform to C89 with a subset of C99 allowed.
As I remember, when the Python C dialect was discussed about 4 years ago, no one seriously proposed moving beyond a subset of C99 addition, because of the state of MSVC.
It's 2021 and all the major compilers support C11 (ignoring the optional parts).
C11 has support for thread locals, static asserts, and anonymous structs and unions. All useful features.
Is there a good reason not to start using C11 now?
Inertia is *a* reason. C11 would require (and allow!) most (nearly all?) people compiling CPython on Windows to upgrade their VS-VC setup. Maybe including some buildbot machines.
Judging from the post for https://bugs.python.org/issue42380 this requires VS 16.8 with MSVC 14.28, released last November. Prior distributions of VS 2019 had earlier versions. Last I knew, regular people without a dev license cannot get just MSVC. So we cannot require a compiler most people cannot get. This post otherwise gives additional reasons why upgrading would be good.
[snip] I have Microsoft Visual Studio Community 2019 Version 16.8.4 (it's free) and it supports C11 and C17.
On Thu, Jan 28, 2021 at 1:31 PM MRAB <python@mrabarnett.plus.com> wrote:
I have Microsoft Visual Studio Community 2019 Version 16.8.4 (it's free) and it supports C11 and C17.
While an upgrade for Community is free, for Pro/Enterprise without an annual license it's not. They've gone 100% subscription now, but people who have outright purchased in the past would now have to make that decision. The other problem is that a large version update can consume a large portion of a day, or even a week if it all goes horribly wrong, especially if your real job is supposed to be building things with it. Then you get to deal with the fun time that is deprecations and new warnings (and potentially new bugs) hitting your other codebases. If you're in a team, it'll have to be a team consensus and effort, since you probably need to be in lockstep. If you're restricted by corporate policy to software that's gone through specific approval processes, that takes even longer. All of this contributes to weighing all the cool new stuff you get against all the irritating everyone who uses it, thus the inertia. In some ways it's a lot easier being a home user who can experiment with the bleeding edge and use whatever they want with their projects, but we don't pay the bills like the big companies with big governance do. -Em
On Thu, Jan 28, 2021, at 22:57, Emily Bowman wrote:
On Thu, Jan 28, 2021 at 1:31 PM MRAB <python@mrabarnett.plus.com> wrote:
I have Microsoft Visual Studio Community 2019 Version 16.8.4 (it's free) and it supports C11 and C17.
While an upgrade for Community is free, for Pro/Enterprise without an annual license it's not.
Something that's hard for me to find a clear answer for in a few minutes of searching: Does the current version of the *Windows SDK* still come with a compiler, and can it be used without the Visual Studio IDE [the situation is confusing because it seems to install through the visual studio installer now] or at least without a license for Visual Studio Pro/Enterprise [i.e. by users who do not qualify for community edition] or with earlier versions of the Visual Studio IDE?
On 29Jan2021 2227, Random832 wrote:
On Thu, Jan 28, 2021, at 22:57, Emily Bowman wrote:
On Thu, Jan 28, 2021 at 1:31 PM MRAB <python@mrabarnett.plus.com> wrote:
I have Microsoft Visual Studio Community 2019 Version 16.8.4 (it's free) and it supports C11 and C17.
While an upgrade for Community is free, for Pro/Enterprise without an annual license it's not.
Something that's hard for me to find a clear answer for in a few minutes of searching:
Does the current version of the *Windows SDK* still come with a compiler, and can it be used without the Visual Studio IDE [the situation is confusing because it seems to install through the visual studio installer now] or at least without a license for Visual Studio Pro/Enterprise [i.e. by users who do not qualify for community edition] or with earlier versions of the Visual Studio IDE?
The Windows SDK contains system headers/libs, no compilers. Visual Studio includes MSVC, and is one of only a few bundles that does (and if you have access to any of the others, you'll know by the massive hole in your bank account, so let's assume Visual Studio). It will also install the Windows SDK on your behalf if the box is checked, which it is by default when you select C++ support. *Anyone* is allow to use Visual Studio Community to work on open-source software. So the only way to not qualify is to refuse to share whatever you're working on. Obviously some people will be in that position, but it's really not that onerous an ask (almost like the GPL, if you squint ;) ). Anyone contributing to CPython is allowed to use VS Community to do so. [1] Additionally, you can install VS Community alongside any other edition or version and they will be kept separate (as much as is possible, which is on Microsoft's side to deal with and shouldn't impact licensing). Finally, the VS Build Tools are only licensed to you if you have a Visual Studio license, which means you need to have qualified for VS Community (or paid for a higher one) to use the build tools. Again, if you're building open-source software, you've qualified, so it's basically a non-issue for us. Hopefully that settles some concerns. IANAL, but I am on the Visual Studio team and have been involved in discussions around how VS Community licensing applies to open source ecosystems ever since it was first created. If you're still concerned, go pay a lawyer to give you more opinions, because you honestly aren't going to find a more informed opinion for free on the internet ;) Cheers, Steve 1: See page 8 of https://visualstudio.microsoft.com/wp-content/uploads/2020/03/Visual-Studio-...
Hi Mark, I tried to use C11 _Thread_local (Thread Local Storage, TLS) only with GCC and clang and I got issues on macOS: https://github.com/python/cpython/pull/23976 My PR uses __thread if _Thread_local is not available. I don't think that MSC (Visual Studio) implements C11 _Thread_local, but it provides __declspec(thread). I tried it and I got issues with DLL. I didn't dig into this issue. MSC loves C++ but seems stuck at C99 (I'm not even sure if it's fully implemented). It seems like declaring a TLS in libpython and using it from an extension module (another dynamic library) is causing issues depending on the linker. It "should" work on macOS, but it doesn't. See https://bugs.python.org/issue40522 for the produced machine code on x86-64. In short, it's usually a single MOV using the FS register (two MOV in the worst case). Victor On Thu, Jan 28, 2021 at 5:29 PM Mark Shannon <mark@hotpy.org> wrote:
Hi everyone,
PEP 7 says that C code should conform to C89 with a subset of C99 allowed. It's 2021 and all the major compilers support C11 (ignoring the optional parts).
C11 has support for thread locals, static asserts, and anonymous structs and unions. All useful features.
Is there a good reason not to start using C11 now?
Cheers, Mark.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PLXETSQE... Code of Conduct: http://python.org/psf/codeofconduct/
-- Night gathers, and now my watch begins. It shall not end until my death.
On 29.01.2021 20:15, Victor Stinner wrote:
Hi Mark,
I tried to use C11 _Thread_local (Thread Local Storage, TLS) only with GCC and clang and I got issues on macOS: https://github.com/python/cpython/pull/23976
My PR uses __thread if _Thread_local is not available.
I don't think that MSC (Visual Studio) implements C11 _Thread_local, but it provides __declspec(thread). I tried it and I got issues with DLL. I didn't dig into this issue. MSC loves C++ but seems stuck at C99 (I'm not even sure if it's fully implemented).
According to https://docs.microsoft.com/en-us/cpp/build/reference/std-specify-language-st... and https://docs.microsoft.com/en-us/cpp/overview/install-c17-support?view=msvc-... , they only partially support C99, and C11 and C17 support is available in the Insider Preview version. The preview version supports all the required but no optional features (including optional features that were required in C99) and suggest to use features test macros. They also say that the 2 standard modes are functonally equivalent except the version macro since C17 is a bugfix release. AFAICS, this means that global C99 requirement is out of the question; C11/C17 can be considered when that MSVC version is released -- but alternative solutions are still needed for code chunks using optional features.
It seems like declaring a TLS in libpython and using it from an extension module (another dynamic library) is causing issues depending on the linker. It "should" work on macOS, but it doesn't.
See https://bugs.python.org/issue40522 for the produced machine code on x86-64. In short, it's usually a single MOV using the FS register (two MOV in the worst case).
Victor
On Thu, Jan 28, 2021 at 5:29 PM Mark Shannon <mark@hotpy.org> wrote:
Hi everyone,
PEP 7 says that C code should conform to C89 with a subset of C99 allowed. It's 2021 and all the major compilers support C11 (ignoring the optional parts).
C11 has support for thread locals, static asserts, and anonymous structs and unions. All useful features.
Is there a good reason not to start using C11 now?
Cheers, Mark.
_______________________________________________ Python-Dev mailing list -- python-dev@python.org To unsubscribe send an email to python-dev-leave@python.org https://mail.python.org/mailman3/lists/python-dev.python.org/ Message archived at https://mail.python.org/archives/list/python-dev@python.org/message/PLXETSQE... Code of Conduct: http://python.org/psf/codeofconduct/
-- Regards, Ivan
On Fri, 29 Jan 2021 18:15:37 +0100 Victor Stinner <vstinner@python.org> wrote:
Hi Mark,
I tried to use C11 _Thread_local (Thread Local Storage, TLS) only with GCC and clang and I got issues on macOS: https://github.com/python/cpython/pull/23976
My PR uses __thread if _Thread_local is not available.
I don't think that MSC (Visual Studio) implements C11 _Thread_local, but it provides __declspec(thread). I tried it and I got issues with DLL. I didn't dig into this issue. MSC loves C++ but seems stuck at C99 (I'm not even sure if it's fully implemented).
It seems like declaring a TLS in libpython and using it from an extension module (another dynamic library) is causing issues depending on the linker. It "should" work on macOS, but it doesn't.
You can hide the access behind a function call. Slightly more costly, but shouldn't be that expensive on modern machines. Regards Antoine.
On Sat, Jan 30, 2021 at 10:37 AM Antoine Pitrou <antoine@python.org> wrote:
You can hide the access behind a function call. Slightly more costly, but shouldn't be that expensive on modern machines.
Oh sure, on the API side, it can be an "opaque" function call (on the ABI side) or static inline function, depending if the call can use TLS or not. If possible, I would prefer to use a faster static inline function in most cases ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death.
On 29Jan2021 1715, Victor Stinner wrote:
It seems like declaring a TLS in libpython and using it from an extension module (another dynamic library) is causing issues depending on the linker. It "should" work on macOS, but it doesn't.
I'm pretty sure this is not defined in any calling convention that would be able to cross dynamic library boundaries, or at least I've never seen it shown how this information would be enecoded. In general, provided the features don't leak out of our compilation process (i.e. cross dynamic library boundaries or show up in public/implicit header files), I don't have any issue allowing them to be used. Though I don't want to speak for the people who maintain CPython distros on smaller/slower/more focused platforms. I'm not sure how best to reach them, but I'd rather do that than find out when they complain on the issue tracker. Only thing I'm certain of is that we shouldn't assume that we know everyone who ever builds CPython. Cheers, Steve
participants (14)
-
Antoine Pitrou -
Brett Cannon -
Charalampos Stratakis -
Emily Bowman -
Eric Snow -
Gregory P. Smith -
Ivan Pozdeev -
Mark Shannon -
MRAB -
Nathaniel Smith -
Random832 -
Steve Dower -
Terry Reedy -
Victor Stinner