bitfields - short - and xlc compiler
a) hope this is not something you expect to be on -list, if so - my apologies! Getting this message (here using c99 as compiler name, but same issue with xlc as compiler name) c99 -qarch=pwr4 -qbitfields=signed -DNDEBUG -O -I. -IInclude -I./Include -I/data/prj/aixtools/python/python-2.7.11.2/Include -I/data/prj/aixtools/python/python-2.7.11.2 -c /data/prj/aixtools/python/python-2.7.11.2/Modules/_ctypes/_ctypes_test.c -o build/temp.aix-5.3-2.7/data/prj/aixtools/python/python-2.7.11.2/Modules/_ctypes/_ctypes_test.o "/data/prj/aixtools/python/python-2.7.11.2/Modules/_ctypes/_ctypes_test.c", line 387.5: 1506-009 (S) Bit field M must be of type signed int, unsigned int or int. "/data/prj/aixtools/python/python-2.7.11.2/Modules/_ctypes/_ctypes_test.c", line 387.5: 1506-009 (S) Bit field N must be of type signed int, unsigned int or int. "/data/prj/aixtools/python/python-2.7.11.2/Modules/_ctypes/_ctypes_test.c", line 387.5: 1506-009 (S) Bit field O must be of type signed int, unsigned int or int. "/data/prj/aixtools/python/python-2.7.11.2/Modules/_ctypes/_ctypes_test.c", line 387.5: 1506-009 (S) Bit field P must be of type signed int, unsigned int or int. "/data/prj/aixtools/python/python-2.7.11.2/Modules/_ctypes/_ctypes_test.c", line 387.5: 1506-009 (S) Bit field Q must be of type signed int, unsigned int or int. "/data/prj/aixtools/python/python-2.7.11.2/Modules/_ctypes/_ctypes_test.c", line 387.5: 1506-009 (S) Bit field R must be of type signed int, unsigned int or int. "/data/prj/aixtools/python/python-2.7.11.2/Modules/_ctypes/_ctypes_test.c", line 387.5: 1506-009 (S) Bit field S must be of type signed int, unsigned int or int. for: struct BITS { int A: 1, B:2, C:3, D:4, E: 5, F: 6, G: 7, H: 8, I: 9; short M: 1, N: 2, O: 3, P: 4, Q: 5, R: 6, S: 7; }; in short xlC v11 does not like short (xlC v7 might have accepted it, but "32-bit machines were common then". I am guessing that 16-bit is not well liked on 64-bit hw now. reference for xlC v7, where short was (apparently) still accepted: http://www.serc.iisc.ernet.in/facilities/ComputingFacilities/systems/cluster... I am taking this is from xlC v7 documentation from the URL, not because I know it personally. So - my question: if "short" is unacceptable for POWER, or maybe only xlC (not tried with gcc) - how terrible is this, and is it possible to adjust the test so - the test is accurate? I am going to modify the test code so it is struct BITS { signed int A: 1, B:2, C:3, D:4, E: 5, F: 6, G: 7, H: 8, I: 9; unsigned int M: 1, N: 2, O: 3, P: 4, Q: 5, R: 6, S: 7; }; And see what happens - BUT - what does this have for impact on python - assuming that "short" bitfields are not supported? p.s. not submitting this a bug (now) as it may just be that "you" consider it a bug in xlC to not support (signed) short bit fields. p.p.s. Note: xlc, by default, considers bitfields to be unsigned. I was trying to force them to signed with -qbitfields=signed - and I still got messages. So, going back to defaults.
Update: Is this going to be impossible? test_short fails om AIX when using xlC in any case. How terrible is this? ====================================================================== FAIL: test_shorts (ctypes.test.test_bitfields.C_Test) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/prj/aixtools/python/python-2.7.11.2/Lib/ctypes/test/test_bitfields.py", line 48, in test_shorts self.assertEqual((name, i, getattr(b, name)), (name, i, func(byref(b), name))) AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1) First differing element 2: -1 1 - ('M', 1, -1) ? - + ('M', 1, 1) ---------------------------------------------------------------------- Ran 440 tests in 1.538s FAILED (failures=1, skipped=91) Traceback (most recent call last): File "./Lib/test/test_ctypes.py", line 15, in <module> test_main() File "./Lib/test/test_ctypes.py", line 12, in test_main run_unittest(unittest.TestSuite(suites)) File "/data/prj/aixtools/python/python-2.7.11.2/Lib/test/test_support.py", line 1428, in run_unittest _run_suite(suite) File "/data/prj/aixtools/python/python-2.7.11.2/Lib/test/test_support.py", line 1411, in _run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "/data/prj/aixtools/python/python-2.7.11.2/Lib/ctypes/test/test_bitfields.py", line 48, in test_shorts self.assertEqual((name, i, getattr(b, name)), (name, i, func(byref(b), name))) AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1) First differing element 2: -1 1 - ('M', 1, -1) ? - + ('M', 1, 1) On 17-Mar-16 23:31, Michael Felt wrote:
On 2016-03-18 00:56, Michael Felt wrote:
Update: Is this going to be impossible?
From what I've been able to find out, the C89 standard limits bitfields to int, signed int and unsigned int, and the C99 standard added _Bool, although some compilers allow other integer types too. It looks like your compiler doesn't allow those additional types.
On Mar 17, 2016, at 18:35, MRAB <python@mrabarnett.plus.com> wrote:
Yeah, C99 (6.7.2.1) allows "a qualified or unqualified version of _Bool, signed int, unsigned int, or some other implementation-defined type", and same for C11. This means that a compiler could easily allow an implementation-defined type that's identical to and interconvertible with short, say "i16", to be used in bitfields, but not short itself. And yet, gcc still allows short "even in strictly conforming mode" (4.9), and it looks like Clang and Intel do the same. Meanwhile, MSVC specifically says it's illegal ("The type-specifier for the declarator must be unsigned int, signed int, or int") but then defines the semantics (you can't have a 17-bit short, bit fields act as the underlying type when accessed, alignment is forced to a boundary appropriate for the underlying type). They do mention that allowing char and long types is a Microsoft extension, but still nothing about short, even though it's used in most of the examples on the page. Anyway, is the question what ctypes should do? If a platform's compiler allows "short M: 1", especially if it has potentially different alignment than "int M: 1", ctypes on that platform had better make ("M", c_short, 1) match the former, right? So it sounds like you need some configure switch to test that your compiler doesn't allow short bit fields, so your ctypes build at least skips that part of _ctypes_test.c and test_bitfields.py, and maybe even doesn't allow them in Python code.
On 2016-03-18 05:57, Andrew Barnert via Python-Dev wrote: per legacy when compilers did (and GCC still does - verified that when I compile with gcc the test does not signal failure) So, more with regard to c) - is there something I could/should be looking at in Python itself, in order to message that the code is not supported by the compiler?
On Mar 20, 2016, at 09:07, Michael Felt <michael@felt.demon.nl> wrote:
a) this does not look solveable using xlC, and I expect from the comment above re: MSVC, that it will, or should also fail there.
And, imho, if anything is to done, it is a decision to be made by "Python".
Sure, but isn't that exactly why you're posting to this list?
b) aka - it sounds like a defect, at least in the test.
Agreed. But I think the test is reasonable on at least MSVC, gcc, clang, and icc. So what you need is some way to run the test on those compilers, but not on compilers that can't handle it. So it sounds like you need a flag coming from autoconf that can be tested in C (and probably in Python as well) that tells you whether the compiler can handle it. And I don't think there is any such flag. Which means someone would have to add the configure test. And if people who use MSVC, gcc, and clang are all unaffected, I'm guessing that someone would have to be someone who cares about xlC or some other compiler, like you. The alternative would be to just change the docs to make it explicit that using non-int bitfields isn't supported but may work in platform-specific ways. If you got everyone to agree to that, surely you could just remove the tests, right? But if people are actually writing C code that follows the examples on the MSVC bitfield docs page, and need to talk to that code from ctypes, I don't know if it would be acceptable to stop officially supporting that.
On 3/20/2016 4:04 PM, Andrew Barnert via Python-Dev wrote:
Agreed. But I think the test is reasonable on at least MSVC, gcc, clang, and icc. So what you need is some way to run the test on those compilers, but not on compilers that can't handle it.
The test could be conditioned on the compiler used.
platform.python_compiler() 'MSC v.1900 64 bit (AMD64)'
The doc could say something about the feature only being available with certain compilers. -- Terry Jan Reedy
Update: Is this going to be impossible? test_short fails om AIX when using xlC in any case. How terrible is this? ====================================================================== FAIL: test_shorts (ctypes.test.test_bitfields.C_Test) ---------------------------------------------------------------------- Traceback (most recent call last): File "/data/prj/aixtools/python/python-2.7.11.2/Lib/ctypes/test/test_bitfields.py", line 48, in test_shorts self.assertEqual((name, i, getattr(b, name)), (name, i, func(byref(b), name))) AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1) First differing element 2: -1 1 - ('M', 1, -1) ? - + ('M', 1, 1) ---------------------------------------------------------------------- Ran 440 tests in 1.538s FAILED (failures=1, skipped=91) Traceback (most recent call last): File "./Lib/test/test_ctypes.py", line 15, in <module> test_main() File "./Lib/test/test_ctypes.py", line 12, in test_main run_unittest(unittest.TestSuite(suites)) File "/data/prj/aixtools/python/python-2.7.11.2/Lib/test/test_support.py", line 1428, in run_unittest _run_suite(suite) File "/data/prj/aixtools/python/python-2.7.11.2/Lib/test/test_support.py", line 1411, in _run_suite raise TestFailed(err) test.test_support.TestFailed: Traceback (most recent call last): File "/data/prj/aixtools/python/python-2.7.11.2/Lib/ctypes/test/test_bitfields.py", line 48, in test_shorts self.assertEqual((name, i, getattr(b, name)), (name, i, func(byref(b), name))) AssertionError: Tuples differ: ('M', 1, -1) != ('M', 1, 1) First differing element 2: -1 1 - ('M', 1, -1) ? - + ('M', 1, 1) On 17-Mar-16 23:31, Michael Felt wrote:
On 2016-03-18 00:56, Michael Felt wrote:
Update: Is this going to be impossible?
From what I've been able to find out, the C89 standard limits bitfields to int, signed int and unsigned int, and the C99 standard added _Bool, although some compilers allow other integer types too. It looks like your compiler doesn't allow those additional types.
On Mar 17, 2016, at 18:35, MRAB <python@mrabarnett.plus.com> wrote:
Yeah, C99 (6.7.2.1) allows "a qualified or unqualified version of _Bool, signed int, unsigned int, or some other implementation-defined type", and same for C11. This means that a compiler could easily allow an implementation-defined type that's identical to and interconvertible with short, say "i16", to be used in bitfields, but not short itself. And yet, gcc still allows short "even in strictly conforming mode" (4.9), and it looks like Clang and Intel do the same. Meanwhile, MSVC specifically says it's illegal ("The type-specifier for the declarator must be unsigned int, signed int, or int") but then defines the semantics (you can't have a 17-bit short, bit fields act as the underlying type when accessed, alignment is forced to a boundary appropriate for the underlying type). They do mention that allowing char and long types is a Microsoft extension, but still nothing about short, even though it's used in most of the examples on the page. Anyway, is the question what ctypes should do? If a platform's compiler allows "short M: 1", especially if it has potentially different alignment than "int M: 1", ctypes on that platform had better make ("M", c_short, 1) match the former, right? So it sounds like you need some configure switch to test that your compiler doesn't allow short bit fields, so your ctypes build at least skips that part of _ctypes_test.c and test_bitfields.py, and maybe even doesn't allow them in Python code.
On 2016-03-18 05:57, Andrew Barnert via Python-Dev wrote: per legacy when compilers did (and GCC still does - verified that when I compile with gcc the test does not signal failure) So, more with regard to c) - is there something I could/should be looking at in Python itself, in order to message that the code is not supported by the compiler?
On Mar 20, 2016, at 09:07, Michael Felt <michael@felt.demon.nl> wrote:
a) this does not look solveable using xlC, and I expect from the comment above re: MSVC, that it will, or should also fail there.
And, imho, if anything is to done, it is a decision to be made by "Python".
Sure, but isn't that exactly why you're posting to this list?
b) aka - it sounds like a defect, at least in the test.
Agreed. But I think the test is reasonable on at least MSVC, gcc, clang, and icc. So what you need is some way to run the test on those compilers, but not on compilers that can't handle it. So it sounds like you need a flag coming from autoconf that can be tested in C (and probably in Python as well) that tells you whether the compiler can handle it. And I don't think there is any such flag. Which means someone would have to add the configure test. And if people who use MSVC, gcc, and clang are all unaffected, I'm guessing that someone would have to be someone who cares about xlC or some other compiler, like you. The alternative would be to just change the docs to make it explicit that using non-int bitfields isn't supported but may work in platform-specific ways. If you got everyone to agree to that, surely you could just remove the tests, right? But if people are actually writing C code that follows the examples on the MSVC bitfield docs page, and need to talk to that code from ctypes, I don't know if it would be acceptable to stop officially supporting that.
On 3/20/2016 4:04 PM, Andrew Barnert via Python-Dev wrote:
Agreed. But I think the test is reasonable on at least MSVC, gcc, clang, and icc. So what you need is some way to run the test on those compilers, but not on compilers that can't handle it.
The test could be conditioned on the compiler used.
platform.python_compiler() 'MSC v.1900 64 bit (AMD64)'
The doc could say something about the feature only being available with certain compilers. -- Terry Jan Reedy
participants (4)
-
Andrew Barnert -
Michael Felt -
MRAB -
Terry Reedy