New submission from Meador Inge
I believe this is the better thing to do rather than detailing how GCC and MSVC allocated their bitfields because that would just encourage people to use this feature incorrectly.
So clearly documenting how a feature works will cause people to use the feature incorrectly? I think not. In any case, I agree that documenting the low-level specifics of each compiler's algorithm is too much.
Most bugs opened on bit fields are because people are toying with the underlying buffer and get other results than what they expect.
The issues that I have looked at (issue6069, issue11920, and
issue11920) all involve fundamental misunderstandings of *how* the
structure layout is determined. I don't know if I would generalize
these misunderstanding as "toying with the underlying buffer". Some
times people need to know the exact layout for proper C interop. In
some of the bugs reported folks are casting buffers in an attempt
to discover the structure layout since it is not clearly documented.
The general content of your patch seems reasonable. I will provide
more specific comments shortly.
First, I'm saying "toying with the underlying buffer" because none of the bugs are actual issues of the form "I created this bitfield structure with Python, passed it to C function but C structure was different". That would be a bitfield bug. All of these bugs are people setting raw memory to some bytes, then looking at bitfield members and not seeing what they expect.
Please qualify "all" instead of generalizing. I can point to two issues (issue11990 "I'm generating python code from real c code.", issue12945 "We have raw data packages from some tools. These packages contains bitfields, arrays, simple data and so on.") where C code or raw data was, in fact, involved and the reporters just don't understand what layout algorithm is being used. They may not need to know the specifics of the algorithm, but they *do* need to know if it matches the compiler they are using to do interop or the one that generated the raw data. The reason that we are seeing folks cast raw memory into a cyptes bitfield structure is because they do not understand how the structure layout algorithm works and are trying to figure it out via these examples.
Second, one of your review comments is: "GCC is used for most Unix systems and Microsoft VC++ is used on Windows.". This is not how ctypes works. Ctypes implements the bitfield allocation algorithm itself, it doesn't use the compiler with which it is built. Basically it says #ifdef WIN32 - allocate like VC++ - #else - allocate like GCC. So it doesn't really matter with which compiler you are building Python. It will still do GCC style allocation on Solaris.
I understand how it works. This quote is taken somewhat out of
context as the preceding sentence is important. Perhaps saying GCC-
style and VC++-style would have been more clear. The reason that I
mentioned the compiler used to build Python is that it is an easy
reference point and more times than not the bitfield allocation and
layout *do* match that of the compiler used to build the interpreter.
Anyway, I am fine with dropping the "used to build the Python
interpreter" and going with something similar to what you originally
Also, in general, the compiler used to build the ctypes extension
*does* matter. Look in 'cfield.c' where all of the native alignments
computed at compile time. These alignments affect the structure
layout and are defined by the compiler building the ctypes extension.
Look in 'cfield.c' where all of the native alignments
Well, not *all* the native alignments, but many of them.
Terry J. Reedy