ctypes: Configurable bitfield allocation strategy

I recently started looking at some ctypes issues. I dug a bit into http://bugs.python.org/issue6069 and then I found http://bugs.python.org/issue11920. They both boil down to the fact that bitfield allocation is up to the compiler, which is different in GCC and MSVC. Currently we have hard-coded allocation strategy based on paltform in cfields.c:
if (bitsize /* this is a bitfield request */
So when creating a bitfield structure, it's size can be different on Linux vs Windows. class MyStructure(ctypes.BigEndianStructure): _pack_ = 1 # aligned to 8 bits, not ctypes default of 32 _fields_ = [ ("Data0", ctypes.c_uint32, 32), ("Data1", ctypes.c_uint8, 3), ("Data2", ctypes.c_uint16, 12), ] sizeof for above structure is 6 on GCC build and 7 on MSVC build. This leads to some confusion and issues, because we can't always interop correctly with code compiled with different compiler than the one Python is compiled with on the platform. Short term solution is to add a warning in the documentation about this. Longer term though, I think it would be better to add a property on the Structure class for configurable allocation strategy, for example Native (default), GCC, MSVC and when allocating the bitfield, use given strategy. Native would behave similar to what happens now, but we would also allow GCC-style allocation on Windows for example and the ability to extend this if we ever run into similar issues with other compilers. This shouldn't be too difficult to implement, will be backwards compatible and it would improve interop. I would like to hear some opinions on this. Thank you, Vlad

On 6/25/2011 12:33 PM, Vlad Riscutia wrote:
Just curious, are you saying that this is the 'cause' of the two bug reports, or 'just' something you discovered while investigating them?
Short term solution is to add a warning in the documentation about this.
For 2.7/3.2, yes.
If this would allow the MSVC-compilied Python to better access dlls compiled with gcc (cygwin) on Windows, definitely -- in 3.3. If the new feature is (currently) only useful on Windows, doc should say so. -- Terry Jan Reedy

This is the cause of both bug reports and yes, it should improve interop with GCC-compiled code on Windows. My point is that this is not a platform thing, it's more of a compiler thing. Currently issues appear on Windows for interop between MSVC Python and other GCC code but since bitfield allocation is up to the compiler, it could appear on any other platform due to different compilers being used. On Sat, Jun 25, 2011 at 12:09 PM, Terry Reedy <tjreedy@udel.edu> wrote:

Well it's not really layout, because alignment is handled by pack option. It is how the field gets allocated. At this point I believe it will be more complex to come up with custom allocation option, precisely because it's up to each compiler to allocate the structure. Such flexibility will add a lot of complexity and it might not payoff. This is debatable, but I would go with a 3 option strategy at this point - native, GCC, MSVC and if we actually find interop issues with other compilers we can look into custom allocation. I will try to refactor the C code to more easily accommodate a mode option (while leaving behavior the same for now), then we can decide how the library interface should look like. Thank you, Vlad On Sat, Jun 25, 2011 at 4:37 PM, Greg Ewing <greg.ewing@canterbury.ac.nz>wrote:

Opened http://bugs.python.org/issue12528 to track this. Thank you, Vlad On Sun, Jun 26, 2011 at 5:48 PM, Vlad Riscutia <riscutiavlad@gmail.com>wrote:

Am 25.06.2011 18:33, schrieb Vlad Riscutia:
I would like to hear some opinions on this.
Sounds fine to me in principle. Warnings can be added to the documentation at any time; please submit a patch to the tracker. As for the API change - make sure you post your proposed API to the list before spending time to implement it. Regards, Martin

On 6/25/2011 12:33 PM, Vlad Riscutia wrote:
Just curious, are you saying that this is the 'cause' of the two bug reports, or 'just' something you discovered while investigating them?
Short term solution is to add a warning in the documentation about this.
For 2.7/3.2, yes.
If this would allow the MSVC-compilied Python to better access dlls compiled with gcc (cygwin) on Windows, definitely -- in 3.3. If the new feature is (currently) only useful on Windows, doc should say so. -- Terry Jan Reedy

This is the cause of both bug reports and yes, it should improve interop with GCC-compiled code on Windows. My point is that this is not a platform thing, it's more of a compiler thing. Currently issues appear on Windows for interop between MSVC Python and other GCC code but since bitfield allocation is up to the compiler, it could appear on any other platform due to different compilers being used. On Sat, Jun 25, 2011 at 12:09 PM, Terry Reedy <tjreedy@udel.edu> wrote:

Well it's not really layout, because alignment is handled by pack option. It is how the field gets allocated. At this point I believe it will be more complex to come up with custom allocation option, precisely because it's up to each compiler to allocate the structure. Such flexibility will add a lot of complexity and it might not payoff. This is debatable, but I would go with a 3 option strategy at this point - native, GCC, MSVC and if we actually find interop issues with other compilers we can look into custom allocation. I will try to refactor the C code to more easily accommodate a mode option (while leaving behavior the same for now), then we can decide how the library interface should look like. Thank you, Vlad On Sat, Jun 25, 2011 at 4:37 PM, Greg Ewing <greg.ewing@canterbury.ac.nz>wrote:

Opened http://bugs.python.org/issue12528 to track this. Thank you, Vlad On Sun, Jun 26, 2011 at 5:48 PM, Vlad Riscutia <riscutiavlad@gmail.com>wrote:

Am 25.06.2011 18:33, schrieb Vlad Riscutia:
I would like to hear some opinions on this.
Sounds fine to me in principle. Warnings can be added to the documentation at any time; please submit a patch to the tracker. As for the API change - make sure you post your proposed API to the list before spending time to implement it. Regards, Martin
participants (4)
-
"Martin v. Löwis"
-
Greg Ewing
-
Terry Reedy
-
Vlad Riscutia