[Python-Dev] ctypes: Configurable bitfield allocation strategy

[Terry Reedy]

On 6/25/2011 12:33 PM, Vlad Riscutia wrote:
> I recently started looking at some ctypes issues. I dug a bit into
> http://bugs.python.org/issue6069 and then I found
> http://bugs.python.org/issue11920. They both boil down to the fact that
> bitfield allocation is up to the compiler, which is different in GCC and
> MSVC. Currently we have hard-coded allocation strategy based on paltform
> in cfields.c:
>
>>  if (bitsize /* this is a bitfield request */
>
>>         &&  *pfield_size /* we have a bitfield open */
>>  #ifdef MS_WIN32
>>         /* MSVC, GCC with -mms-bitfields */
>>         &&  dict->size * 8 == *pfield_size
>>  #else
>>         /* GCC */
>>         &&  dict->size * 8<= *pfield_size
>>  #endif
>>         &&  (*pbitofs + bitsize)<= *pfield_size) {
>>         /* continue bit field */
>>         fieldtype = CONT_BITFIELD;
>>  #ifndef MS_WIN32
>>     } else if (bitsize /* this is a bitfield request */
>>         &&  *pfield_size /* we have a bitfield open */
>>         &&  dict->size * 8>= *pfield_size
>>         &&  (*pbitofs + bitsize)<= dict->size * 8) {
>>         /* expand bit field */
>>         fieldtype = EXPAND_BITFIELD;
>>  #endif
>
> So when creating a bitfield structure, it's size can be different on
> Linux vs Windows.
>
> class MyStructure(ctypes.BigEndianStructure):
>      _pack_      = 1    # aligned to 8 bits, not ctypes default of 32
>      _fields_    = [
>                     ("Data0",   ctypes.c_uint32, 32),
>                     ("Data1",   ctypes.c_uint8, 3),
>                     ("Data2",   ctypes.c_uint16, 12),
>                    ]
>
> sizeof for above structure is 6 on GCC build and 7 on MSVC build. This
> leads to some confusion and issues, because we can't always interop
> correctly with code compiled with different compiler than the one Python
> is compiled with on the platform.

Just curious, are you saying that this is the 'cause' of the two bug 
reports, or 'just' something you discovered while investigating them?

 > Short term solution is to add a warning in the documentation about this.

For 2.7/3.2, yes.

 > Longer term though, I think it
> would be better to add a property on the Structure class for
> configurable allocation strategy, for example Native (default), GCC,
> MSVC and when allocating the bitfield, use given strategy. Native would
> behave similar to what happens now, but we would also allow GCC-style
> allocation on Windows for example and the ability to extend this if we
> ever run into similar issues with other compilers. This shouldn't be too
> difficult to implement, will be backwards compatible and it would
> improve interop. I would like to hear some opinions on this.

If this would allow the MSVC-compilied Python to better access dlls 
compiled with gcc (cygwin) on Windows, definitely -- in 3.3.
If the new feature is (currently) only useful on Windows, doc should say so.

-- 
Terry Jan Reedy