[Python-Dev] ctypes: Configurable bitfield allocation strategy

[Vlad Riscutia]

I recently started looking at some ctypes issues. I dug a bit into
http://bugs.python.org/issue6069 and then I found
http://bugs.python.org/issue11920. They both boil down to the fact that
bitfield allocation is up to the compiler, which is different in GCC and
MSVC. Currently we have hard-coded allocation strategy based on paltform in
cfields.c:

> if (bitsize /* this is a bitfield request */

>        && *pfield_size /* we have a bitfield open */
> #ifdef MS_WIN32
>        /* MSVC, GCC with -mms-bitfields */
>        && dict->size * 8 == *pfield_size
> #else
>        /* GCC */
>        && dict->size * 8 <= *pfield_size
> #endif
>        && (*pbitofs + bitsize) <= *pfield_size) {
>        /* continue bit field */
>        fieldtype = CONT_BITFIELD;
> #ifndef MS_WIN32
>    } else if (bitsize /* this is a bitfield request */
>        && *pfield_size /* we have a bitfield open */
>        && dict->size * 8 >= *pfield_size
>        && (*pbitofs + bitsize) <= dict->size * 8) {
>        /* expand bit field */
>        fieldtype = EXPAND_BITFIELD;
> #endif

So when creating a bitfield structure, it's size can be different on Linux
vs Windows.

class MyStructure(ctypes.BigEndianStructure):
    _pack_      = 1    # aligned to 8 bits, not ctypes default of 32
    _fields_    = [
                   ("Data0",   ctypes.c_uint32, 32),
                   ("Data1",   ctypes.c_uint8, 3),
                   ("Data2",   ctypes.c_uint16, 12),
                  ]

sizeof for above structure is 6 on GCC build and 7 on MSVC build. This leads
to some confusion and issues, because we can't always interop correctly with
code compiled with different compiler than the one Python is compiled with
on the platform. Short term solution is to add a warning in the
documentation about this. Longer term though, I think it would be better to
add a property on the Structure class for configurable allocation strategy,
for example Native (default), GCC, MSVC and when allocating the bitfield,
use given strategy. Native would behave similar to what happens now, but we
would also allow GCC-style allocation on Windows for example and the ability
to extend this if we ever run into similar issues with other compilers. This
shouldn't be too difficult to implement, will be backwards compatible and it
would improve interop. I would like to hear some opinions on this.

Thank you,
Vlad
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20110625/7b4f6fea/attachment.html>