ctypes: Configurable bitfield allocation strategy
I recently started looking at some ctypes issues. I dug a bit into http://bugs.python.org/issue6069 and then I found http://bugs.python.org/issue11920. They both boil down to the fact that bitfield allocation is up to the compiler, which is different in GCC and MSVC. Currently we have hard-coded allocation strategy based on paltform in cfields.c:
if (bitsize /* this is a bitfield request */
&& *pfield_size /* we have a bitfield open */#ifdef MS_WIN32 /* MSVC, GCC with -mms-bitfields */ && dict->size * 8 == *pfield_size #else /* GCC */ && dict->size * 8 <= *pfield_size #endif && (*pbitofs + bitsize) <= *pfield_size) { /* continue bit field */ fieldtype = CONT_BITFIELD; #ifndef MS_WIN32 } else if (bitsize /* this is a bitfield request */ && *pfield_size /* we have a bitfield open */ && dict->size * 8 >= *pfield_size && (*pbitofs + bitsize) <= dict->size * 8) { /* expand bit field */ fieldtype = EXPAND_BITFIELD; #endif
So when creating a bitfield structure, it's size can be different on Linux vs Windows.
class MyStructure(ctypes.BigEndianStructure): _pack_ = 1 # aligned to 8 bits, not ctypes default of 32 _fields_ = [ ("Data0", ctypes.c_uint32, 32), ("Data1", ctypes.c_uint8, 3), ("Data2", ctypes.c_uint16, 12), ]
sizeof for above structure is 6 on GCC build and 7 on MSVC build. This leads to some confusion and issues, because we can't always interop correctly with code compiled with different compiler than the one Python is compiled with on the platform. Short term solution is to add a warning in the documentation about this. Longer term though, I think it would be better to add a property on the Structure class for configurable allocation strategy, for example Native (default), GCC, MSVC and when allocating the bitfield, use given strategy. Native would behave similar to what happens now, but we would also allow GCC-style allocation on Windows for example and the ability to extend this if we ever run into similar issues with other compilers. This shouldn't be too difficult to implement, will be backwards compatible and it would improve interop. I would like to hear some opinions on this.
Thank you, Vlad
On 6/25/2011 12:33 PM, Vlad Riscutia wrote:
I recently started looking at some ctypes issues. I dug a bit into http://bugs.python.org/issue6069 and then I found http://bugs.python.org/issue11920. They both boil down to the fact that bitfield allocation is up to the compiler, which is different in GCC and MSVC. Currently we have hard-coded allocation strategy based on paltform in cfields.c:
if (bitsize /* this is a bitfield request */
&& *pfield_size /* we have a bitfield open */#ifdef MS_WIN32 /* MSVC, GCC with -mms-bitfields */ && dict->size * 8 == *pfield_size #else /* GCC */ && dict->size * 8<= *pfield_size #endif && (*pbitofs + bitsize)<= *pfield_size) { /* continue bit field */ fieldtype = CONT_BITFIELD; #ifndef MS_WIN32 } else if (bitsize /* this is a bitfield request */ && *pfield_size /* we have a bitfield open */ && dict->size * 8>= *pfield_size && (*pbitofs + bitsize)<= dict->size * 8) { /* expand bit field */ fieldtype = EXPAND_BITFIELD; #endif
So when creating a bitfield structure, it's size can be different on Linux vs Windows.
class MyStructure(ctypes.BigEndianStructure): _pack_ = 1 # aligned to 8 bits, not ctypes default of 32 _fields_ = [ ("Data0", ctypes.c_uint32, 32), ("Data1", ctypes.c_uint8, 3), ("Data2", ctypes.c_uint16, 12), ]
sizeof for above structure is 6 on GCC build and 7 on MSVC build. This leads to some confusion and issues, because we can't always interop correctly with code compiled with different compiler than the one Python is compiled with on the platform.
Just curious, are you saying that this is the 'cause' of the two bug reports, or 'just' something you discovered while investigating them?
Short term solution is to add a warning in the documentation about this.
For 2.7/3.2, yes.
Longer term though, I think it would be better to add a property on the Structure class for configurable allocation strategy, for example Native (default), GCC, MSVC and when allocating the bitfield, use given strategy. Native would behave similar to what happens now, but we would also allow GCC-style allocation on Windows for example and the ability to extend this if we ever run into similar issues with other compilers. This shouldn't be too difficult to implement, will be backwards compatible and it would improve interop. I would like to hear some opinions on this.
If this would allow the MSVC-compilied Python to better access dlls compiled with gcc (cygwin) on Windows, definitely -- in 3.3. If the new feature is (currently) only useful on Windows, doc should say so.
This is the cause of both bug reports and yes, it should improve interop with GCC-compiled code on Windows. My point is that this is not a platform thing, it's more of a compiler thing. Currently issues appear on Windows for interop between MSVC Python and other GCC code but since bitfield allocation is up to the compiler, it could appear on any other platform due to different compilers being used.
On Sat, Jun 25, 2011 at 12:09 PM, Terry Reedy tjreedy@udel.edu wrote:
On 6/25/2011 12:33 PM, Vlad Riscutia wrote:
I recently started looking at some ctypes issues. I dug a bit into http://bugs.python.org/**issue6069 http://bugs.python.org/issue6069 and then I found http://bugs.python.org/**issue11920 http://bugs.python.org/issue11920. They both boil down to the fact that bitfield allocation is up to the compiler, which is different in GCC and MSVC. Currently we have hard-coded allocation strategy based on paltform in cfields.c:
if (bitsize /* this is a bitfield request */
&& *pfield_size /* we have a bitfield open */#ifdef MS_WIN32 /* MSVC, GCC with -mms-bitfields */ && dict->size * 8 == *pfield_size #else /* GCC */ && dict->size * 8<= *pfield_size #endif && (*pbitofs + bitsize)<= *pfield_size) { /* continue bit field */ fieldtype = CONT_BITFIELD; #ifndef MS_WIN32 } else if (bitsize /* this is a bitfield request */ && *pfield_size /* we have a bitfield open */ && dict->size * 8>= *pfield_size && (*pbitofs + bitsize)<= dict->size * 8) { /* expand bit field */ fieldtype = EXPAND_BITFIELD; #endif
So when creating a bitfield structure, it's size can be different on Linux vs Windows.
class MyStructure(ctypes.**BigEndianStructure): _pack_ = 1 # aligned to 8 bits, not ctypes default of 32 _fields_ = [ ("Data0", ctypes.c_uint32, 32), ("Data1", ctypes.c_uint8, 3), ("Data2", ctypes.c_uint16, 12), ]
sizeof for above structure is 6 on GCC build and 7 on MSVC build. This leads to some confusion and issues, because we can't always interop correctly with code compiled with different compiler than the one Python is compiled with on the platform.
Just curious, are you saying that this is the 'cause' of the two bug reports, or 'just' something you discovered while investigating them?
Short term solution is to add a warning in the documentation about this.
For 2.7/3.2, yes.
Longer term though, I think it
would be better to add a property on the Structure class for configurable allocation strategy, for example Native (default), GCC, MSVC and when allocating the bitfield, use given strategy. Native would behave similar to what happens now, but we would also allow GCC-style allocation on Windows for example and the ability to extend this if we ever run into similar issues with other compilers. This shouldn't be too difficult to implement, will be backwards compatible and it would improve interop. I would like to hear some opinions on this.
If this would allow the MSVC-compilied Python to better access dlls compiled with gcc (cygwin) on Windows, definitely -- in 3.3. If the new feature is (currently) only useful on Windows, doc should say so.
-- Terry Jan Reedy
______________________________**_________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/**mailman/listinfo/python-devhttp://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** riscutiavlad%40gmail.comhttp://mail.python.org/mailman/options/python-dev/riscutiavlad%40gmail.com
Vlad Riscutia wrote:
Longer term though, I think it would be better to add a property on the Structure class for configurable allocation strategy, for example Native (default), GCC, MSVC
It could also be good to have a mode which lets you specify *exactly* how the bits are laid out, independent of any particular compiler.
Well it's not really layout, because alignment is handled by pack option. It is how the field gets allocated. At this point I believe it will be more complex to come up with custom allocation option, precisely because it's up to each compiler to allocate the structure. Such flexibility will add a lot of complexity and it might not payoff. This is debatable, but I would go with a 3 option strategy at this point - native, GCC, MSVC and if we actually find interop issues with other compilers we can look into custom allocation.
I will try to refactor the C code to more easily accommodate a mode option (while leaving behavior the same for now), then we can decide how the library interface should look like.
Thank you, Vlad
On Sat, Jun 25, 2011 at 4:37 PM, Greg Ewing greg.ewing@canterbury.ac.nzwrote:
Vlad Riscutia wrote:
Longer term though, I think it would be better to add a property on the Structure class for configurable allocation strategy, for example Native (default), GCC, MSVC
It could also be good to have a mode which lets you specify *exactly* how the bits are laid out, independent of any particular compiler.
-- Greg
______________________________**_________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/**mailman/listinfo/python-devhttp://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** riscutiavlad%40gmail.comhttp://mail.python.org/mailman/options/python-dev/riscutiavlad%40gmail.com
Opened http://bugs.python.org/issue12528 to track this.
Thank you, Vlad
On Sun, Jun 26, 2011 at 5:48 PM, Vlad Riscutia riscutiavlad@gmail.comwrote:
Well it's not really layout, because alignment is handled by pack option. It is how the field gets allocated. At this point I believe it will be more complex to come up with custom allocation option, precisely because it's up to each compiler to allocate the structure. Such flexibility will add a lot of complexity and it might not payoff. This is debatable, but I would go with a 3 option strategy at this point - native, GCC, MSVC and if we actually find interop issues with other compilers we can look into custom allocation.
I will try to refactor the C code to more easily accommodate a mode option (while leaving behavior the same for now), then we can decide how the library interface should look like.
Thank you, Vlad
On Sat, Jun 25, 2011 at 4:37 PM, Greg Ewing greg.ewing@canterbury.ac.nzwrote:
Vlad Riscutia wrote:
Longer term though, I think it would be better to add a property on the Structure class for configurable allocation strategy, for example Native (default), GCC, MSVC
It could also be good to have a mode which lets you specify *exactly* how the bits are laid out, independent of any particular compiler.
-- Greg
______________________________**_________________ Python-Dev mailing list Python-Dev@python.org http://mail.python.org/**mailman/listinfo/python-devhttp://mail.python.org/mailman/listinfo/python-dev Unsubscribe: http://mail.python.org/**mailman/options/python-dev/** riscutiavlad%40gmail.comhttp://mail.python.org/mailman/options/python-dev/riscutiavlad%40gmail.com
Am 25.06.2011 18:33, schrieb Vlad Riscutia:
I would like to hear some opinions on this.
Sounds fine to me in principle. Warnings can be added to the documentation at any time; please submit a patch to the tracker. As for the API change - make sure you post your proposed API to the list before spending time to implement it.
Regards, Martin
Actually I already attached patch implementing everything to the issue (not too much time spent :)). I'm hoping someone can review.
Thank you, Vlad
On Mon, Jul 11, 2011 at 12:47 AM, "Martin v. Löwis" martin@v.loewis.dewrote:
Am 25.06.2011 18:33, schrieb Vlad Riscutia:
I would like to hear some opinions on this.
Sounds fine to me in principle. Warnings can be added to the documentation at any time; please submit a patch to the tracker. As for the API change - make sure you post your proposed API to the list before spending time to implement it.
Regards, Martin
Anyone care to take a look at this?
Thank you, Vlad
On Mon, Jul 11, 2011 at 8:59 AM, Vlad Riscutia riscutiavlad@gmail.comwrote:
Actually I already attached patch implementing everything to the issue (not too much time spent :)). I'm hoping someone can review.
Thank you, Vlad
On Mon, Jul 11, 2011 at 12:47 AM, "Martin v. Löwis" martin@v.loewis.dewrote:
Am 25.06.2011 18:33, schrieb Vlad Riscutia:
I would like to hear some opinions on this.
Sounds fine to me in principle. Warnings can be added to the documentation at any time; please submit a patch to the tracker. As for the API change - make sure you post your proposed API to the list before spending time to implement it.
Regards, Martin
participants (4)
-
"Martin v. Löwis" -
Greg Ewing -
Terry Reedy -
Vlad Riscutia