Word size inconsistencies in C extension modules
Hi folks. While working on an in-house application that uses the curses module, we noticed that it didn't work as expected on an AIX system (powerpc 64-bit big-endian LP64), using python 2.3.5. On a hunch, I took a look through the _cursesmodule.c code and noticed the use of PyArg_ParseTuple()'s "l" decoding mode to retrieve a "long" from python into a C type (attr_t) that on AIX is an int. On 64-bit LP64 platforms, sizeof(long) > sizeof(int), so this doesn't quite work, especially on big-endian systems. Further research into curses shows that different platforms use a different underlying C type for the attr_t type (int, unsigned int, long, unsigned long), so changing the PyArg_ParseTuple() to using the "i" decoding mode probably wasn't portable. I documented this problem and provided a patch that fixes it against the head of the svn trunk in http://bugs.python.org/issue1114 (because the problem appears to still exist in the latest code.) My workaround was to use a separate explicit C "long" to decode the value from python into, and then just assign that to the final value and hope that the type promotion does the right thing on the native platfomr. My questions are: (a) What's the "preferred" style in python extension modules of parsing a number from python into a C type, where the C type size may change on different platforms? Is my method of guessing what the largest common size will be (long, unsigned long, ...), reading into that, and assigning to the final type, acceptable? (b) Is there a desire to see the standard python C extension modules cleaned up to use the answer to (a), especially where said modules may be susceptable to the word size problems I mentioned? (64bit big-endian platforms such as powerpc and sparc64 are good for detecting word-size lossage) cheers, Luke.
(a) What's the "preferred" style in python extension modules of parsing a number from python into a C type, where the C type size may change on different platforms? Is my method of guessing what the largest common size will be (long, unsigned long, ...), reading into that, and assigning to the final type, acceptable?
Yes, that's the best thing we have come up with. You then have the issue on potential truncation on assignment: if the value passed fits into a long (say) but not an attr_t, it would be good if an error was raised. In the past, we have typically coded that ValueError explicitly after the ParseTuple call. In principle, it is possible to deal with these in ParseTuple. To do so: a) in configure.in, make a configure-time check to compute the size of the type, and possibly its signedness. b) in _cursesmodule.c, make a conditional define of ATTR_T_FMT, which would be either "i" or "l" (or #error if it's neither the size of int nor the size of long). Then rely on string concatenation in using that define.
(b) Is there a desire to see the standard python C extension modules cleaned up to use the answer to (a), especially where said modules may be susceptable to the word size problems I mentioned?
Most certainly. There shouldn't be that many places left, though; most have been fixed over the years already. I have a GCC patch which checks for correctness of ParseTuple calls (in terms of data size) if you are interested. Regards, Martin
On Mon, Sep 10, 2007 at 07:37:02AM +0200, "Martin v. L?wis" wrote: | In principle, it is possible to deal with these in ParseTuple. | To do so: | a) in configure.in, make a configure-time check to compute the | size of the type, and possibly its signedness. | b) in _cursesmodule.c, make a conditional define of ATTR_T_FMT, | which would be either "i" or "l" (or #error if it's neither | the size of int nor the size of long). Then rely on string | concatenation in using that define. Are there some good examples in the Python source where this technique has been used already? Or were you proposing a cleaner solution that could be experimented with? | I have a GCC patch which checks for correctness of ParseTuple | calls (in terms of data size) if you are interested. Sounds like a useful variation of the standard -Wformat stuff. This probably wouldn't have helped in the AIX situation I experienced (because the IBM compiler was used in that situation), but it could be useful on other BE LP64 platforms that are more gcc-friendly (e.g, NetBSD/sparc64).
Luke Mewburn schrieb:
On Mon, Sep 10, 2007 at 07:37:02AM +0200, "Martin v. L?wis" wrote: | In principle, it is possible to deal with these in ParseTuple. | To do so: | a) in configure.in, make a configure-time check to compute the | size of the type, and possibly its signedness. | b) in _cursesmodule.c, make a conditional define of ATTR_T_FMT, | which would be either "i" or "l" (or #error if it's neither | the size of int nor the size of long). Then rely on string | concatenation in using that define.
Are there some good examples in the Python source where this technique has been used already?
Not directly. A check for the size of a library type can be found for fpos_t, but there, no ParseTuple depends on it. An example for using variable formatters (though again not for ParseTuple) is PY_FORMAT_SIZE_T.
Or were you proposing a cleaner solution that could be experimented with?
More that, yes.
| I have a GCC patch which checks for correctness of ParseTuple | calls (in terms of data size) if you are interested.
Sounds like a useful variation of the standard -Wformat stuff.
Indeed, it's an extension to it. Unfortunately, introducing new kinds of formats is only possible by editing GCC (and then, the existing framework is focussed on %-style patterns, so I had to bypass that framework as well - but there were hooks for doing so). Regards, Martin
participants (2)
-
"Martin v. Löwis" -
Luke Mewburn