Re: [Numpy-discussion] Questions about the array interface.
--- Scott Gilbert <xscottg@yahoo.com> wrote:
--- Andrew Straw <strawman@astraw.com> wrote:
I feared it was something like that. (No platform independent way to represent special values like nan, inf, and so on.) So I think if we're going to require an encoding character such as '<' or '>' we should also include one that means native which CAN handle these special values... And document why it's needed and why it may get one into trouble.
Let's say we used ']' for big endian native, and '[' for little endian native? Does that just indicate the possible presence of NaNs for Infs in the data?
Adding those codes doesn't have any affect on whether or not libraries can deal with them. I guess I'm not understanding something.
I think I'm understanding my problem in understanding :-). There IS a platform independant way to represent NaNs and Infs. It's pretty clearly spelled out in IEEE-754: http://stevehollasch.com/cgindex/coding/ieeefloat.html I think something we've been assuming is that the array data is basically IEEE-754 compliant (maybe it needs to be byteswapped). If that's not true, then we're going to need some new typecodes. We're not supporting the ability to pass VAX floating point around (Are we????). The problem is that you can't make any safe assumptions about whether your current platform will deal with IEEE-754 data in any predictable way if it contains NaNs or Infs. So additional typecodes won't really solve anything. Tim Peter's explanation is a good representation of Python's official position regarding floating point issues, but a much simpler explanation is possible... The struct module in "standard mode" decodes the data one character at a time and builds a float from them. You can see this in the _PyFloat_Unpack8 function in the floatobject.c file. In other words, this routine probably works on a VAX too (taking a IEEE-754 double and building a VAX floating point as it goes). You can also see the comment in there that says it doesn't handle NaNs or Infs. I don't think we need another indicator for '>' big-endian or '<' for little-endian. Cheers, -Scott
On 09.04.2005, at 01:04, Scott Gilbert wrote:
I think something we've been assuming is that the array data is basically IEEE-754 compliant (maybe it needs to be byteswapped). If that's not true, then we're going to need some new typecodes. We're not supporting the ability to pass VAX floating point around (Are we????).
This discussion has been coming up regularly for a few years. Until now the concensus has always been that Python should make no assumptions that go beyond what a C compiler can promise. Which means no assumptions about floating-point representation. Of course the computing world is changing, and IEEE format may well be ubiquitous by now. Vaxes must be in the museum by now. But how about mainframes? IBM mainframes didn't use IEEE when I used them (last time 15 years ago), and they are still around, possibly compatible to their ancestors. Another detail to consider is that although most machines use the IEEE representation, hardly any respects the IEEE rules for floating point operations in all detail. In particular, trusting that Inf and NaN will be treated as IEEE postulates is a risky business. Konrad. -- ------------------------------------------------------------------------ ------- Konrad Hinsen Laboratoire Leon Brillouin, CEA Saclay, 91191 Gif-sur-Yvette Cedex, France Tel.: +33-1 69 08 79 25 Fax: +33-1 69 08 82 61 E-Mail: khinsen@cea.fr ------------------------------------------------------------------------ -------
konrad.hinsen@laposte.net wrote:
On 09.04.2005, at 01:04, Scott Gilbert wrote:
I think something we've been assuming is that the array data is basically IEEE-754 compliant (maybe it needs to be byteswapped). If that's not true, then we're going to need some new typecodes. We're not supporting the ability to pass VAX floating point around (Are we????).
No, in moving from the struct modules character codes we are trying to do something more platform independent because it is very likely that different platforms will want to exchange binary data. IEEE-754 is a great standard to build an interface around. Data sharing was the whole reason the standard emerged and a lot of companies got on board.
This discussion has been coming up regularly for a few years. Until now the concensus has always been that Python should make no assumptions that go beyond what a C compiler can promise. Which means no assumptions about floating-point representation.
Of course the computing world is changing, and IEEE format may well be ubiquitous by now. Vaxes must be in the museum by now. But how about mainframes? IBM mainframes didn't use IEEE when I used them (last time 15 years ago), and they are still around, possibly compatible to their ancestors.
I found the following piece, written about 6 years ago interesting: http://www.research.ibm.com/journal/rd/435/schwarz.html Basically, it states that chips in newer IBM mainframes support the IEEE 754 standard.
Another detail to consider is that although most machines use the IEEE representation, hardly any respects the IEEE rules for floating point operations in all detail. In particular, trusting that Inf and NaN will be treated as IEEE postulates is a risky business.
But, this can be handled with platform-dependendent C-code when and if problems arise. -Travis
Here's an email Todd Miller sent me (I hoped he'd send it directly to the list, but I'll forward it. Todd, I hope you don't mind.) Todd Miller wrote:
On Fri, 2005-04-08 at 15:46 -0700, Andrew Straw wrote:
Hi Todd,
Could you join in on this thread? I think you wrote the ieeespecial stuff in numarray, so it's clear you have a much better understanding of the issues than I do...
Cheers! Andrew
My own understanding is limited, but I can say a few things that might make the status of numarray clearer. My assumptions for numarray were that:
1. Floating point values are 32-bit or 64-bit entities which are stored in IEEE-754 format. This is a basic assumption of numarray.ieeespecial so I expect it simply won't work on a VAX. There's no checking for this.
2. The platforms that I care about, AMD/Intel Windows/Linux, PowerPC OS-X, and Ultra-SPARC Solaris, all seem to provide IEEE-754 floating point. ieeespecial has been tested to work there.
3. I viewed IEEE-754 floating point numbers as 32-bit or 64-bit unsigned integers, and contiguous ranges on those integers are used to represent special values like NAN and INF. Platform byte ordering for the IEEE-754 floating point numbers mirrors byte ordering for integers so the ieeespecial NAN detection code works in a cross platform way *and* values exported from one IEEE-754 platform will work with ieeespecial when imported on another. It's important to note that special values are not unique: there is no single NAN value; it's a bit range.
4. numarray leaks IEEE-754 special values out into Python floating point scalars. This may be bad form. I do this because (1) they repr understandably if not in a platform independent way and (2) people need to get at them. I noticed recently that ieeespecial.nan == ieeespecial.nan returns incorrect answers (True!) for Python-2.3 and correct ones (False) for Python-2.4. I haven't looked at what the array version does yet: array(nan) == array(nan). The point to be taken from this is that the level at which numarray ieee special value handling works or doesn't work is really restricted to (1) detecting certain ieee-754 bit ranges (2) the basic behavior of C code for C89 complilers for array code (no guarantees) (3) the behavior of Python itself (improving).
In the context of the array protocol (looking very nice by the way) my thinking is that non-IEEE-754 floating point could be described with bit fields and that the current type codes should mean IEEE-754.
Some minor things I noticed in the array interface:
1. The packing order of bit fields is not clear. In C, my experience is that some compilers pack bit structs towards the higher order bits of an integer, and some towards the lower. More info to clarify that would be helpful.
2. I saw no mention that we're talking about a protocol. I'm sure that's clear to everyone following this discussion closely, but I didn't see it in the spec. It might make sense to allude to the C helper functions and potential for additions to the Python type struct even if they're not spelled out.
Regards, Todd
On Apr 9, 2005, at 9:54 AM, Travis Oliphant wrote:
konrad.hinsen@laposte.net wrote:
On 09.04.2005, at 01:04, Scott Gilbert wrote:
I think something we've been assuming is that the array data is basically IEEE-754 compliant (maybe it needs to be byteswapped). If that's not true, then we're going to need some new typecodes. We're not supporting the ability to pass VAX floating point around (Are we????).
No, in moving from the struct modules character codes we are trying to do something more platform independent because it is very likely that different platforms will want to exchange binary data. IEEE-754 is a great standard to build an interface around. Data sharing was the whole reason the standard emerged and a lot of companies got on board.
This discussion has been coming up regularly for a few years. Until now the concensus has always been that Python should make no assumptions that go beyond what a C compiler can promise. Which means no assumptions about floating-point representation.
Of course the computing world is changing, and IEEE format may well be ubiquitous by now. Vaxes must be in the museum by now. But how about mainframes? IBM mainframes didn't use IEEE when I used them (last time 15 years ago), and they are still around, possibly compatible to their ancestors.
I found the following piece, written about 6 years ago interesting:
http://www.research.ibm.com/journal/rd/435/schwarz.html
Basically, it states that chips in newer IBM mainframes support the IEEE 754 standard.
Another detail to consider is that although most machines use the IEEE representation, hardly any respects the IEEE rules for floating point operations in all detail. In particular, trusting that Inf and NaN will be treated as IEEE postulates is a risky business.
But, this can be handled with platform-dependendent C-code when and if problems arise. -Travis
------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
On Sat, 2005-04-09 at 12:35 -0700, Andrew Straw wrote:
Here's an email Todd Miller sent me (I hoped he'd send it directly to the list, but I'll forward it. Todd, I hope you don't mind.)
No, I don't mind. I intended to send it to the list but left in a rush this morning. Todd
On Fri, 2005-04-08 at 15:46 -0700, Andrew Straw wrote:
Hi Todd,
Could you join in on this thread? I think you wrote the ieeespecial stuff in numarray, so it's clear you have a much better understanding of the issues than I do...
Cheers! Andrew
My own understanding is limited, but I can say a few things that might make the status of numarray clearer. My assumptions for numarray were that:
1. Floating point values are 32-bit or 64-bit entities which are stored in IEEE-754 format. This is a basic assumption of numarray.ieeespecial so I expect it simply won't work on a VAX. There's no checking for this.
2. The platforms that I care about, AMD/Intel Windows/Linux, PowerPC OS-X, and Ultra-SPARC Solaris, all seem to provide IEEE-754 floating point. ieeespecial has been tested to work there.
3. I viewed IEEE-754 floating point numbers as 32-bit or 64-bit unsigned integers, and contiguous ranges on those integers are used to represent special values like NAN and INF. Platform byte ordering for the IEEE-754 floating point numbers mirrors byte ordering for integers so the ieeespecial NAN detection code works in a cross platform way *and* values exported from one IEEE-754 platform will work with ieeespecial when imported on another. It's important to note that special values are not unique: there is no single NAN value; it's a bit range.
4. numarray leaks IEEE-754 special values out into Python floating point scalars. This may be bad form. I do this because (1) they repr understandably if not in a platform independent way and (2) people need to get at them. I noticed recently that ieeespecial.nan == ieeespecial.nan returns incorrect answers (True!) for Python-2.3 and correct ones (False) for Python-2.4. I haven't looked at what the array version does yet: array(nan) == array(nan). The point to be taken from this is that the level at which numarray ieee special value handling works or doesn't work is really restricted to (1) detecting certain ieee-754 bit ranges (2) the basic behavior of C code for C89 complilers for array code (no guarantees) (3) the behavior of Python itself (improving).
In the context of the array protocol (looking very nice by the way) my thinking is that non-IEEE-754 floating point could be described with bit fields and that the current type codes should mean IEEE-754.
Some minor things I noticed in the array interface:
1. The packing order of bit fields is not clear. In C, my experience is that some compilers pack bit structs towards the higher order bits of an integer, and some towards the lower. More info to clarify that would be helpful.
2. I saw no mention that we're talking about a protocol. I'm sure that's clear to everyone following this discussion closely, but I didn't see it in the spec. It might make sense to allude to the C helper functions and potential for additions to the Python type struct even if they're not spelled out.
Regards, Todd
On Apr 9, 2005, at 9:54 AM, Travis Oliphant wrote:
konrad.hinsen@laposte.net wrote:
On 09.04.2005, at 01:04, Scott Gilbert wrote:
I think something we've been assuming is that the array data is basically IEEE-754 compliant (maybe it needs to be byteswapped). If that's not true, then we're going to need some new typecodes. We're not supporting the ability to pass VAX floating point around (Are we????).
No, in moving from the struct modules character codes we are trying to do something more platform independent because it is very likely that different platforms will want to exchange binary data. IEEE-754 is a great standard to build an interface around. Data sharing was the whole reason the standard emerged and a lot of companies got on board.
This discussion has been coming up regularly for a few years. Until now the concensus has always been that Python should make no assumptions that go beyond what a C compiler can promise. Which means no assumptions about floating-point representation.
Of course the computing world is changing, and IEEE format may well be ubiquitous by now. Vaxes must be in the museum by now. But how about mainframes? IBM mainframes didn't use IEEE when I used them (last time 15 years ago), and they are still around, possibly compatible to their ancestors.
I found the following piece, written about 6 years ago interesting:
http://www.research.ibm.com/journal/rd/435/schwarz.html
Basically, it states that chips in newer IBM mainframes support the IEEE 754 standard.
Another detail to consider is that although most machines use the IEEE representation, hardly any respects the IEEE rules for floating point operations in all detail. In particular, trusting that Inf and NaN will be treated as IEEE postulates is a risky business.
But, this can be handled with platform-dependendent C-code when and if problems arise. -Travis
------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
------------------------------------------------------- SF email is sponsored by - The IT Product Guide Read honest & candid reviews on hundreds of IT Products from real users. Discover which products truly live up to the hype. Start reading now. http://ads.osdn.com/?ad_id=6595&alloc_id=14396&op=click _______________________________________________ Numpy-discussion mailing list Numpy-discussion@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/numpy-discussion
On Apr 9, 2005, at 18:54, Travis Oliphant wrote:
No, in moving from the struct modules character codes we are trying to do something more platform independent because it is very likely that different platforms will want to exchange binary data. IEEE-754 is a great standard to build
For data exchange between platforms, i.e. through files and network connections, XDR is arguably a better choice. It actually uses IEEE for floats, but XDR libraries provide conversion code for other platforms. It also takes care of byte ordering.
an interface around. Data sharing was the whole reason the standard emerged and a lot of companies got on board.
I think the main reason was standardization of precision, range, and operations, to make floating-point code more portable. This has had moderate success, as 100% IEEE platforms are rare if they exist at all.
Another detail to consider is that although most machines use the IEEE representation, hardly any respects the IEEE rules for floating point operations in all detail. In particular, trusting that Inf and NaN will be treated as IEEE postulates is a risky business.
But, this can be handled with platform-dependendent C-code when and if problems arise.
Can it? I have faint memories about Tim Peters explaining why and how handling IEEE in C code is a pain. Anyway, it would be a good idea to get his opinion on whatever proposal about IEEE before implementing it. Konrad.
participants (5)
-
Andrew Straw -
konrad.hinsen＠laposte.net -
Scott Gilbert -
Todd Miller -
Travis Oliphant