Hi,
Following the discussion on python 2.6 support for numpy, I tried last svn on mac os X, and I get a number of failures which I don't understand, which seem to be linked to dtype code, more exactly to endianness:
====================================================================== FAIL: test_basic (test_multiarray.TestClip) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/david/pylocal/lib/python2.6/site-packages/numpy/core/tests/test_multiarray.py", line 677, in test_basic self._clip_type('float',1024,-12.8,100.2, inplace=inplace) File "/Users/david/pylocal/lib/python2.6/site-packages/numpy/core/tests/test_multiarray.py", line 671, in _clip_type assert_equal(x.dtype.byteorder,byteorder) File "/Users/david/pylocal/lib/python2.6/site-packages/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: '>' DESIRED: '='
====================================================================== FAIL: test_binary (test_multiarray.TestFromstring) ---------------------------------------------------------------------- Traceback (most recent call last): File "/Users/david/pylocal/lib/python2.6/site-packages/numpy/core/tests/test_multiarray.py", line 120, in test_binary assert_array_equal(a, array([1,2,3,4])) File "/Users/david/pylocal/lib/python2.6/site-packages/numpy/testing/utils.py", line 303, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/Users/david/pylocal/lib/python2.6/site-packages/numpy/testing/utils.py", line 295, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal
(mismatch 100.0%) x: array([ 4.60060299e-41, 8.96831017e-44, 2.30485571e-41, 4.60074312e-41], dtype=float32) y: array([1, 2, 3, 4])
...
http://scipy.org/scipy/numpy/ticket/958
Does anyone have a clue about where to look at ? I am wondering why it only appears on Mac OS X,
David
On Mon, Nov 24, 2008 at 8:18 PM, David Cournapeau cournape@gmail.comwrote:
Hi,
Following the discussion on python 2.6 support for numpy, I tried last svn on mac os X, and I get a number of failures which I don't understand, which seem to be linked to dtype code, more exactly to endianness:
====================================================================== FAIL: test_basic (test_multiarray.TestClip)
Traceback (most recent call last): File "/Users/david/pylocal/lib/python2.6/site-packages/numpy/core/tests/test_multiarray.py", line 677, in test_basic self._clip_type('float',1024,-12.8,100.2, inplace=inplace) File "/Users/david/pylocal/lib/python2.6/site-packages/numpy/core/tests/test_multiarray.py", line 671, in _clip_type assert_equal(x.dtype.byteorder,byteorder) File "/Users/david/pylocal/lib/python2.6/site-packages/numpy/testing/utils.py", line 183, in assert_equal raise AssertionError(msg) AssertionError: Items are not equal: ACTUAL: '>' DESIRED: '='
====================================================================== FAIL: test_binary (test_multiarray.TestFromstring)
Traceback (most recent call last): File "/Users/david/pylocal/lib/python2.6/site-packages/numpy/core/tests/test_multiarray.py", line 120, in test_binary assert_array_equal(a, array([1,2,3,4])) File "/Users/david/pylocal/lib/python2.6/site-packages/numpy/testing/utils.py", line 303, in assert_array_equal verbose=verbose, header='Arrays are not equal') File "/Users/david/pylocal/lib/python2.6/site-packages/numpy/testing/utils.py", line 295, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not equal
(mismatch 100.0%) x: array([ 4.60060299e-41, 8.96831017e-44, 2.30485571e-41, 4.60074312e-41], dtype=float32) y: array([1, 2, 3, 4])
Sure enough, it's byteswapped. Is it correct that:
1) This problem is specific to 2.6 and 2.5 works. 2) It's on Intel hardware?
What about normal doubles, do they work? Does python work? Is it possible there are two versions of (python, lib,...), one for ppc and the other for Intel that are getting confused?
Chuck
On Tue, Nov 25, 2008 at 12:41 PM, Charles R Harris charlesr.harris@gmail.com wrote:
- This problem is specific to 2.6 and 2.5 works.
Yes
- It's on Intel hardware?
Yes.
Here is a minimal test which shows the problem:
import numpy as np assert np.dtype('<f4').isnative
This fails on python 2.6, but works on 2.5. In both cases, sys.byteorder returns little. So for some reasons, dtype description (see other failures) is not working on 2.6.
David
On Mon, Nov 24, 2008 at 9:02 PM, David Cournapeau cournape@gmail.comwrote:
On Tue, Nov 25, 2008 at 12:41 PM, Charles R Harris charlesr.harris@gmail.com wrote:
- This problem is specific to 2.6 and 2.5 works.
Yes
- It's on Intel hardware?
Yes.
Here is a minimal test which shows the problem:
import numpy as np assert np.dtype('<f4').isnative
So what does dtype(float32).descr and dtype(float32).byteorder show?
Chuck
On Mon, Nov 24, 2008 at 9:38 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
On Mon, Nov 24, 2008 at 9:02 PM, David Cournapeau cournape@gmail.comwrote:
On Tue, Nov 25, 2008 at 12:41 PM, Charles R Harris charlesr.harris@gmail.com wrote:
- This problem is specific to 2.6 and 2.5 works.
Yes
- It's on Intel hardware?
Yes.
Here is a minimal test which shows the problem:
import numpy as np assert np.dtype('<f4').isnative
So what does dtype(float32).descr and dtype(float32).byteorder show?
Numpy gets it's byte order from the macro WORDS_BIGENDIAN defined by Python. Try
$[charris@f9 numpy.git]$ grep -r -n WORDS_BIGENDIAN /usr/include/python2.5/* /usr/include/python2.5/pyconfig-32.h:902:#define WORDS_BIGENDIAN 1 /usr/include/python2.5/pyconfig-32.h:905:/* #undef WORDS_BIGENDIAN */
or the OS X equivalent.
Chuck
On Mon, Nov 24, 2008 at 9:58 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
On Mon, Nov 24, 2008 at 9:38 PM, Charles R Harris < charlesr.harris@gmail.com> wrote:
On Mon, Nov 24, 2008 at 9:02 PM, David Cournapeau cournape@gmail.comwrote:
On Tue, Nov 25, 2008 at 12:41 PM, Charles R Harris charlesr.harris@gmail.com wrote:
- This problem is specific to 2.6 and 2.5 works.
Yes
- It's on Intel hardware?
Yes.
Here is a minimal test which shows the problem:
import numpy as np assert np.dtype('<f4').isnative
So what does dtype(float32).descr and dtype(float32).byteorder show?
Numpy gets it's byte order from the macro WORDS_BIGENDIAN defined by Python. Try
$[charris@f9 numpy.git]$ grep -r -n WORDS_BIGENDIAN /usr/include/python2.5/* /usr/include/python2.5/pyconfig-32.h:902:#define WORDS_BIGENDIAN 1 /usr/include/python2.5/pyconfig-32.h:905:/* #undef WORDS_BIGENDIAN */
or the OS X equivalent.
Well, it may not be that easy to figure. The (generated) pyconfig-32.h has
/* Define to 1 if your processor stores words with the most significant byte first (like Motorola and SPARC, unlike Intel and VAX).
The block below does compile-time checking for endianness on platforms that use GCC and therefore allows compiling fat binaries on OSX by using
'-arch ppc -arch i386' as the compile flags. The phrasing was choosen such that the configure-result is used on systems that don't use GCC. */ #ifdef __BIG_ENDIAN__ #define WORDS_BIGENDIAN 1 #else #ifndef __LITTLE_ENDIAN__ /* #undef WORDS_BIGENDIAN */ #endif #endif
And I guess that __BIG_ENDIAN__ is a compiler flag, it isn't in any of the include files. In any case, this looks like a Python bug or the Python folks have switched their API on us.
Chuck
On Mon, 2008-11-24 at 22:06 -0700, Charles R Harris wrote:
Well, it may not be that easy to figure. The (generated) pyconfig-32.h has
/* Define to 1 if your processor stores words with the most significant byte first (like Motorola and SPARC, unlike Intel and VAX).
The block below does compile-time checking for endianness onplatforms that use GCC and therefore allows compiling fat binaries on OSX by using '-arch ppc -arch i386' as the compile flags. The phrasing was choosen such that the configure-result is used on systems that don't use GCC. */ #ifdef __BIG_ENDIAN__ #define WORDS_BIGENDIAN 1 #else #ifndef __LITTLE_ENDIAN__ /* #undef WORDS_BIGENDIAN */ #endif #endif
Hm, interesting: just by grepping, I do have WORDS_BIGENDIAN defined to 1 on *both* python 2.5 and python 2.6 on Mac OS X (running Intel). Looking closer, I do have the above code (conditional) in 2.5, but not in 2.6: it is inconditionally defined to BIGENDIAN on 2.6 !! That's actually part of something I have wondered for quite some time about fat binaries: how do you handle config headers, since they are generated only once for every fat binary, but they should really be generated for each arch.
And I guess that __BIG_ENDIAN__ is a compiler flag, it isn't in any of the include files. In any case, this looks like a Python bug or the Python folks have switched their API on us.
Hm, actually, it is a bug in numpy as much as in python: python should NOT include any config.h in their public namespace, and we should not rely on it.
But with this info, it should be relatively easy to fix (by setting the correct endianness by ourselves with some detection code)
David
FYI, I can't reproduce David's failures on my machine (intel core2 duo w/ 10.5.5) * python 2.6 from macports * numpy svn 6098 * GCC 4.0.1 (Apple Inc. build 5488)
I have only 1 failure: FAIL: test_umath.TestComplexFunctions.test_against_cmath ---------------------------------------------------------------------- Traceback (most recent call last): File "/opt/local/lib/python2.6/site-packages/nose-0.10.4-py2.6.egg/ nose/case.py", line 182, in runTest self.test(*self.arg) File "/Users/pierregm/Computing/.pythonenvs/default26/lib/python2.6/ site-packages/numpy/core/tests/test_umath.py", line 423, in test_against_cmath assert abs(a - b) < atol, "%s %s: %s; cmath: %s"%(fname,p,a,b) AssertionError: arcsin 2: (1.57079632679-1.31695789692j); cmath: (1.57079632679+1.31695789692j)
----------------------------------------------------------------------
(Well, there's another one in numpy.ma.min, but that's a different matter).
On Nov 25, 2008, at 2:19 AM, David Cournapeau wrote:
On Mon, 2008-11-24 at 22:06 -0700, Charles R Harris wrote:
Well, it may not be that easy to figure. The (generated) pyconfig-32.h has
/* Define to 1 if your processor stores words with the most significant byte first (like Motorola and SPARC, unlike Intel and VAX).
The block below does compile-time checking for endianness on platforms that use GCC and therefore allows compiling fat binaries on OSX by using '-arch ppc -arch i386' as the compile flags. The phrasing was choosen such that the configure-result is used on systems that don't use GCC. */ #ifdef __BIG_ENDIAN__ #define WORDS_BIGENDIAN 1 #else #ifndef __LITTLE_ENDIAN__ /* #undef WORDS_BIGENDIAN */ #endif #endif
Hm, interesting: just by grepping, I do have WORDS_BIGENDIAN defined to 1 on *both* python 2.5 and python 2.6 on Mac OS X (running Intel). Looking closer, I do have the above code (conditional) in 2.5, but not in 2.6: it is inconditionally defined to BIGENDIAN on 2.6 !! That's actually part of something I have wondered for quite some time about fat binaries: how do you handle config headers, since they are generated only once for every fat binary, but they should really be generated for each arch.
And I guess that __BIG_ENDIAN__ is a compiler flag, it isn't in any of the include files. In any case, this looks like a Python bug or the Python folks have switched their API on us.
Hm, actually, it is a bug in numpy as much as in python: python should NOT include any config.h in their public namespace, and we should not rely on it.
But with this info, it should be relatively easy to fix (by setting the correct endianness by ourselves with some detection code)
David
Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Pierre GM wrote:
FYI, I can't reproduce David's failures on my machine (intel core2 duo w/ 10.5.5)
- python 2.6 from macports
I think that's the main difference. I feel more and more that the problem is linked to fat binaries (more exactly multi arch build in one autoconf run: since only one pyconfig.h is generated for all archs, only one value is defined for CPU specific configurations). On my machine, pyconfig.h has WORDS_BIGENDIAN defined to one, which I can only explain by the binary being built on ppc (unfortunately, I can't find this information from python itself - maybe in the release notes). And that cannot work on Intel.
The general solution would be to generate different arch specific config files, and import them conditionally in the main config file. But doing so in a platform-neutral manner is not trivial,
David
David Cournapeau wrote:
Pierre GM wrote:
FYI, I can't reproduce David's failures on my machine (intel core2 duo w/ 10.5.5)
- python 2.6 from macports
I think that's the main difference. I feel more and more that the problem is linked to fat binaries (more exactly multi arch build in one autoconf run: since only one pyconfig.h is generated for all archs, only one value is defined for CPU specific configurations). On my machine, pyconfig.h has WORDS_BIGENDIAN defined to one, which I can only explain by the binary being built on ppc (unfortunately, I can't find this information from python itself - maybe in the release notes). And that cannot work on Intel.
Ok, I think I fixed the problem in the dynamic_cpu_configuration branch. I get only two test failures, which appear also on windows and linux (the same as yours). I think the code is OK, but if anyone has two minutes to review it, it would be better before merging it into the trunk.
I used the path of least resistance: instead of using the WORDS_BIGENDIAN macro, I added a numpy header which gives the endianness every time it is included. IOW, instead of the endianness to be fixed at numpy build time (which would fail for universal builds), it is set everytime the numpy headers are included (which is the only way to make it work). A better solution IMO would be to avoid any endianness dependency at all in the headers, but that does not seem possible without breaking the API (because the endianness-related macro PyArray_NBO and co would need to be set as functions instead).
cheers,
David
David
On Tue, Nov 25, 2008 at 10:55 PM, David Cournapeau david@ar.media.kyoto-u.ac.jp wrote:
I used the path of least resistance: instead of using the WORDS_BIGENDIAN macro, I added a numpy header which gives the endianness every time it is included. IOW, instead of the endianness to be fixed at numpy build time (which would fail for universal builds), it is set everytime the numpy headers are included (which is the only way to make it work). A better solution IMO would be to avoid any endianness dependency at all in the headers, but that does not seem possible without breaking the API (because the endianness-related macro PyArray_NBO and co would need to be set as functions instead).
Hm, for reference, I came across this:
http://www.mail-archive.com/python-dev@python.org/msg14382.html
So some people thought about the same problem.
David
On Tue, Nov 25, 2008 at 8:03 AM, David Cournapeau cournape@gmail.comwrote:
On Tue, Nov 25, 2008 at 10:55 PM, David Cournapeau david@ar.media.kyoto-u.ac.jp wrote:
I used the path of least resistance: instead of using the WORDS_BIGENDIAN macro, I added a numpy header which gives the endianness every time it is included. IOW, instead of the endianness to be fixed at numpy build time (which would fail for universal builds), it is set everytime the numpy headers are included (which is the only way to make it work). A better solution IMO would be to avoid any endianness dependency at all in the headers, but that does not seem possible without breaking the API (because the endianness-related macro PyArray_NBO and co would need to be set as functions instead).
Hm, for reference, I came across this:
http://www.mail-archive.com/python-dev@python.org/msg14382.html
So some people thought about the same problem.
Apart from the Mac, the ppc can be configured to run either bigendian or littleendian, so the hardware encompasses more than just the cpu, it's the whole darn board.
Chuck
On Wed, Nov 26, 2008 at 12:59 AM, Charles R Harris charlesr.harris@gmail.com wrote:
Apart from the Mac, the ppc can be configured to run either bigendian or littleendian, so the hardware encompasses more than just the cpu, it's the whole darn board.
Yep, many CPU families have double endian support (MIPS, ARM, PA-RISC, ALPHA. There is also "mixed" endian. Honestly, I think it is safe to assume that we don't need to care so much about those configurations for the time being. If it is a problem, we can then discuss about our headers being endian-free (which is really the best approach).
David
On Mon, 2008-11-24 at 21:38 -0700, Charles R Harris wrote:
On Mon, Nov 24, 2008 at 9:02 PM, David Cournapeau cournape@gmail.com wrote: On Tue, Nov 25, 2008 at 12:41 PM, Charles R Harris charlesr.harris@gmail.com wrote: >
> 1) This problem is specific to 2.6 and 2.5 works. Yes > 2) It's on Intel hardware? Yes. Here is a minimal test which shows the problem: import numpy as np assert np.dtype('<f4').isnativeSo what does dtype(float32).descr and dtype(float32).byteorder show?
The expected: [('', '<f4')] and '='. I don't think the problem is with the dtype itself (otherwise, almost all the tests would fail), but rather in its description. Since I don't know much about dtype internals, I don't really know where to look
David
participants (5)
-
Charles R Harris -
David Cournapeau -
David Cournapeau -
David Cournapeau -
Pierre GM