PyArray_CanCastSafely(exact,inexact) on 64-bit
Hi Travis, Number of scipy.test failures occuring on 64-bit systems are due to the fact that PyArray_CanCastSafely(PyArray_LONG,PyArray_DOUBLE) returns true. Though sizeof(long)==sizeof(double) holds on 64-bit machines, shouldn't PyArray_CanCastSafely return false on (exact,inexact) arguments and vice versa? By the definition of can_cast, no bitwise information is lost, but it is not meaningful to pass (double*)(<pointer to long array>) to a numeric function, for example. Or may be I should use some different function than PyArray_CanCastSafely in this situation? Pearu
Pearu Peterson wrote:
Hi Travis,
Number of scipy.test failures occuring on 64-bit systems are due to the fact that PyArray_CanCastSafely(PyArray_LONG,PyArray_DOUBLE) returns true.
We had this discussion about a week ago, so you should definitely look at other comments from that discussion. What exactly are you trying to do that is causing the errors?
Though sizeof(long)==sizeof(double) holds on 64-bit machines, shouldn't PyArray_CanCastSafely return false on (exact,inexact) arguments and vice versa?
It's never done that in the past, so this would represent quite a switch. It has always been possible to convert from integers to floats "safely." On old Numeric you could cast from int8 and int16 to float32 "safely" and from int32 to float64 "safely". The problem is with 64-bit integers. They can't be represented by 64-bit floats, and the original code only allowed them to be converted safely to long double floats. But, because on 64-bit platforms, the long is often 64-bits and that's the default Python integer, it would mean always doing long double calculations. PyArray_CanCastSafely is used largely for determining the "coercion" rules for ufuncs. The current function causes sqrt(2) to take place with double precision.
By the definition of can_cast, no bitwise information is lost, but it is not meaningful to pass (double*)(<pointer to long array>) to a numeric function, for example.
But, that's never been what CanCastSafely has been used to represent. Perhaps we need a different function that distinguishes between different kinds. Again, what are you trying to do, specifically?
Or may be I should use some different function than PyArray_CanCastSafely in this situation?
I suspect so... -Travis
On Sun, 30 Oct 2005, Travis Oliphant wrote:
Pearu Peterson wrote:
By the definition of can_cast, no bitwise information is lost, but it is not meaningful to pass (double*)(<pointer to long array>) to a numeric function, for example.
But, that's never been what CanCastSafely has been used to represent. Perhaps we need a different function that distinguishes between different kinds. Again, what are you trying to do, specifically?
Or may be I should use some different function than PyArray_CanCastSafely in this situation?
I suspect so...
PyArray_CanCastSafely has been used in array_from_pyobj function (see fortranobject.c) to decide whether the data pointer of an input array can be directly passed on to the Fortran or C function: if ((! (intent & F2PY_INTENT_COPY)) && PyArray_ITEMSIZE(arr)==descr->elsize && PyArray_CanCastSafely(arr->descr->type_num,type_num) ) { if ((intent&F2PY_INTENT_C)?PyArray_ISCARRAY(arr):PyArray_ISFARRAY(arr)) { if ((intent & F2PY_INTENT_OUT)) { Py_INCREF(arr); } /* Returning input array */ return arr; } } /* else apply arr.astype(<type_num>) */ But I now realize that this is a incorrect usage of PyArray_CanCastSafely. The above codelet should probably then read as if ((! (intent & F2PY_INTENT_COPY)) && PyArray_ITEMSIZE(arr)==descr->elsize && ( (PyArray_ISINTEGER(arr) && PyTypeNum_ISINTEGER(type_num)) || (PyArray_ISFLOAT(arr) && PyTypeNum_ISFLOAT(type_num)) || (PyArray_ISCOMPLEX(arr) && PyTypeNum_ISCOMPLEX(type_num)) || (PyArray_ISSTRING(arr) && PyTypeNum_ISSTRING(type_num)) ) ) { ... though I am not sure about the STRING case, e.g. what happens when passing unicode arr->data to fortran character*(*) argument, for instance. Thanks, Pearu
After fixing the usage of PyArray_CanCastSafely, now all scipy tests, except one, are succesfull on a 64-bit machine: ====================================================================== ERROR: check_integer (scipy.io.array_import.test_array_import.test_read_array) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.3/site-packages/scipy/io/tests/test_array_import.py", line 62, in check_integer b = io.read_array(fname,atype=N.Int) File "/usr/local/lib/python2.3/site-packages/scipy/io/array_import.py", line 359, in read_array raise ValueError, "One of the array types is invalid, k=%d" % k ValueError: One of the array types is invalid, k=0 ---------------------------------------------------------------------- Ran 1334 tests in 57.772s FAILED (errors=1) Pearu
Pearu Peterson wrote:
After fixing the usage of PyArray_CanCastSafely, now all scipy tests, except one, are succesfull on a 64-bit machine:
====================================================================== ERROR: check_integer (scipy.io.array_import.test_array_import.test_read_array) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.3/site-packages/scipy/io/tests/test_array_import.py", line 62, in check_integer b = io.read_array(fname,atype=N.Int) File "/usr/local/lib/python2.3/site-packages/scipy/io/array_import.py", line 359, in read_array raise ValueError, "One of the array types is invalid, k=%d" % k ValueError: One of the array types is invalid, k=0
---------------------------------------------------------------------- Ran 1334 tests in 57.772s
Fantastic. I'm sure we can track down this problem. -Travis
On Sun, 30 Oct 2005, Pearu Peterson wrote:
After fixing the usage of PyArray_CanCastSafely, now all scipy tests, except one, are succesfull on a 64-bit machine:
====================================================================== ERROR: check_integer (scipy.io.array_import.test_array_import.test_read_array) ---------------------------------------------------------------------- Traceback (most recent call last): File "/usr/local/lib/python2.3/site-packages/scipy/io/tests/test_array_import.py", line 62, in check_integer b = io.read_array(fname,atype=N.Int) File "/usr/local/lib/python2.3/site-packages/scipy/io/array_import.py", line 359, in read_array raise ValueError, "One of the array types is invalid, k=%d" % k ValueError: One of the array types is invalid, k=0
---------------------------------------------------------------------- Ran 1334 tests in 57.772s
FAILED (errors=1)
That is great news! I also get this on my 64 Bit machine! Just in case it has fallen through the cracks: Concerning the check_integer problem: def check_integer(self): from scipy import stats a = stats.randint.rvs(1,20,size=(3,4)) fname = tempfile.mktemp('.dat') io.write_array(fname,a) b = io.read_array(fname,atype=N.Int) assert_array_equal(a,b) os.remove(fname) Executing this line by line shows the error for b = io.read_array(fname,atype=N.Int) Doing b = io.read_array(fname) reads in the array, but it gives floats. However, b = io.read_array(fname,atype=N.Int32) works. If this is the intended behaviour (also on 32Bit), the unit test should be changed accordingly... BTW: wouldn't io be much better in newcore instead of newscipy? It seems like something of very common use. Best, Arnd
Arnd Baecker wrote:
That is great news! I also get this on my 64 Bit machine!
Wonderful...
Just in case it has fallen through the cracks: Concerning the check_integer problem:
def check_integer(self): from scipy import stats a = stats.randint.rvs(1,20,size=(3,4)) fname = tempfile.mktemp('.dat') io.write_array(fname,a) b = io.read_array(fname,atype=N.Int) assert_array_equal(a,b) os.remove(fname)
Executing this line by line shows the error for b = io.read_array(fname,atype=N.Int)
Doing b = io.read_array(fname) reads in the array, but it gives floats.
However, b = io.read_array(fname,atype=N.Int32) works.
If this is the intended behaviour (also on 32Bit), the unit test should be changed accordingly...
What is the type of 'a' (i.e. stats.randint.rvs(1,20,size=(3,4)) ) on your platform? This does look like a problem of type mismatch on 64-bit that we can handle much better now. It looks like randint is returning 32-bit numbers, but N.Int is 'l' which on your 64-bit platform is a 64-bit integer. This would definitely be the problem. I've changed the test...
BTW: wouldn't io be much better in newcore instead of newscipy? It seems like something of very common use.
Yes, that is a strong possibility. The only issue is that scipy arrays can now be directly written to files (using the tofile method) and read from files (using fromfile function). The scipy.io code essentially re-does all of that (but not for the new flexible arrays). It would need to be adapted before I think it would be a good fit. -Travis
On Mon, 31 Oct 2005, Travis Oliphant wrote:
Arnd Baecker wrote:
That is great news! I also get this on my 64 Bit machine!
Wonderful...
Just in case it has fallen through the cracks: Concerning the check_integer problem:
def check_integer(self): from scipy import stats a = stats.randint.rvs(1,20,size=(3,4)) fname = tempfile.mktemp('.dat') io.write_array(fname,a) b = io.read_array(fname,atype=N.Int) assert_array_equal(a,b) os.remove(fname)
Executing this line by line shows the error for b = io.read_array(fname,atype=N.Int)
Doing b = io.read_array(fname) reads in the array, but it gives floats.
However, b = io.read_array(fname,atype=N.Int32) works.
If this is the intended behaviour (also on 32Bit), the unit test should be changed accordingly...
What is the type of 'a' (i.e. stats.randint.rvs(1,20,size=(3,4)) ) on your platform?
This does look like a problem of type mismatch on 64-bit that we can handle much better now.
It looks like randint is returning 32-bit numbers, but N.Int is 'l' which on your 64-bit platform is a 64-bit integer. This would definitely be the problem.
In [1]: import scipy In [2]: print scipy.__core_version__, scipy.__scipy_version__ 0.4.3.1408 0.4.2_1409 In [3]: a=scipy.stats.randint.rvs(1,20,size=(3,4)) In [4]: a.dtypechar Out[4]: 'l' In [5]: a.dtypestr Out[5]: '<i8' In [6]: scipy.Int Out[6]: 'p' In [7]: scipy.typecodes Out[7]: {'All': '?bhilqBHILQfdgFDGSUVO', 'AllFloat': 'fdgFDG', 'AllInteger': 'bBhHiIlLqQ', 'Character': 'S1', 'Complex': 'FDG', 'Float': 'fdg', 'Integer': 'bhilq', 'UnsignedInteger': 'BHILQ'} In [8]: scipy.Int,scipy.Int0,scipy.Int8,scipy.Int16,scipy.Int32 Out[8]: ('p', 'p', 'b', 'h', 'i') So `scipy.Int` is not 'l'?? And what is 'p'? (is it intp as described in your newcore manual? - should this arise here?)
I've changed the test...
Hmm - can't see any difference to previous versions? Best, Arnd
PyArray_CanCastSafely has been used in array_from_pyobj function (see fortranobject.c) to decide whether the data pointer of an input array can be directly passed on to the Fortran or C function:
if ((! (intent & F2PY_INTENT_COPY)) && PyArray_ITEMSIZE(arr)==descr->elsize && PyArray_CanCastSafely(arr->descr->type_num,type_num) ) { if ((intent&F2PY_INTENT_C)?PyArray_ISCARRAY(arr):PyArray_ISFARRAY(arr)) { if ((intent & F2PY_INTENT_OUT)) { Py_INCREF(arr); } /* Returning input array */ return arr; } } /* else apply arr.astype(<type_num>) */
There are new functions in the C-API called PyArray_EquivalentTypes and PyArray_EquivArrType that can be used to check what you want to check. These functions are used internally to decide whether or not two types are compatible and can be substituted one for another. -Travis
participants (3)
-
Arnd Baecker -
Pearu Peterson -
Travis Oliphant