Integer overflow in test_einsum (1.7.1)
Hello, One of the test cases in test_einsum causes integer overflow for i2 type. The test goes like this:
import numpy as np dtype = 'i2' n = 15 a = np.arange(4*n, dtype=dtype).reshape(4,n) b = np.arange(n*6, dtype=dtype).reshape(n,6) c = np.arange(24, dtype=dtype).reshape(4,6)
It then calculates AxB using einsum. The problem is that the values in the last row of the result do not fit into i2:
np.einsum("ij,jk", a, b, dtype='f8', casting='unsafe') array([[ 6090., 6195., 6300., 6405., 6510., 6615.], [ 15540., 15870., 16200., 16530., 16860., 17190.], [ 24990., 25545., 26100., 26655., 27210., 27765.], [ 34440., 35220., 36000., 36780., 37560., 38340.]])
In my build this produces different results depending on whether out or .astype is used:
np.einsum("ij,jk", a, b, dtype='f8', casting='unsafe').astype(dtype) array([[ 6090, 6195, 6300, 6405, 6510, 6615], [ 15540, 15870, 16200, 16530, 16860, 17190], [ 24990, 25545, 26100, 26655, 27210, 27765], [-31096, -30316, -29536, -28756, -27976, -27196]], dtype=int16)
np.einsum("ij,jk", a, b, out=c, dtype='f8', casting='unsafe') array([[ 6090, 6195, 6300, 6405, 6510, 6615], [ 15540, 15870, 16200, 16530, 16860, 17190], [ 24990, 25545, 26100, 26655, 27210, 27765], [-32768, -32768, -32768, -32768, -32768, -32768]], dtype=int16)
The test wants these (actually the same using numpy.dot) to be equal, so this difference causes it to fail. Both ways to handle overflow seem reasonable to me. Does numpy in general assign a defined behavior to integer overflow (e.g. two's complement)? Is this use of integer overflow in the test intentional and is expected to work, or is my build broken? Best regards, Eugene This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email.
On Tue, May 14, 2013 at 11:26 AM, Toder, Evgeny <evgeny.toder@jpmorgan.com>wrote:
Hello,****
** **
One of the test cases in test_einsum causes integer overflow for i2 type. The test goes like this:****
** **
import numpy as np****
dtype = 'i2'****
n = 15****
a = np.arange(4*n, dtype=dtype).reshape(4,n)****
b = np.arange(n*6, dtype=dtype).reshape(n,6)****
c = np.arange(24, dtype=dtype).reshape(4,6)****
** **
It then calculates AxB using einsum. The problem is that the values in the last row of the result do not fit into i2:****
** **
np.einsum("ij,jk", a, b, dtype='f8', casting='unsafe')****
array([[ 6090., 6195., 6300., 6405., 6510., 6615.],****
[ 15540., 15870., 16200., 16530., 16860., 17190.],****
[ 24990., 25545., 26100., 26655., 27210., 27765.],****
[ 34440., 35220., 36000., 36780., 37560., 38340.]])****
** **
In my build this produces different results depending on whether out or .astype is used:****
** **
np.einsum("ij,jk", a, b, dtype='f8', casting='unsafe').astype(dtype)** **
array([[ 6090, 6195, 6300, 6405, 6510, 6615],****
[ 15540, 15870, 16200, 16530, 16860, 17190],****
[ 24990, 25545, 26100, 26655, 27210, 27765],****
[-31096, -30316, -29536, -28756, -27976, -27196]], dtype=int16)****
** **
np.einsum("ij,jk", a, b, out=c, dtype='f8', casting='unsafe')****
array([[ 6090, 6195, 6300, 6405, 6510, 6615],****
[ 15540, 15870, 16200, 16530, 16860, 17190],****
[ 24990, 25545, 26100, 26655, 27210, 27765],****
[-32768, -32768, -32768, -32768, -32768, -32768]], dtype=int16)****
** **
The test wants these (actually the same using numpy.dot) to be equal, so this difference causes it to fail. Both ways to handle overflow seem reasonable to me. ****
** **
Does numpy in general assign a defined behavior to integer overflow (e.g. two’s complement)?****
Is this use of integer overflow in the test intentional and is expected to work, or is my build broken?****
I don't know if it was intended in the test, but numpy doesn't guard against integer overflow. Rather, it takes the C approach and does modular arithmetic. I don't know how to fix that or even if it could be at this point given the historical lack of support. I suppose at some point there could be user type that made such checks, but it would be slower. Chuck
C only does modular arithmetic for unsigned types. Signed types (like i2 used here, is should be signed short) are not allowed to overflow in C. I.e. not only is the result not guaranteed to be modular, or not guaranteed to be consistent, the behavior of the whole program is not specified if it allows signed integer overflow. In practice, your often get two's complement arithmetic, because this is what all current CPUs implement. So, does numpy implement modular (two's complement) arithmetic for signed types, or does it leave it to C compiler, which does not provide any guarantees? Eugene From: numpy-discussion-bounces@scipy.org [mailto:numpy-discussion-bounces@scipy.org] On Behalf Of Charles R Harris Sent: Tuesday, May 14, 2013 2:25 PM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Integer overflow in test_einsum (1.7.1) I don't know if it was intended in the test, but numpy doesn't guard against integer overflow. Rather, it takes the C approach and does modular arithmetic. I don't know how to fix that or even if it could be at this point given the historical lack of support. I suppose at some point there could be user type that made such checks, but it would be slower. Chuck This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email.
14.05.2013 21:52, Toder, Evgeny kirjoitti:
So, does numpy implement modular (two’s complement) arithmetic for signed types [clip]
Numpy leaves integer arithmetic to the C compiler. Python and its C modules however by default specify -fwrapv for gcc and its ilk, so the behavior might seem consistent on these platforms. -- Pauli Virtanen
Thank you, Pauli. I guess this means that the test needs to be fixed? Eugene -----Original Message----- From: numpy-discussion-bounces@scipy.org [mailto:numpy-discussion-bounces@scipy.org] On Behalf Of Pauli Virtanen Sent: Tuesday, May 14, 2013 3:31 PM To: numpy-discussion@scipy.org Subject: Re: [Numpy-discussion] Integer overflow in test_einsum (1.7.1) 14.05.2013 21:52, Toder, Evgeny kirjoitti:
So, does numpy implement modular (two's complement) arithmetic for signed types [clip]
Numpy leaves integer arithmetic to the C compiler. Python and its C modules however by default specify -fwrapv for gcc and its ilk, so the behavior might seem consistent on these platforms. -- Pauli Virtanen _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion This email is confidential and subject to important disclaimers and conditions including on offers for the purchase or sale of securities, accuracy and completeness of information, viruses, confidentiality, legal privilege, and legal entity disclaimers, available at http://www.jpmorgan.com/pages/disclosures/email.
participants (3)
-
Charles R Harris -
Pauli Virtanen -
Toder, Evgeny