Crash using "reshape"...
I am having a problem with "reshape" crashing: > python Python 2.6.4 (r264:75706, Jan 16 2010, 21:11:47) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.version.version '1.6.2' >>> npData = numpy.ones([701,701,7899],dtype=numpy.dtype('b')) >>> npDataSubset = npData[[slice(0,700),slice(0,700),slice(0,5000)]] >>> npDataOutput = npDataSubset.reshape([700*700*5000],order='F') Segmentation fault If I change the "5000" to a "4000", everything is fine. I'm not running out of memory - my system had 48 GB of memory and nothing else is using a significant portion of this memory. Note: 700x700x4000 = 1,960,000,000 < 2^31 and 700x700x5000 = 2450000000
2^31. I suspect somewhere in the underlying code there is a signed 32-bit integer being used for an index/pointer offset (this is running on a 64-bit machine).
I did some searching of the archives and didn't find a match for this problem. Thank you for any and all help! Terry J. (Ligocki, tjligocki@lbl.gov) ------------------------------------------------------------------------ *Wishes * /When you wish upon a falling star, your dreams can come true. Unless it's really a meteorite hurtling to the Earth which will destroy all life. Then you're pretty much hosed no matter what you wish for. Unless it's death by meteor. / (Despair, Inc.)
On 11/21/12 10:12 AM, Terry J. Ligocki wrote:
I am having a problem with "reshape" crashing:
> python Python 2.6.4 (r264:75706, Jan 16 2010, 21:11:47) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.version.version '1.6.2' >>> npData = numpy.ones([701,701,7899],dtype=numpy.dtype('b')) >>> npDataSubset = npData[[slice(0,700),slice(0,700),slice(0,5000)]] >>> npDataOutput = npDataSubset.reshape([700*700*5000],order='F') Segmentation fault
If I change the "5000" to a "4000", everything is fine. I'm not running out of memory - my system had 48 GB of memory and nothing else is using a significant portion of this memory.
Note: 700x700x4000 = 1,960,000,000 < 2^31 and 700x700x5000 = 2450000000 > 2^31. I suspect somewhere in the underlying code there is a signed 32-bit integer being used for an index/pointer offset (this is running on a 64-bit machine).
Yes, looks like a 32-bit issue. Sometimes you can have 32-bit software installed in 64-bit machines, so that might be your problem. What's the equivalent of numpy.intp in your machine? Mine is: In []: import numpy as np In []: np.intp Out[]: numpy.int64 If you see 'numpy.int32' here then that is the problem. -- Francesc Alted
I just checked, "numpy.intp" is "
On 11/21/12 10:12 AM, Terry J. Ligocki wrote:
I am having a problem with "reshape" crashing:
> python Python 2.6.4 (r264:75706, Jan 16 2010, 21:11:47) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.version.version '1.6.2' >>> npData = numpy.ones([701,701,7899],dtype=numpy.dtype('b')) >>> npDataSubset = npData[[slice(0,700),slice(0,700),slice(0,5000)]] >>> npDataOutput = npDataSubset.reshape([700*700*5000],order='F') Segmentation fault
If I change the "5000" to a "4000", everything is fine. I'm not running out of memory - my system had 48 GB of memory and nothing else is using a significant portion of this memory.
Note: 700x700x4000 = 1,960,000,000 < 2^31 and 700x700x5000 = 2450000000 > 2^31. I suspect somewhere in the underlying code there is a signed 32-bit integer being used for an index/pointer offset (this is running on a 64-bit machine). Yes, looks like a 32-bit issue. Sometimes you can have 32-bit software installed in 64-bit machines, so that might be your problem. What's the equivalent of numpy.intp in your machine? Mine is:
In []: import numpy as np
In []: np.intp Out[]: numpy.int64
If you see 'numpy.int32' here then that is the problem.
/She's got to be this town's best mess But it ain't nothin' that her face would suggest / - God Don't Make Lonely Girls (The Wallflowers)
On 11/21/12 11:25 AM, Terry J. Ligocki wrote:
I just checked, "numpy.intp" is "
" in my installation of Python and NumPy. It was a good thing to check but it looks like there's still may be a signed 32-bit integer somewhere in the code (or my build(s))...
Okay. I can reproduce that too (using 1.6.1). Could you please file a ticket for this? Smells like a bug to me. -- Francesc Alted
Hey, On Wed, 2012-11-21 at 01:12 -0800, Terry J. Ligocki wrote:
I am having a problem with "reshape" crashing: > python Python 2.6.4 (r264:75706, Jan 16 2010, 21:11:47) [GCC 4.3.2] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.version.version '1.6.2' >>> npData = numpy.ones([701,701,7899],dtype=numpy.dtype('b')) >>> npDataSubset = npData[[slice(0,700),slice(0,700),slice(0,5000)]] >>> npDataOutput = npDataSubset.reshape([700*700*5000],order='F') Segmentation fault If I change the "5000" to a "4000", everything is fine. I'm not running out of memory - my system had 48 GB of memory and nothing else is using a significant portion of this memory.
Note: 700x700x4000 = 1,960,000,000 < 2^31 and 700x700x5000 = 2450000000 > 2^31. I suspect somewhere in the underlying code there is a signed 32-bit integer being used for an index/pointer offset (this is running on a 64-bit machine).
yes, int is used for npy_intp in one occasione. I have created a PR that fixes the issue: https://github.com/numpy/numpy/pull/2754 Regards, Sebastian
I did some searching of the archives and didn't find a match for this problem. Thank you for any and all help!
Terry J. (Ligocki, tjligocki@lbl.gov)
______________________________________________________________________ Wishes
When you wish upon a falling star, your dreams can come true. Unless it's really a meteorite hurtling to the Earth which will destroy all life. Then you're pretty much hosed no matter what you wish for. Unless it's death by meteor.
(Despair, Inc.)
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
participants (3)
-
Francesc Alted
-
Sebastian Berg
-
Terry J. Ligocki