Python crashes while printing a huge numpy array (Windows 1.6b2)
Hello, due a mistake i found the following bug: import numpy as np x=np.zeros((262144, 262144)) print x.shape print x The last line crashes the python interpreter without any feedback. Windows 7, Python 2.6.5, Numpy 1.6b2
On Sat, Apr 23, 2011 at 10:33 PM, Till Stensitzki <mail.till@gmx.de> wrote:
Hello, due a mistake i found the following bug:
import numpy as np x=np.zeros((262144, 262144)) print x.shape
Hm, this is a bit weird. The line x = np.zeros... should fail (you most likely do not have 512 Gb Ram available on your machine :) ). Maybe a check is wrong due to some wrong configuration on windows. Are you on windows 32 or 64 bits ? cheers, David
On Sat, Apr 23, 2011 at 9:41 AM, David Cournapeau <cournape@gmail.com> wrote:
On Sat, Apr 23, 2011 at 10:33 PM, Till Stensitzki <mail.till@gmx.de> wrote:
Hello, due a mistake i found the following bug:
import numpy as np x=np.zeros((262144, 262144)) print x.shape
Hm, this is a bit weird. The line x = np.zeros... should fail (you most likely do not have 512 Gb Ram available on your machine :) ). Maybe a check is wrong due to some wrong configuration on windows. Are you on windows 32 or 64 bits ?
32 bit python on Windows 7 (64) I get the correct value error with python 2.6.5 and numpy 1.5.1 but on python 2.5.4 with numpy 1.3.0 I get the crash instead of the ValueError. Josef
cheers,
David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Sat, Apr 23, 2011 at 10:01 AM, <josef.pktd@gmail.com> wrote:
On Sat, Apr 23, 2011 at 9:41 AM, David Cournapeau <cournape@gmail.com> wrote:
On Sat, Apr 23, 2011 at 10:33 PM, Till Stensitzki <mail.till@gmx.de> wrote:
Hello, due a mistake i found the following bug:
import numpy as np x=np.zeros((262144, 262144)) print x.shape
Hm, this is a bit weird. The line x = np.zeros... should fail (you most likely do not have 512 Gb Ram available on your machine :) ). Maybe a check is wrong due to some wrong configuration on windows. Are you on windows 32 or 64 bits ?
32 bit python on Windows 7 (64)
I get the correct value error with python 2.6.5 and numpy 1.5.1
but on python 2.5.4 with numpy 1.3.0 I get the crash instead of the ValueError.
to rule out two more cases numpy 1.5.1 on python 2.7.1 32 bit : ValueError numpy 1.5.1 on python 3.2 64 bit: MemoryError Josef
Josef
cheers,
David _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Den 23.04.2011 16:09, skrev josef.pktd@gmail.com:
to rule out two more cases
numpy 1.5.1 on python 2.7.1 32 bit : ValueError numpy 1.5.1 on python 3.2 64 bit: MemoryError
numpy 1.5.1 on Python 2.7.1 64 bit: MemoryError Sturla
Hm, this is a bit weird. The line x = np.zeros... should fail (you most likely do not have 512 Gb Ram available on your machine :) ). Maybe a check is wrong due to some wrong configuration on windows. Are you on windows 32 or 64 bits ?
I took the outer product of two (512,512)-matrices instead of two 515-vectors :) 32 bit Windows here.
On Sat, Apr 23, 2011 at 8:11 AM, Till Stensitzki <mail.till@gmx.de> wrote:
Hm, this is a bit weird. The line x = np.zeros... should fail (you most likely do not have 512 Gb Ram available on your machine :) ). Maybe a check is wrong due to some wrong configuration on windows. Are you on windows 32 or 64 bits ?
I took the outer product of two (512,512)-matrices instead of two 515-vectors :)
32 bit Windows here.
Do you also have an earlier version of numpy installed? As David says, this should raise an error for recent numpy and I'm wondering if you are inadvertently running an earlier version. Chuck
Do you also have an earlier version of numpy installed? As David says, this should raise an error for recent numpy and I'm wondering if you are inadvertently running an earlier version.Chuck
I only have one python installation and numpy.__version__ shows 1.6b. I could reinstall numpy, if it would help.
On Sat, Apr 23, 2011 at 9:58 AM, Till Stensitzki <mail.till@gmx.de> wrote:
Do you also have an earlier version of numpy installed? As David says, this should raise an error for recent numpy and I'm wondering if you are inadvertently running an earlier version.Chuck
I only have one python installation and numpy.__version__ shows 1.6b. I could reinstall numpy, if it would help.
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi, I can get this with 64-bit Win 7, 32-bit Python 2.6, 2.7 (below) and 3.1 and numpy 1.6b (fresh install) IDLE and the command line. I can also confirm the 'ValueError' with Python2.6 and numpy 1.51 on the same system. Actually this is 'weird' when printing and crashed with the range - accessing unassigned memory? A smaller array gives an numpy error or memory error in idle. Bruce
import numpy as np x=np.zeros((262144, 262144)) x array([], shape=(262144, 262144), dtype=float64) x[0,0] 2.1453735050108555e-314 x[1:10,1:10]
================================ RESTART ================================ import numpy as np
x=np.zeros((26214, 26214))
Traceback (most recent call last): File "<pyshell#7>", line 1, in <module> x=np.zeros((26214, 26214)) ValueError: array is too big.
x=np.zeros((262144, 26214))
Traceback (most recent call last): File "<pyshell#8>", line 1, in <module> x=np.zeros((262144, 26214)) MemoryError
On Sat, Apr 23, 2011 at 11:09 AM, Bruce Southey <bsouthey@gmail.com> wrote:
On Sat, Apr 23, 2011 at 9:58 AM, Till Stensitzki <mail.till@gmx.de> wrote:
Do you also have an earlier version of numpy installed? As David says,
this
should raise an error for recent numpy and I'm wondering if you are inadvertently running an earlier version.Chuck
I only have one python installation and numpy.__version__ shows 1.6b. I could reinstall numpy, if it would help.
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
Hi, I can get this with 64-bit Win 7, 32-bit Python 2.6, 2.7 (below) and 3.1 and numpy 1.6b (fresh install) IDLE and the command line. I can also confirm the 'ValueError' with Python2.6 and numpy 1.51 on the same system.
Actually this is 'weird' when printing and crashed with the range - accessing unassigned memory? A smaller array gives an numpy error or memory error in idle.
Bruce
import numpy as np x=np.zeros((262144, 262144)) x array([], shape=(262144, 262144), dtype=float64) x[0,0] 2.1453735050108555e-314 x[1:10,1:10]
================================ RESTART ================================ import numpy as np
x=np.zeros((26214, 26214))
Traceback (most recent call last): File "<pyshell#7>", line 1, in <module> x=np.zeros((26214, 26214)) ValueError: array is too big.
x=np.zeros((262144, 26214))
Traceback (most recent call last): File "<pyshell#8>", line 1, in <module> x=np.zeros((262144, 26214)) MemoryError _____
This was fixed before, maybe it got broken again. Since this looks windows specific, I'm guessing it has something to do with the size of long being 32 bits. The previous problem was integer overflow when multiplying the dimensions together to get the array size when repeated divisions of the maximum size should have be used instead. Chuck
On 4/23/2011 10:41 AM, Charles R Harris wrote:
On Sat, Apr 23, 2011 at 11:09 AM, Bruce Southey <bsouthey@gmail.com <mailto:bsouthey@gmail.com>> wrote:
On Sat, Apr 23, 2011 at 9:58 AM, Till Stensitzki <mail.till@gmx.de <mailto:mail.till@gmx.de>> wrote: > > > Do you also have an earlier version of numpy installed? As David says, this > >should raise an error for recent numpy and > >I'm wondering if you are inadvertently > >running an earlier version.Chuck > > > I only have one python installation and > numpy.__version__ shows 1.6b. > I could reinstall numpy, if it would help. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >
Hi, I can get this with 64-bit Win 7, 32-bit Python 2.6, 2.7 (below) and 3.1 and numpy 1.6b (fresh install) IDLE and the command line. I can also confirm the 'ValueError' with Python2.6 and numpy 1.51 on the same system.
Actually this is 'weird' when printing and crashed with the range - accessing unassigned memory? A smaller array gives an numpy error or memory error in idle.
Bruce
> >> import numpy as np > >> x=np.zeros((262144, 262144)) > >> x array([], shape=(262144, 262144), dtype=float64) > >> x[0,0] 2.1453735050108555e-314 > >> x[1:10,1:10]
> >> ================================ RESTART ================================ > >> import numpy as np
> >> x=np.zeros((26214, 26214))
Traceback (most recent call last): File "<pyshell#7>", line 1, in <module> x=np.zeros((26214, 26214)) ValueError: array is too big. > >> > >> x=np.zeros((262144, 26214))
Traceback (most recent call last): File "<pyshell#8>", line 1, in <module> x=np.zeros((262144, 26214)) MemoryError _____
This was fixed before, maybe it got broken again. Since this looks windows specific, I'm guessing it has something to do with the size of long being 32 bits.
The previous problem was integer overflow when multiplying the dimensions together to get the array size when repeated divisions of the maximum size should have be used instead.
Chuck
Could be related to this change: <https://github.com/numpy/numpy/commit/fcc6cc73ddcb1fc85446ba9256ac24ecdda6c6...> Christoph
On Sat, Apr 23, 2011 at 3:38 PM, Christoph Gohlke <cgohlke@uci.edu> wrote:
On 4/23/2011 10:41 AM, Charles R Harris wrote:
On Sat, Apr 23, 2011 at 11:09 AM, Bruce Southey <bsouthey@gmail.com <mailto:bsouthey@gmail.com>> wrote:
On Sat, Apr 23, 2011 at 9:58 AM, Till Stensitzki <mail.till@gmx.de <mailto:mail.till@gmx.de>> wrote: > > > Do you also have an earlier version of numpy installed? As David says, this > >should raise an error for recent numpy and > >I'm wondering if you are inadvertently > >running an earlier version.Chuck > > > I only have one python installation and > numpy.__version__ shows 1.6b. > I could reinstall numpy, if it would help. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >
Hi, I can get this with 64-bit Win 7, 32-bit Python 2.6, 2.7 (below) and 3.1 and numpy 1.6b (fresh install) IDLE and the command line. I can also confirm the 'ValueError' with Python2.6 and numpy 1.51 on the same system.
Actually this is 'weird' when printing and crashed with the range - accessing unassigned memory? A smaller array gives an numpy error or memory error in idle.
Bruce
> >> import numpy as np > >> x=np.zeros((262144, 262144)) > >> x array([], shape=(262144, 262144), dtype=float64) > >> x[0,0] 2.1453735050108555e-314 > >> x[1:10,1:10]
> >> ================================ RESTART ================================ > >> import numpy as np
> >> x=np.zeros((26214, 26214))
Traceback (most recent call last): File "<pyshell#7>", line 1, in <module> x=np.zeros((26214, 26214)) ValueError: array is too big. > >> > >> x=np.zeros((262144, 26214))
Traceback (most recent call last): File "<pyshell#8>", line 1, in <module> x=np.zeros((262144, 26214)) MemoryError _____
This was fixed before, maybe it got broken again. Since this looks windows specific, I'm guessing it has something to do with the size of long being 32 bits.
The previous problem was integer overflow when multiplying the dimensions together to get the array size when repeated divisions of the maximum size should have be used instead.
Chuck
Could be related to this change:
< https://github.com/numpy/numpy/commit/fcc6cc73ddcb1fc85446ba9256ac24ecdda6c6...
My, that does look suspicious ;) Could you revert that loop and test it out? There was also a function for doing that check, I don't recall which, and it should probably be checked to make sure it remains as was. Chuck
On Sat, Apr 23, 2011 at 3:47 PM, Charles R Harris <charlesr.harris@gmail.com
wrote:
On Sat, Apr 23, 2011 at 3:38 PM, Christoph Gohlke <cgohlke@uci.edu> wrote:
On 4/23/2011 10:41 AM, Charles R Harris wrote:
On Sat, Apr 23, 2011 at 11:09 AM, Bruce Southey <bsouthey@gmail.com <mailto:bsouthey@gmail.com>> wrote:
On Sat, Apr 23, 2011 at 9:58 AM, Till Stensitzki <mail.till@gmx.de <mailto:mail.till@gmx.de>> wrote: > > > Do you also have an earlier version of numpy installed? As David says, this > >should raise an error for recent numpy and > >I'm wondering if you are inadvertently > >running an earlier version.Chuck > > > I only have one python installation and > numpy.__version__ shows 1.6b. > I could reinstall numpy, if it would help. > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org> > http://mail.scipy.org/mailman/listinfo/numpy-discussion >
Hi, I can get this with 64-bit Win 7, 32-bit Python 2.6, 2.7 (below) and 3.1 and numpy 1.6b (fresh install) IDLE and the command line. I can also confirm the 'ValueError' with Python2.6 and numpy 1.51 on the same system.
Actually this is 'weird' when printing and crashed with the range - accessing unassigned memory? A smaller array gives an numpy error or memory error in idle.
Bruce
> >> import numpy as np > >> x=np.zeros((262144, 262144)) > >> x array([], shape=(262144, 262144), dtype=float64) > >> x[0,0] 2.1453735050108555e-314 > >> x[1:10,1:10]
> >> ================================ RESTART ================================ > >> import numpy as np
> >> x=np.zeros((26214, 26214))
Traceback (most recent call last): File "<pyshell#7>", line 1, in <module> x=np.zeros((26214, 26214)) ValueError: array is too big. > >> > >> x=np.zeros((262144, 26214))
Traceback (most recent call last): File "<pyshell#8>", line 1, in <module> x=np.zeros((262144, 26214)) MemoryError _____
This was fixed before, maybe it got broken again. Since this looks windows specific, I'm guessing it has something to do with the size of long being 32 bits.
The previous problem was integer overflow when multiplying the dimensions together to get the array size when repeated divisions of the maximum size should have be used instead.
Chuck
Could be related to this change:
< https://github.com/numpy/numpy/commit/fcc6cc73ddcb1fc85446ba9256ac24ecdda6c6...
My, that does look suspicious ;) Could you revert that loop and test it out?
There was also a function for doing that check, I don't recall which, and it should probably be checked to make sure it remains as was.
There was also a test that should have caught this. Evidently it wasn't quite up to snuff. Chuck
On 4/23/2011 2:47 PM, Charles R Harris wrote:
On Sat, Apr 23, 2011 at 3:38 PM, Christoph Gohlke <cgohlke@uci.edu <mailto:cgohlke@uci.edu>> wrote:
On 4/23/2011 10:41 AM, Charles R Harris wrote: > > > On Sat, Apr 23, 2011 at 11:09 AM, Bruce Southey <bsouthey@gmail.com <mailto:bsouthey@gmail.com> > <mailto:bsouthey@gmail.com <mailto:bsouthey@gmail.com>>> wrote: > > On Sat, Apr 23, 2011 at 9:58 AM, Till Stensitzki <mail.till@gmx.de <mailto:mail.till@gmx.de> > <mailto:mail.till@gmx.de <mailto:mail.till@gmx.de>>> wrote: > > > > > Do you also have an earlier version of numpy installed? As David > says, this > > >should raise an error for recent numpy and > > >I'm wondering if you are inadvertently > > >running an earlier version.Chuck > > > > > > I only have one python installation and > > numpy.__version__ shows 1.6b. > > I could reinstall numpy, if it would help. > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org> <mailto:NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org>> > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > Hi, > I can get this with 64-bit Win 7, 32-bit Python 2.6, 2.7 (below) and > 3.1 and numpy 1.6b (fresh install) IDLE and the command line. I can > also confirm the 'ValueError' with Python2.6 and numpy 1.51 on the > same system. > > Actually this is 'weird' when printing and crashed with the range - > accessing unassigned memory? > A smaller array gives an numpy error or memory error in idle. > > Bruce > > > > >> import numpy as np > > >> x=np.zeros((262144, 262144)) > > >> x > array([], shape=(262144, 262144), dtype=float64) > > >> x[0,0] > 2.1453735050108555e-314 > > >> x[1:10,1:10] > > > >> ================================ RESTART > ================================ > > >> import numpy as np > > > >> x=np.zeros((26214, 26214)) > > Traceback (most recent call last): > File "<pyshell#7>", line 1, in <module> > x=np.zeros((26214, 26214)) > ValueError: array is too big. > > >> > > >> x=np.zeros((262144, 26214)) > > Traceback (most recent call last): > File "<pyshell#8>", line 1, in <module> > x=np.zeros((262144, 26214)) > MemoryError > _____ > > > This was fixed before, maybe it got broken again. Since this looks > windows specific, I'm guessing it has something to do with the size of > long being 32 bits. > > The previous problem was integer overflow when multiplying the > dimensions together to get the array size when repeated divisions of the > maximum size should have be used instead. > > Chuck
Could be related to this change:
<https://github.com/numpy/numpy/commit/fcc6cc73ddcb1fc85446ba9256ac24ecdda6c6...>
My, that does look suspicious ;) Could you revert that loop and test it out?
There was also a function for doing that check, I don't recall which, and it should probably be checked to make sure it remains as was.
Chuck
Reverting the change worked. A patch is attached.
import numpy as np np.__version__ '1.6.0b3' x=np.zeros((262144, 262144)) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: array is too big.
Christoph
On Sat, Apr 23, 2011 at 4:17 PM, Christoph Gohlke <cgohlke@uci.edu> wrote:
On 4/23/2011 2:47 PM, Charles R Harris wrote:
On Sat, Apr 23, 2011 at 3:38 PM, Christoph Gohlke <cgohlke@uci.edu <mailto:cgohlke@uci.edu>> wrote:
On 4/23/2011 10:41 AM, Charles R Harris wrote:
On Sat, Apr 23, 2011 at 11:09 AM, Bruce Southey
<bsouthey@gmail.com <mailto:bsouthey@gmail.com>
<mailto:bsouthey@gmail.com <mailto:bsouthey@gmail.com>>> wrote:
On Sat, Apr 23, 2011 at 9:58 AM, Till Stensitzki <mail.till@gmx.de <mailto:mail.till@gmx.de> <mailto:mail.till@gmx.de <mailto:mail.till@gmx.de>>> wrote:
Do you also have an earlier version of numpy installed? As David
says, this
should raise an error for recent numpy and I'm wondering if you are inadvertently running an earlier version.Chuck
I only have one python installation and numpy.__version__ shows 1.6b. I could reinstall numpy, if it would help.
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org> <mailto:NumPy-Discussion@scipy.org <mailto:NumPy-Discussion@scipy.org
Hi, I can get this with 64-bit Win 7, 32-bit Python 2.6, 2.7 (below) and 3.1 and numpy 1.6b (fresh install) IDLE and the command line. I can also confirm the 'ValueError' with Python2.6 and numpy 1.51 on the same system.
Actually this is 'weird' when printing and crashed with the range - accessing unassigned memory? A smaller array gives an numpy error or memory error in idle.
Bruce
import numpy as np x=np.zeros((262144, 262144)) x array([], shape=(262144, 262144), dtype=float64) x[0,0] 2.1453735050108555e-314 x[1:10,1:10]
================================ RESTART ================================ import numpy as np
x=np.zeros((26214, 26214))
Traceback (most recent call last): File "<pyshell#7>", line 1, in <module> x=np.zeros((26214, 26214)) ValueError: array is too big.
x=np.zeros((262144, 26214))
Traceback (most recent call last): File "<pyshell#8>", line 1, in <module> x=np.zeros((262144, 26214)) MemoryError _____
This was fixed before, maybe it got broken again. Since this looks windows specific, I'm guessing it has something to do with the size of long being 32 bits.
The previous problem was integer overflow when multiplying the dimensions together to get the array size when repeated divisions of the maximum size should have be used instead.
Chuck
Could be related to this change:
< https://github.com/numpy/numpy/commit/fcc6cc73ddcb1fc85446ba9256ac24ecdda6c6...
My, that does look suspicious ;) Could you revert that loop and test it out?
There was also a function for doing that check, I don't recall which, and it should probably be checked to make sure it remains as was.
Chuck
Reverting the change worked. A patch is attached.
import numpy as np
np.__version__
'1.6.0b3'
x=np.zeros((262144, 262144))
Traceback (most recent call last): File "<stdin>", line 1, in <module>
ValueError: array is too big.
Thanks, applied. Chuck
participants (7)
-
Bruce Southey -
Charles R Harris -
Christoph Gohlke -
David Cournapeau -
josef.pktd＠gmail.com -
Sturla Molden -
Till Stensitzki