python eats memory like the cookie monster eats cookies
Good morning, not sure if I got the right list, but I hope that somebody here will be able to shed some light on a Python-related memory problem. The following code eats over >2GB of memory and fails with MemoyError after just a few iterations. def ZeroPadData(A, shape): a = Numeric.zeros(shape, 'w') a.savespace() for y in xrange(A.shape[0]): for x in xrange(A.shape[1]): a[y, x] = A[y, x] return a def EatMemoryLikeTheCookieMonster(limit=10): A = Numeric.ones([1998, 3022]) count = 0 a = A while count < limit: print count count += 1 a = ZeroPadData(a, [2048, 4096]) b = fft2(a) b = ifft2(b) a = b[:1998,:3022].real EatMemoryLikeTheCookieMonster() This is for Python 2.4.3 on Mac OS X 10.4.8 (intel) using SciPy 0.5.2. If anyone could enlighten me about what I am doing wrong, I would very much appreciate it. Thank you, Niels.
Niels Provos wrote:
Good morning,
not sure if I got the right list, but I hope that somebody here will be able to shed some light on a Python-related memory problem. The following code eats over >2GB of memory and fails with MemoyError after just a few iterations.
def ZeroPadData(A, shape): a = Numeric.zeros(shape, 'w') a.savespace()
for y in xrange(A.shape[0]): for x in xrange(A.shape[1]): a[y, x] = A[y, x]
return a
def EatMemoryLikeTheCookieMonster(limit=10): A = Numeric.ones([1998, 3022])
count = 0 a = A while count < limit: print count count += 1
a = ZeroPadData(a, [2048, 4096])
b = fft2(a) b = ifft2(b)
a = b[:1998,:3022].real
EatMemoryLikeTheCookieMonster()
This is for Python 2.4.3 on Mac OS X 10.4.8 (intel) using SciPy 0.5.2.
Could you also post a complete example? Why are you using Numeric? scipy 0.5.2 requires numpy, not Numeric. Where are the fft2() and ifft2() functions coming from, scipy.fftpack or numpy? -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
The missing imports are import Numeric # for zeros and ones from scipy.fftpack import fft2,ifft2 Curiously, replacing Numeric.zeros with scipy.zeros makes the problem go away. Why? Thank you, Niels. On 2/4/07, Robert Kern <robert.kern@gmail.com> wrote:
Niels Provos wrote:
Good morning,
not sure if I got the right list, but I hope that somebody here will be able to shed some light on a Python-related memory problem. The following code eats over >2GB of memory and fails with MemoyError after just a few iterations.
def ZeroPadData(A, shape): a = Numeric.zeros(shape, 'w') a.savespace()
for y in xrange(A.shape[0]): for x in xrange(A.shape[1]): a[y, x] = A[y, x]
return a
def EatMemoryLikeTheCookieMonster(limit=10): A = Numeric.ones([1998, 3022])
count = 0 a = A while count < limit: print count count += 1
a = ZeroPadData(a, [2048, 4096])
b = fft2(a) b = ifft2(b)
a = b[:1998,:3022].real
EatMemoryLikeTheCookieMonster()
This is for Python 2.4.3 on Mac OS X 10.4.8 (intel) using SciPy 0.5.2.
Could you also post a complete example? Why are you using Numeric? scipy 0.5.2 requires numpy, not Numeric. Where are the fft2() and ifft2() functions coming from, scipy.fftpack or numpy?
-- Robert Kern
"I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco _______________________________________________ Numpy-discussion mailing list Numpy-discussion@scipy.org http://projects.scipy.org/mailman/listinfo/numpy-discussion
Niels Provos wrote:
The missing imports are
import Numeric # for zeros and ones from scipy.fftpack import fft2,ifft2
Curiously, replacing Numeric.zeros with scipy.zeros makes the problem go away. Why?
Possibly a bug in Numeric. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
On 2/4/07, Robert Kern <robert.kern@gmail.com> wrote:
Niels Provos wrote:
The missing imports are
import Numeric # for zeros and ones from scipy.fftpack import fft2,ifft2
Curiously, replacing Numeric.zeros with scipy.zeros makes the problem go away. Why?
Possibly a bug in Numeric.
-- Robert Kern
Is there *any* support for old Numeric on this list !? Maybe it should be officially stated that the one way to go is numpy and that problems with Numeric ( or numarray ) can only be noticed but will likely not get fixed.... -Sebastian
Sebastian Haase wrote:
Is there *any* support for old Numeric on this list !?
Not unless if you are offering some.
Maybe it should be officially stated that the one way to go is numpy and that problems with Numeric ( or numarray ) can only be noticed but will likely not get fixed....
That's pretty much what we've been officially stating for some time now. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
Niels Provos wrote:
Good morning,
not sure if I got the right list, but I hope that somebody here will be able to shed some light on a Python-related memory problem. The following code eats over >2GB of memory and fails with MemoyError after just a few iterations.
Here is a minimal example that demonstrates the problem. It appears that assigning a numpy scalar to an element in an n-D Numeric array (where n is strictly > 1) leaks memory. Neither the data-types of the Numeric array nor the numpy scalar type seem to matter. import Numeric import numpy a = Numeric.zeros([1, 1]) while True: a[0, 0] = numpy.int32(0) -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." -- Umberto Eco
participants (3)
-
Niels Provos -
Robert Kern -
Sebastian Haase