Memory leak in numpy?
Hi all-
I think I just found a memory leak in numpy, or maybe I just don’t understand generators. Anyway, the following snippet will quickly eat a ton of RAM:
P = randint(0,2, (20,13))
for i in range(50): for ai in ndindex((2,)*13): j = P.dot(ai)
If you replace the last line with something like j = ai, the memory leak goes away. I’m not exactly sure what’s going on but the .dot seems to be causing the memory taken by the tuple ai to be held.
This devours RAM in python 2.7.5 (OS X Mavericks default I believe), numpy version 1.8.0.dev-3084618. I’m upgrading to the latest Superpack (numpy 1.9) right now but I somehow doubt this behavior will change.
Any thoughts?
Best, Chris
-- Chris Laumann Sent with Airmail
Hmmm, I see no reason why that would eat up memory. I just tried it out on my own system (numpy 1.6.1, CentOS 6, python 2.7.1), and had no issues, Memory usage stayed flat for the 10 seconds it took to go through the loop. Note, I am not using ATLAS or BLAS, so maybe the issue lies there? (i don't know if numpy defers the dot-product over to ATLAS or BLAS if they are available)
Perhaps it is an ESRI/Arcpy issue then. I don't see anything that could be doing that, though, as it is very minimal.
From: numpy-discussion-bounces@scipy.org [mailto:numpy-discussion-bounces@scipy.org] On Behalf Of Benjamin Root Sent: Wednesday, January 29, 2014 11:10 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Memory leak in numpy?
Hmmm, I see no reason why that would eat up memory. I just tried it out on my own system (numpy 1.6.1, CentOS 6, python 2.7.1), and had no issues, Memory usage stayed flat for the 10 seconds it took to go through the loop. Note, I am not using ATLAS or BLAS, so maybe the issue lies there? (i don't know if numpy defers the dot-product over to ATLAS or BLAS if they are available)
Upon further investigation, I do believe it is within the scipy code where there is a leak. I commented out my call to processBinaryImage(), which is all scipy code calls, and my memory usage remains flat with approximately a 1MB variation. Any ideas? Right now I am getting around it by checking to see how far I got through my dataset, but I have to restart the program after each memory crash.
From: numpy-discussion-bounces@scipy.org [mailto:numpy-discussion-bounces@scipy.org] On Behalf Of Joseph McGlinchy Sent: Wednesday, January 29, 2014 11:17 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Memory leak in numpy?
Perhaps it is an ESRI/Arcpy issue then. I don't see anything that could be doing that, though, as it is very minimal.
From: numpy-discussion-bounces@scipy.orgmailto:numpy-discussion-bounces@scipy.org [mailto:numpy-discussion-bounces@scipy.org] On Behalf Of Benjamin Root Sent: Wednesday, January 29, 2014 11:10 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Memory leak in numpy?
Hmmm, I see no reason why that would eat up memory. I just tried it out on my own system (numpy 1.6.1, CentOS 6, python 2.7.1), and had no issues, Memory usage stayed flat for the 10 seconds it took to go through the loop. Note, I am not using ATLAS or BLAS, so maybe the issue lies there? (i don't know if numpy defers the dot-product over to ATLAS or BLAS if they are available)
On Wed, Jan 29, 2014 at 7:39 PM, Joseph McGlinchy JMcGlinchy@esri.com wrote:
Upon further investigation, I do believe it is within the scipy code where there is a leak. I commented out my call to processBinaryImage(), which is all scipy code calls, and my memory usage remains flat with approximately a 1MB variation. Any ideas?
I'd suggest continuing along this line, and keep chopping things out until you have a minimal program that still shows the problem -- that'll probably make it much clearer where the problem is actually coming from...
-n
On 29.01.2014 20:44, Nathaniel Smith wrote:
On Wed, Jan 29, 2014 at 7:39 PM, Joseph McGlinchy JMcGlinchy@esri.com wrote:
Upon further investigation, I do believe it is within the scipy code where there is a leak. I commented out my call to processBinaryImage(), which is all scipy code calls, and my memory usage remains flat with approximately a 1MB variation. Any ideas?
I'd suggest continuing along this line, and keep chopping things out until you have a minimal program that still shows the problem -- that'll probably make it much clearer where the problem is actually coming from...
-n
depending on how long the program runs you can try running it under massif the valgrind memory usage proftool, that should give you a good clue where the source is.
Unfortunately I do not have Linux or much time to invest in researching and learning an alternative to Valgrind :/
My current workaround, which works very well, is to move my scipy part of the script to its own script and then use os.system() to call it with the appropriate arguments.
Thanks everyone for the replies! Is there a proper way to close the thread?
-Joe
-----Original Message----- From: numpy-discussion-bounces@scipy.org [mailto:numpy-discussion-bounces@scipy.org] On Behalf Of Julian Taylor Sent: Wednesday, January 29, 2014 11:53 AM To: Discussion of Numerical Python Subject: Re: [Numpy-discussion] Memory leak in numpy?
On 29.01.2014 20:44, Nathaniel Smith wrote:
On Wed, Jan 29, 2014 at 7:39 PM, Joseph McGlinchy JMcGlinchy@esri.com wrote:
Upon further investigation, I do believe it is within the scipy code where there is a leak. I commented out my call to processBinaryImage(), which is all scipy code calls, and my memory usage remains flat with approximately a 1MB variation. Any ideas?
I'd suggest continuing along this line, and keep chopping things out until you have a minimal program that still shows the problem -- that'll probably make it much clearer where the problem is actually coming from...
-n
depending on how long the program runs you can try running it under massif the valgrind memory usage proftool, that should give you a good clue where the source is.
_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
participants (5)
-
Benjamin Root -
Chris Laumann -
Joseph McGlinchy -
Julian Taylor -
Nathaniel Smith