Memory leak with numpy master
Hi list, I think that I am hit a memory leak with numpy master. The following code enables to reproduce it: ________________________________________________________________________________ import numpy as np n = 100 m = np.eye(n) for i in range(30000): #np.linalg.slogdet(m) t, result_t = np.linalg.linalg._commonType(m) a = np.linalg.linalg._fastCopyAndTranspose(t, m) pivots = np.zeros((n,), np.linalg.linalg.fortran_int) results = np.linalg.lapack_lite.dgetrf(n, n, a, n, pivots, 0) d = np.diagonal(a) if not i % 1000: print i ________________________________________________________________________________ If you execute this code, you'll see the memory go steadily up. The reason that I came up with such a strange looking code is that in my codebase, I do repeated calls to np.linalg.slogdet. I came up with the code above by simplifying what is done in slogdet. I don't think that I can simplify any further and still reproduce the memory leak. Should I submit a bug report (in other words, can people reproduce?)? Cheers, Gaël
Hi, with numpy '1.6.1', I have no problem. With numpy 1.7.0b2, I can reproduce the problem. HTH Fred On Mon, Sep 24, 2012 at 1:04 PM, Gael Varoquaux <gael.varoquaux@normalesup.org> wrote:
Hi list,
I think that I am hit a memory leak with numpy master. The following code enables to reproduce it:
________________________________________________________________________________ import numpy as np n = 100 m = np.eye(n) for i in range(30000): #np.linalg.slogdet(m)
t, result_t = np.linalg.linalg._commonType(m) a = np.linalg.linalg._fastCopyAndTranspose(t, m)
pivots = np.zeros((n,), np.linalg.linalg.fortran_int) results = np.linalg.lapack_lite.dgetrf(n, n, a, n, pivots, 0) d = np.diagonal(a)
if not i % 1000: print i ________________________________________________________________________________
If you execute this code, you'll see the memory go steadily up.
The reason that I came up with such a strange looking code is that in my codebase, I do repeated calls to np.linalg.slogdet. I came up with the code above by simplifying what is done in slogdet. I don't think that I can simplify any further and still reproduce the memory leak.
Should I submit a bug report (in other words, can people reproduce?)?
Cheers,
Gaël _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@scipy.org http://mail.scipy.org/mailman/listinfo/numpy-discussion
On Mon, Sep 24, 2012 at 7:19 PM, Gael Varoquaux <gael.varoquaux@normalesup.org> wrote:
Hi Fred,
On Mon, Sep 24, 2012 at 02:17:16PM -0400, Frédéric Bastien wrote:
with numpy '1.6.1', I have no problem.
With numpy 1.7.0b2, I can reproduce the problem.
OK, thanks. I think that I'll start a bisect to figure out when it crept in.
This also seems to reproduce it: while True: a = np.zeros((1000, 1000)) a.diagonal() which means I probably forgot a DECREF while doing the PyArray_Diagonal changes... --n
On Mon, Sep 24, 2012 at 7:45 PM, Nathaniel Smith <njs@pobox.com> wrote:
On Mon, Sep 24, 2012 at 7:19 PM, Gael Varoquaux <gael.varoquaux@normalesup.org> wrote:
Hi Fred,
On Mon, Sep 24, 2012 at 02:17:16PM -0400, Frédéric Bastien wrote:
with numpy '1.6.1', I have no problem.
With numpy 1.7.0b2, I can reproduce the problem.
OK, thanks. I think that I'll start a bisect to figure out when it crept in.
This also seems to reproduce it:
while True: a = np.zeros((1000, 1000)) a.diagonal()
which means I probably forgot a DECREF while doing the PyArray_Diagonal changes...
On Mon, Sep 24, 2012 at 07:59:11PM +0100, Nathaniel Smith wrote:
which means I probably forgot a DECREF while doing the PyArray_Diagonal changes...
Awesome. I can confirm that this fixes the problem. Script below to check. You are my hero! Gael _______________________________________________________________________________ os.system('python setup.py build_ext -i') def get_mem_usage(): pid = os.getpid() usage = open('/proc/%i/statm' % pid, 'r').read().split(' ')[0] return int(usage) import numpy as np n = 100 m = np.eye(n) for i in range(30000): #np.linalg.slogdet(m) t, result_t = np.linalg.linalg._commonType(m) a = np.linalg.linalg._fastCopyAndTranspose(t, m) pivots = np.zeros((n,), np.linalg.linalg.fortran_int) results = np.linalg.lapack_lite.dgetrf(n, n, a, n, pivots, 0) d = np.diagonal(a) if i == 0: initial_usage = get_mem_usage() if not i % 1000: usage = get_mem_usage() print i, usage if usage > 4*initial_usage: sys.exit(10) sys.exit(0) _______________________________________________________________________________
participants (3)
-
Frédéric Bastien -
Gael Varoquaux -
Nathaniel Smith