[Numpy-discussion] Python memory management issues using, Linux. Maybe Numpy related.

[Jeffrey Spencer]

There are a few places where we (improperly) directly call malloc()
instead of PyMem_Malloc(), so yes, you should rebuild numpy against
TCMalloc in addition to the Python interpreter.

How do I build against the -ltcmalloc library? Just not sure how to 
input compile time arguments with the build script.

from numpy import arange, sum
>
>  for x in range(10000000):
>            inhibVal = sum(arange(15))

Memory usage stays constant with Ubuntu 11.04, 64-bit, using the numpy
1.5.1 package from ubuntu, and using 1.6.1.dev-a265004.


efiring at manini:~$ uname -a
Linux manini 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC
2011 x86_64 x86_64 x86_64 GNU/Linux


Eric

Also noticed that the same issue occurs as described with using the prod 
instead of sum. Other commands like copy and reshape have no issues. I 
have a machine with 11.04 on it so I will give that a go later to see 
what goes.

Thanks, Jeff

Message: 2 Date: Sun, 22 May 2011 12:32:43 -0500 From: Robert Kern 
<robert.kern at gmail.com> Subject: Re: [Numpy-discussion] Python memory 
management issues using Linux. Maybe Numpy related. To: Discussion of 
Numerical Python <numpy-discussion at scipy.org> Message-ID: 
<BANLkTikUgYm8COoZEhNP2tJGYRb_PBH1og at mail.gmail.com> Content-Type: 
text/plain; charset=UTF-8 On Sat, May 21, 2011 at 23:27, Jeffrey Spencer 
<jeffspencerd at gmail.com> wrote:

> >  I have had issues with Linux (Ubuntu 10.04 x86_64) not releasing memory
> >  back to the OS. I have code here the exhibits the issue:
> >
> >  http://stackoverflow.com/questions/5975255/memory-allocated-to-python-in-the-os-is-never-released-back-in-linux-even-after-g
> >
> >  Then I read athttp://pushingtheweb.com/2010/06/python-and-tcmalloc/
> >  that this was a problem fixed by compiling your own version of Python
> >  using the TCMalloc library which I have now done. The issue is still
> >  exhibited so I was keen to know if possibly it was because numpy is
> >  compiled and linked to the default memory management library (not sure
> >  if this is true or how this works but would I also need to compile numpy
> >  linked to the TCMalloc library). Let me know if anyone has any insight
> >  on this issue. Or if it isn't related to numpy at all.
There are a few places where we (improperly) directly call malloc()
instead of PyMem_Malloc(), so yes, you should rebuild numpy against
TCMalloc in addition to the Python interpreter.

-- Robert Kern "I have come to believe that the whole world is an 
enigma, a harmless enigma that is made terrible by our own mad attempt 
to interpret it as though it had an underlying truth." ? -- Umberto Eco

-- 
________________________
Jeffrey Spencer
jeffspencerd at gmail.com

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/numpy-discussion/attachments/20110523/95251754/attachment.html>