Mailman 3 May 2011 - NumPy-Discussion

[ANN] Guaranteed solution of nonlinear equation(s)
by Dmitrey 24 May '11

24 May '11

Hi all, I have made my free solver interalg (http://openopt.org/interalg) be capable of solving nonlinear equations and systems of them. Unlike scipy optimize fsolve it doesn't matter which functions are involved - convex, nonconvex, multiextremum etc. Even some discontinuous funcs can be handled. If no solution exists, interalg determines it rather quickly. For more info see http://forum.openopt.org/viewtopic.php?id=423 Regards, D.

1 0

filtering rows from a numpy.ndarray
by Jan Wuyts 24 May '11

24 May '11

I've read in a datafile to an ndarray with this command: data=np.genfromtxt('n.txt', delimiter='\t', dtype=[('paramname','S26'), ('study','S26'), ('class','S10'), ('count','f4'), ('average','f4'), ('sd','f4')) I would like to filter the array to only rows where paramname == 'bmi'. What would be the most elegant way of doing this? I found 1 way for doing this but it doesn't look very efficient nor elegant. bmidata=np.asarray([x for x in data if x['paramname']=='bmi'], dtype=data.dtype) any simpler ways? Thanks, Jan.

2 1

random seed replicate 2d randn with 1d loop
by josef.pktd＠gmail.com 23 May '11

23 May '11

I have a function in two versions, one vectorized, one with loop the vectorized function gets all randn variables in one big array rvs = distr.rvs(args, **{'size':(nobs, nrep)}) the looping version has: for irep in xrange(nrep): rvs = distr.rvs(args, **{'size':nobs}) the rest should be identical (except for vectorization Is there a guarantee that the 2d arrays are filled up in a specific order so that the loop and vectorized version produce the same result, given the same seed? In my example it doesn't work, ( bootstrap p-values for Anderson-Darling A^2 with estimated loc and scale) >>> np.random.seed(8765679) >>> [bootstrap2(c/ (1 + 4./nobs - 25./nobs**2), NewNorm(), args=(0,1), nobs=nobs, nrep=10000) for c in [0.576, 0.656, 0.787, 0.918, 1.092]] [0.14449999999999999, 0.0901, 0.042200000000000001, 0.0212, 0.0076] >>> np.random.seed(8765679) >>> [bootstrap(NewNorm(), args=(0,1), nobs=nobs, nrep=10000, value=c/ (1 + 4./nobs - 25./nobs**2)) for c in [0.576, 0.656, 0.787, 0.918, 1.092]] [0.14710000000000001, 0.092899999999999996, 0.044499999999999998, 0.022800000000000001, 0.0080999999999999996] >>> np.array([15.0, 10.0, 5.0, 2.5, 1.0])/100. #Stephens array([ 0.15 , 0.1 , 0.05 , 0.025, 0.01 ]) Josef

4 7

Re: [Numpy-discussion] Python memory management issues using, Linux. Maybe Numpy related.
by Jeffrey Spencer 23 May '11

23 May '11

I realized if I re-build numpy it takes the interpreter compile options or you can edit site.cfg. I actually solved the issue by getting the newest svn copy of numpy. It must be something with 1.6b2. The newest SVN has no issues with either library and calls to sum or prod. I was getting memory leak with both the tcmalloc and malloc with the 1.6b2 version of numpy. There are a few places where we (improperly) directly call malloc() instead of PyMem_Malloc(), so yes, you should rebuild numpy against TCMalloc in addition to the Python interpreter. How do I build against the -ltcmalloc library? Just not sure how to input compile time arguments with the build script. from numpy import arange, sum > > for x in range(10000000): > inhibVal = sum(arange(15)) Memory usage stays constant with Ubuntu 11.04, 64-bit, using the numpy 1.5.1 package from ubuntu, and using 1.6.1.dev-a265004. efiring@manini:~$ uname -a Linux manini 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux Eric Also noticed that the same issue occurs as described with using the prod instead of sum. Other commands like copy and reshape have no issues. I have a machine with 11.04 on it so I will give that a go later to see what goes. Thanks, Jeff Message: 2 Date: Sun, 22 May 2011 12:32:43 -0500 From: Robert Kern <robert.kern(a)gmail.com> Subject: Re: [Numpy-discussion] Python memory management issues using Linux. Maybe Numpy related. To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> Message-ID: <BANLkTikUgYm8COoZEhNP2tJGYRb_PBH1og(a)mail.gmail.com> Content-Type: text/plain; charset=UTF-8 On Sat, May 21, 2011 at 23:27, Jeffrey Spencer <jeffspencerd(a)gmail.com> wrote: > > I have had issues with Linux (Ubuntu 10.04 x86_64) not releasing memory > > back to the OS. I have code here the exhibits the issue: > > > > http://stackoverflow.com/questions/5975255/memory-allocated-to-python-in-th… > > > > Then I read athttp://pushingtheweb.com/2010/06/python-and-tcmalloc/ > > that this was a problem fixed by compiling your own version of Python > > using the TCMalloc library which I have now done. The issue is still > > exhibited so I was keen to know if possibly it was because numpy is > > compiled and linked to the default memory management library (not sure > > if this is true or how this works but would I also need to compile numpy > > linked to the TCMalloc library). Let me know if anyone has any insight > > on this issue. Or if it isn't related to numpy at all. There are a few places where we (improperly) directly call malloc() instead of PyMem_Malloc(), so yes, you should rebuild numpy against TCMalloc in addition to the Python interpreter. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco -- ________________________ Jeffrey Spencer jeffspencerd(a)gmail.com

1 0

Re: [Numpy-discussion] Python memory management issues using, Linux. Maybe Numpy related.
by Jeffrey Spencer 23 May '11

23 May '11

There are a few places where we (improperly) directly call malloc() instead of PyMem_Malloc(), so yes, you should rebuild numpy against TCMalloc in addition to the Python interpreter. How do I build against the -ltcmalloc library? Just not sure how to input compile time arguments with the build script. from numpy import arange, sum > > for x in range(10000000): > inhibVal = sum(arange(15)) Memory usage stays constant with Ubuntu 11.04, 64-bit, using the numpy 1.5.1 package from ubuntu, and using 1.6.1.dev-a265004. efiring@manini:~$ uname -a Linux manini 2.6.38-8-generic #42-Ubuntu SMP Mon Apr 11 03:31:24 UTC 2011 x86_64 x86_64 x86_64 GNU/Linux Eric Also noticed that the same issue occurs as described with using the prod instead of sum. Other commands like copy and reshape have no issues. I have a machine with 11.04 on it so I will give that a go later to see what goes. Thanks, Jeff Message: 2 Date: Sun, 22 May 2011 12:32:43 -0500 From: Robert Kern <robert.kern(a)gmail.com> Subject: Re: [Numpy-discussion] Python memory management issues using Linux. Maybe Numpy related. To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> Message-ID: <BANLkTikUgYm8COoZEhNP2tJGYRb_PBH1og(a)mail.gmail.com> Content-Type: text/plain; charset=UTF-8 On Sat, May 21, 2011 at 23:27, Jeffrey Spencer <jeffspencerd(a)gmail.com> wrote: > > I have had issues with Linux (Ubuntu 10.04 x86_64) not releasing memory > > back to the OS. I have code here the exhibits the issue: > > > > http://stackoverflow.com/questions/5975255/memory-allocated-to-python-in-th… > > > > Then I read athttp://pushingtheweb.com/2010/06/python-and-tcmalloc/ > > that this was a problem fixed by compiling your own version of Python > > using the TCMalloc library which I have now done. The issue is still > > exhibited so I was keen to know if possibly it was because numpy is > > compiled and linked to the default memory management library (not sure > > if this is true or how this works but would I also need to compile numpy > > linked to the TCMalloc library). Let me know if anyone has any insight > > on this issue. Or if it isn't related to numpy at all. There are a few places where we (improperly) directly call malloc() instead of PyMem_Malloc(), so yes, you should rebuild numpy against TCMalloc in addition to the Python interpreter. -- Robert Kern "I have come to believe that the whole world is an enigma, a harmless enigma that is made terrible by our own mad attempt to interpret it as though it had an underlying truth." ? -- Umberto Eco -- ________________________ Jeffrey Spencer jeffspencerd(a)gmail.com

1 0

Re: [Numpy-discussion] Python memory management issues using, Linux. Maybe Numpy, related.
by Jeffrey Spencer 23 May '11

23 May '11

Further isolated the error and it seems to come in from the sum function in numpy. I will probe further later but the code below causes the memory usage to go up and up. If you change sum below to some other function there are no issues. from numpy import arange, sum for x in range(10000000): inhibVal = sum(arange(15)) Cheers, Jeff Message: 3 Date: Sun, 22 May 2011 09:32:10 -0500 From: Bruce Southey<bsouthey(a)gmail.com> Subject: Re: [Numpy-discussion] Python memory management issues using Linux. Maybe Numpy, related. To: Discussion of Numerical Python<numpy-discussion(a)scipy.org> Message-ID:<BANLkTinAuFp0o6OTwvP=GDhBKNkhkwH-vg(a)mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 On Sun, May 22, 2011 at 8:42 AM, Ralf Gommers <ralf.gommers(a)googlemail.com> wrote: > > > > > > On Sun, May 22, 2011 at 2:54 PM, Jeffrey Spencer<jeffspencerd(a)gmail.com> > > wrote: >> >> >> >> Sorry. I attached here the class and script to run it. It is just a >> >> minimalistic example of what I'm really doing that exhibits the same >> >> behavior. All the script does is run at.createSpecific after instantiating >> >> the class. I have seen this exhibited on many other cases but it runs fine >> >> in Windows XP. Here is what the script does: >> >> >> >> import MemoryLeak >> >> at = MemoryLeak.Model() >> >> at.createSpecific() >> >> > > That manages to eat up all my cpu cycles for about 5 minutes, but no leak > > (on OS X). I also don't see anything in the code that can potentially give > > problems. I suggest you strip this down a lot more so you can isolate the > > problem. > > > > If in your original code you were making a lot of figures in one of those > > for loops without explicitly closing them, that could perhaps be a problem. > > But you removed all I/O and matplotlib related code already (except for > > imports and some keywords). > > > > Ralf > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion(a)scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > A good OS is likely to keep Python module in memory just because it knows that it might be requested again. However, from the little I know (too much LWN), if Linux needs that memory it will reclaim that memory only when it needs it or part of some other process. Measuring memory usage using simple tools (like ps or top) are very far from sufficient, so exactly how do you know that the memory has not been released? Bruce -- ________________________ Jeffrey Spencer jeffspencerd(a)gmail.com

3 2

Python memory management issues using Linux. Maybe Numpy related.
by Jeffrey Spencer 22 May '11

22 May '11

I have had issues with Linux (Ubuntu 10.04 x86_64) not releasing memory back to the OS. I have code here the exhibits the issue: http://stackoverflow.com/questions/5975255/memory-allocated-to-python-in-th… Then I read at http://pushingtheweb.com/2010/06/python-and-tcmalloc/ that this was a problem fixed by compiling your own version of Python using the TCMalloc library which I have now done. The issue is still exhibited so I was keen to know if possibly it was because numpy is compiled and linked to the default memory management library (not sure if this is true or how this works but would I also need to compile numpy linked to the TCMalloc library). Let me know if anyone has any insight on this issue. Or if it isn't related to numpy at all. -- ______________________________________________________ Jeffrey Spencer Department of Electronics and Electrical Engineering Nonlinear Signal Processing Group The University of Melbourne jeffspencerd(a)gmail.com (Mob.) 044 991 7726

3 2

Re: [Numpy-discussion] Python memory management issues using, Linux. Maybe Numpy, related.
by Jeffrey Spencer 22 May '11

22 May '11

Message: 3 Date: Sun, 22 May 2011 09:32:10 -0500 From: Bruce Southey<bsouthey(a)gmail.com> Subject: Re: [Numpy-discussion] Python memory management issues using Linux. Maybe Numpy, related. To: Discussion of Numerical Python<numpy-discussion(a)scipy.org> Message-ID:<BANLkTinAuFp0o6OTwvP=GDhBKNkhkwH-vg(a)mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1 On Sun, May 22, 2011 at 8:42 AM, Ralf Gommers <ralf.gommers(a)googlemail.com> wrote: > > > > > > On Sun, May 22, 2011 at 2:54 PM, Jeffrey Spencer<jeffspencerd(a)gmail.com> > > wrote: >> >> >> >> Sorry. I attached here the class and script to run it. It is just a >> >> minimalistic example of what I'm really doing that exhibits the same >> >> behavior. All the script does is run at.createSpecific after instantiating >> >> the class. I have seen this exhibited on many other cases but it runs fine >> >> in Windows XP. Here is what the script does: >> >> >> >> import MemoryLeak >> >> at = MemoryLeak.Model() >> >> at.createSpecific() >> >> > > That manages to eat up all my cpu cycles for about 5 minutes, but no leak > > (on OS X). I also don't see anything in the code that can potentially give > > problems. I suggest you strip this down a lot more so you can isolate the > > problem. > > > > If in your original code you were making a lot of figures in one of those > > for loops without explicitly closing them, that could perhaps be a problem. > > But you removed all I/O and matplotlib related code already (except for > > imports and some keywords). > > > > Ralf > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion(a)scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > A good OS is likely to keep Python module in memory just because it knows that it might be requested again. However, from the little I know (too much LWN), if Linux needs that memory it will reclaim that memory only when it needs it or part of some other process. Measuring memory usage using simple tools (like ps or top) are very far from sufficient, so exactly how do you know that the memory has not been released? Bruce Bruce, The reason I have the issue is it ends up swapping memory over to the swap. After this, the whole computer runs at a crawl. Even if as your saying it is allocated as free just not reclaimed from Python. The OS should try to get the memory back before swapping everything. I even have my swappiness set lower to avoid sending things to the swap. Jeff -- ________________________ Jeffrey Spencer jeffspencerd(a)gmail.com

1 0

Re: [Numpy-discussion] Python memory management issues using Linux. Maybe Numpy, related.
by Jeffrey Spencer 22 May '11

22 May '11

Sorry. I attached here the class and script to run it. It is just a minimalistic example of what I'm really doing that exhibits the same behavior. All the script does is run at.createSpecific after instantiating the class. I have seen this exhibited on many other cases but it runs fine in Windows XP. Here is what the script does: import MemoryLeak at = MemoryLeak.Model() at.createSpecific() Thanks, Jeff

3 3

datarray -- a couple of questions
by Ben Walsh 22 May '11

22 May '11

Hi I've been looking at various labelled-array packages, including Pandas (https://github.com/wesm/pandas) and datarray (https://github.com/fperez/datarray). I was quite interested in the design discussion for datarray, and I had a couple of questions: 1) Is this mailing list the correct place to ask about this? 2) Pandas supports mixed data types (eg. one column of floats, another column of strings). Does the datarray prototype support this? Is it simply a case of using a numpy.ndarray with dtype='object'? Cheers Ben

2 1