NumPy-Discussion September 2013

numpy-discussion@python.org

68 participants
67 discussions

At.: use less RAM memory and increase execution speed
by Josè Luis Mietta 26 Sep '13

26 Sep '13

Hi experts! I wanna use less RAM memory in my Monte Carlo simulations. In my algorithm I use numpy arrays and xrange() function. I hear that I can reduce RAM used in my lagorithm if I do the next: 1) replace xrange() for range(). 2) replace numpya arrays for python lists 3) use reset() function for deleting useless arrays. Is that true? In adition, I wanna increase execution speed of my code (I use numpy and SciPy functions). How can I apply Cython? Will it help? Please help. Thanks a lot!!

3 2

invalid correlation coefficient from np.ma.corrcoef
by Faraz Mirzaei 26 Sep '13

26 Sep '13

Hi everyone, I'm using np.ma.corrcoef to compute the correlation coefficients among rows of a masked matrix, where the masked elements are the missing data. I've observed that in some cases, the np.ma.corrcoef gives invalid coefficients that are greater than 1 or less than -1. Here's an example: x = array([[ 7, -4, -1, -7, -3, -2], [ 6, -3, 0, 4, 0, 5], [-4, -5, 7, 5, -7, -7], [-5, 5, -8, 0, 1, 4]]) x_ma = np.ma.masked_less_equal(x , -5) C = np.round(np.ma.corrcoef(x_ma), 2) print C [[1.0 0.73 -- -1.68] [0.73 1.0 -0.86 -0.38] [-- -0.86 1.0 --] [-1.68 -0.38 -- 1.0]] As you can see, the [0,3] element is -1.68 which is not a valid correlation coefficient. (Valid correlation coefficients should be between -1 and 1). I looked at the code for np.ma.corrcoef, and this behavior seems to be due to the way that mean values of the rows of the input matrix are computed and subtracted from them. Apparently, the mean value is individually computed for each row, without masking the elements corresponding to the masked elements of the other row of the matrix, with respect to which the correlation coefficient is being computed. I guess the right way should be to recompute the mean value for each row every time that a correlation coefficient is being computed for two rows after propagating pair-wise masked values. Please let me know what you think. Thanks, Faraz

3 8

Valid algorithm for generating a 3D Wiener Process?
by David Goldsmith 25 Sep '13

25 Sep '13

Is this a valid algorithm for generating a 3D Wiener process? (When I graph the results, they certainly look like potential Brownian motion tracks.) def Wiener3D(incr, N): r = incr*(R.randint(3, size=(N,))-1) r[0] = 0 r = r.cumsum() t = 2*np.pi*incr*(R.randint(3, size=(N,))-1) t[0] = 0 t = t.cumsum() p = np.pi*incr*(R.randint(3, size=(N,))-1) p[0] = 0 p = p.cumsum() x = r*np.cos(t)*np.sin(p) y = r*np.sin(t)*np.sin(p) z = r*np.cos(p) return np.array((x,y,z)).T Thanks! DG

3 3

Heads up re: new memory tracking infrastructure in python 3.4
by Nathaniel Smith 25 Sep '13

25 Sep '13

Hi all, Possibly of interest to some folks here, it looks like in Python 3.4 there'll be a generic system for hooking and tracking memory allocations: http://www.python.org/dev/peps/pep-0445/ I'm not planning to do anything with this information myself, but if anyone's been thinking about numpy memory profiling, the new numpy malloc hooks, etc., then it might make sense to see how we could integrate with this more general system. -n

1 0

Numpy newbie question: matrix creation
by Edmondo Porcu 25 Sep '13

25 Sep '13

Dear all, I am a Newbie with Numpy and I would need some advice on how to create a matrix with certain characteristics : - Every entry should be minimum 0 maximum 1 with a step of 0.1 (legal values are 0,0.1,0.2,0.3 etc) - The number of columns of the matrix is a parameter of this matrix creation algorithm - Only the rows where the sum is 1 must be kept Would great appreciate your advice and suggestions Best Regards Edmondo

4 5

Removal of numarray and oldnumeric packages.
by Charles R Harris 24 Sep '13

24 Sep '13

Hi All, I have gotten no feedback on the removal of the numarray and oldnumeric packages. Consequently the removal will take place on 9/28. Scream now or never... Chuck

5 9

Pull request review #3770: Trapezoidal distribution
by Jeremy Hetzel 24 Sep '13

24 Sep '13

I've added a trapezoidal distribution to numpy.random for consideration, pull request 3770: https://github.com/numpy/numpy/pull/3770 Similar to the triangular distribution, the trapezoidal distribution may be used where the underlying distribution is not known, but some knowledge of the limits and mode exists. The trapezoidal distribution generalizes the triangular distribution by allowing the modal values to be expressed as a range instead of a point estimate. The trapezoidal distribution implemented, known as the "generalized trapezoidal distribution," has three additional parameters: growth, decay, and boundary ratio. Adjusting these from the default values create trapezoidal-like distributions with non-linear behavior. Examples can be seen in an R vignette ( http://cran.r-project.org/web/packages/trapezoid/vignettes/trapezoid.pdf ), as well as these papers by J.R. van Dorp and colleagues: 1) van Dorp, J. R. and Kotz, S. (2003) Generalized trapezoidal distributions. Metrika. 58(1):85–97. Preprint available: http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/Metrika2003VanDo… 2) van Dorp, J. R., Rambaud, S.C., Perez, J. G., and Pleguezuelo, R. H. (2007) An elicitation procedure for the generalized trapezoidal distribution with a uniform central stage. Decision Analysis Journal. 4:156–166. Preprint available: http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/DA2007.pdf The docstring for the proposed numpy.random.trapezoidal() is as follows: """ trapezoidal(left, mode1, mode2, right, size=None, m=2, n=2, alpha=1) Draw samples from the generalized trapezoidal distribution. The trapezoidal distribution is defined by minimum (``left``), lower mode (``mode1``), upper mode (``mode1``), and maximum (``right``) parameters. The generalized trapezoidal distribution adds three more parameters: the growth rate (``m``), decay rate (``n``), and boundary ratio (``alpha``) parameters. The generalized trapezoidal distribution simplifies to the trapezoidal distribution when ``m = n = 2`` and ``alpha = 1``. It further simplifies to a triangular distribution when ``mode1 == mode2``. Parameters ---------- left : scalar Lower limit. mode1 : scalar The value where the first peak of the distribution occurs. The value should fulfill the condition ``left <= mode1 <= mode2``. mode2 : scalar The value where the first peak of the distribution occurs. The value should fulfill the condition ``mode1 <= mode2 <= right``. right : scalar Upper limit, should be larger than or equal to `mode2`. size : int or tuple of ints, optional Output shape. Default is None, in which case a single value is returned. m : scalar, optional Growth parameter. n : scalar, optional Decay parameter. alpha : scalar, optional Boundary ratio parameter. Returns ------- samples : ndarray or scalar The returned samples all lie in the interval [left, right]. Notes ----- With ``left``, ``mode1``, ``mode2``, ``right``, ``m``, ``n``, and ``alpha`` parametrized as :math:`a, b, c, d, m, n, \\text{ and } \\alpha`, respectively, the probability density function for the generalized trapezoidal distribution is .. math:: f{\\scriptscriptstyle X}(x\mid\theta) = \\mathcal{C}(\\Theta) \\times \\begin{cases} \\alpha \\left(\\frac{x - \\alpha}{b - \\alpha} \\right)^{m - 1}, & \\text{for } a \\leq x < b \\\\ (1 - \\alpha) \\left(\frac{x - b}{c - b} \\right) + \\alpha, & \\text{for } b \\leq x < c \\\\ \\left(\\frac{d - x}{d - c} \\right)^{n-1}, & \\text{for } c \\leq x \\leq d \\end{cases} with the normalizing constant :math:`\\mathcal{C}(\\Theta)` defined as ..math:: \\mathcal{C}(\\Theta) = \\frac{2mn} {2 \\alpha \\left(b - a\\right) n + \\left(\\alpha + 1 \\right) \\left(c - b \\right)mn + 2 \\left(d - c \\right)m} and where the parameter vector :math:`\\Theta = \\{a, b, c, d, m, n, \\alpha \\}, \\text{ } a \\leq b \\leq c \\leq d, \\text{ and } m, n, \\alpha >0`. Similar to the triangular distribution, the trapezoidal distribution may be used where the underlying distribution is not known, but some knowledge of the limits and mode exists. The trapezoidal distribution generalizes the triangular distribution by allowing the modal values to be expressed as a range instead of a point estimate. The growth, decay, and boundary ratio parameters of the generalized trapezoidal distribution further allow for non-linear behavior to be specified. References ---------- .. [1] van Dorp, J. R. and Kotz, S. (2003) Generalized trapezoidal distributions. Metrika. 58(1):85–97. Preprint available: http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/Metrika2003VanDo… .. [2] van Dorp, J. R., Rambaud, S.C., Perez, J. G., and Pleguezuelo, R. H. (2007) An elicitation proce-dure for the generalized trapezoidal distribution with a uniform central stage. Decision AnalysisJournal. 4:156–166. Preprint available: http://www.seas.gwu.edu/~dorpjr/Publications/JournalPapers/DA2007.pdf Examples -------- Draw values from the distribution and plot the histogram: >>> import matplotlib.pyplot as plt >>> h = plt.hist(np.random.triangular(0, 0.25, 0.75, 1, 100000), bins=200, ... normed=True) >>> plt.show() """ I am unsure if NumPy encourages incorporation of new distributions into numpy.random or instead into separate modules, but found the exercise to be helpful regardless. Thanks, Jeremy

3 4

Binary releases
by Charles R Harris 24 Sep '13

24 Sep '13

Hi All, Numpy 1.8 is about ready for an rc1, which brings up the question of which binary builds so put up on sourceforge. For Windows maybe 1. 32 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC 2. 64 bit windows, python 2.6, 2.7, 3.2, 3.3, compiled with MSVC, linked with MKL I think these should be good for both windows 7 and window 8, correct me if I am wrong. For Mac there is first the question of OS X versions, (10.5?), 10.6, 10.7, 10.8. I don't know if some builds work on more than one OS X version. The 10.5 version is a bit harder to come by than 10.6 and up. It looks like 10.9 is coming up, but it isn't out yet. I have no idea what Python version to match up these, but assuming all of them, then 1. OS X 10.6 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked with Accelerate. 2. OS X 10.7 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked with Accelerate. 3. OS X 10.8 python 2.6, 2.7, 3.2, 3.3, compiled with native compiler, linked with Accelerate. That seems like a lot. It is fairly easy to compile from source on the mac these days, are all those binary packages really needed? I don't know what I am doing with the binary stuff, so any suggestions are welcome. For building it is possible to set up a Macintosh with vmware fusion to handle all of these, but there is time and money involved in that. Any one who is already capable of doing some of these builds is welcome to volunteer. Note, Intel has offered MKL licenses to the open source projects under the NumFocus umbrella, but I don't know of anyone who has taken advantage of that yet or how much time it will take to go through the undoubtedly needed paper work, but I would like to get some of those licenses and move to MSVC so that we can stop rolling around gasping in pain whenever it comes to window builds. Then there is Fortran for windows... Thoughts? Chuck

11 31

Generating a (uniformly distributed) random bit list of length N
by David Goldsmith 23 Sep '13

23 Sep '13

Thanks, St?fan, speed: N ~ 1e9. Thanks again. DG ---------------------------------------------------------------------- > > Message: 1 > Date: Sun, 22 Sep 2013 14:04:09 -0700 > From: David Goldsmith <d.l.goldsmith(a)gmail.com> > Subject: [Numpy-discussion] Generating a (uniformly distributed) > random bit list of length N > To: numpy-discussion(a)scipy.org > Message-ID: > < > CAFtPsZqG7Upjy8s04NPwkn8gJVDVGORu3rPjCVLN6ZKRWnPuTQ(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > Is np.random.randint(2, size=N) the fastest way to do this? Thanks! > > DG >

1 0

openblas?
by Neal Becker 23 Sep '13

23 Sep '13

Does numpy/scipy support building with openblas for blas,lapack instead of atlas?

2 1

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion September 2013