Mailman 3 July 2010 - NumPy-Discussion

Re: [Numpy-discussion] [Numpy-svn] r8413 - trunk/numpy/lib - Author: oliphant - Add percentile function.
by Sebastian Haase 08 Jul '10

08 Jul '10

isn't this related to http://projects.scipy.org/numpy/ticket/626 percentile() and clamp() which was set to invalid -Sebastian On Sun, May 16, 2010 at 12:11 AM, <numpy-svn(a)scipy.org> wrote: > Author: oliphant > Date: 2010-05-15 17:11:10 -0500 (Sat, 15 May 2010) > New Revision: 8413 > > Modified: > trunk/numpy/lib/function_base.py > Log: > Add percentile function. > > Modified: trunk/numpy/lib/function_base.py > =================================================================== > --- trunk/numpy/lib/function_base.py 2010-05-13 12:47:23 UTC (rev 8412) > +++ trunk/numpy/lib/function_base.py 2010-05-15 22:11:10 UTC (rev 8413) > @@ -1,6 +1,6 @@ > __docformat__ = "restructuredtext en" > __all__ = ['select', 'piecewise', 'trim_zeros', > - 'copy', 'iterable', > + 'copy', 'iterable', 'percentile', > 'diff', 'gradient', 'angle', 'unwrap', 'sort_complex', 'disp', > 'extract', 'place', 'nansum', 'nanmax', 'nanargmax', > 'nanargmin', 'nanmin', 'vectorize', 'asarray_chkfinite', 'average', > @@ -2804,7 +2804,7 @@ > > See Also > -------- > - mean > + mean, percentile > > Notes > ----- > @@ -2863,6 +2863,132 @@ > # and check, use out array. > return mean(sorted[indexer], axis=axis, out=out) > > +def percentile(a, q, axis=None, out=None, overwrite_input=False): > + """ > + Compute the qth percentile of the data along the specified axis. > + > + Returns the qth percentile of the array elements. > + > + Parameters > + ---------- > + a : array_like > + Input array or object that can be converted to an array. > + q : float in range of [0,100] (or sequence of floats) > + percentile to compute which must be between 0 and 100 inclusive > + axis : {None, int}, optional > + Axis along which the percentiles are computed. The default (axis=None) > + is to compute the median along a flattened version of the array. > + out : ndarray, optional > + Alternative output array in which to place the result. It must > + have the same shape and buffer length as the expected output, > + but the type (of the output) will be cast if necessary. > + overwrite_input : {False, True}, optional > + If True, then allow use of memory of input array (a) for > + calculations. The input array will be modified by the call to > + median. This will save memory when you do not need to preserve > + the contents of the input array. Treat the input as undefined, > + but it will probably be fully or partially sorted. Default is > + False. Note that, if `overwrite_input` is True and the input > + is not already an ndarray, an error will be raised. > + > + Returns > + ------- > + pcntile : ndarray > + A new array holding the result (unless `out` is specified, in > + which case that array is returned instead). If the input contains > + integers, or floats of smaller precision than 64, then the output > + data-type is float64. Otherwise, the output data-type is the same > + as that of the input. > + > + See Also > + -------- > + mean, median > + > + Notes > + ----- > + Given a vector V of length N, the qth percentile of V is the qth ranked > + value in a sorted copy of V. A weighted average of the two nearest neighbors > + is used if the normalized ranking does not match q exactly. > + The same as the median if q is 0.5; the same as the min if q is 0; > + and the same as the max if q is 1 > + > + Examples > + -------- > + >>> a = np.array([[10, 7, 4], [3, 2, 1]]) > + >>> a > + array([[10, 7, 4], > + [ 3, 2, 1]]) > + >>> np.percentile(a, 0.5) > + 3.5 > + >>> np.percentile(a, 0.5, axis=0) > + array([ 6.5, 4.5, 2.5]) > + >>> np.percentile(a, 0.5, axis=1) > + array([ 7., 2.]) > + >>> m = np.percentile(a, 0.5, axis=0) > + >>> out = np.zeros_like(m) > + >>> np.percentile(a, 0.5, axis=0, out=m) > + array([ 6.5, 4.5, 2.5]) > + >>> m > + array([ 6.5, 4.5, 2.5]) > + >>> b = a.copy() > + >>> np.percentile(b, 0.5, axis=1, overwrite_input=True) > + array([ 7., 2.]) > + >>> assert not np.all(a==b) > + >>> b = a.copy() > + >>> np.percentile(b, 0.5, axis=None, overwrite_input=True) > + 3.5 > + >>> assert not np.all(a==b) > + > + """ > + if q == 0: > + return a.min(axis=axis, out=out) > + elif q == 100: > + return a.max(axis=axis, out=out) > + > + if overwrite_input: > + if axis is None: > + sorted = a.ravel() > + sorted.sort() > + else: > + a.sort(axis=axis) > + sorted = a > + else: > + sorted = sort(a, axis=axis) > + if axis is None: > + axis = 0 > + > + return _compute_qth_percentile(sorted, q, axis, out) > + > +# handle sequence of q's without calling sort multiple times > +def _compute_qth_percentile(sorted, q, axis, out): > + if not isscalar(q): > + return [_compute_qth_percentile(sorted, qi, axis, out) > + for qi in q] > + q = q / 100.0 > + if (q < 0) or (q > 1): > + raise ValueError, "percentile must be either in the range [0,100]" > + > + indexer = [slice(None)] * sorted.ndim > + Nx = sorted.shape[axis] > + index = q*(Nx-1) > + i = int(index) > + if i == index: > + indexer[axis] = slice(i, i+1) > + weights = array(1) > + sumval = 1.0 > + else: > + indexer[axis] = slice(i, i+2) > + j = i + 1 > + weights = array([(j - index), (index - i)],float) > + wshape = [1]*sorted.ndim > + wshape[axis] = 2 > + weights.shape = wshape > + sumval = weights.sum() > + > + # Use add.reduce in both cases to coerce data type as well as > + # check and use out array. > + return add.reduce(sorted[indexer]*weights, axis=axis, out=out)/sumval > + > def trapz(y, x=None, dx=1.0, axis=-1): > """ > Integrate along the given axis using the composite trapezoidal rule. > > _______________________________________________ > Numpy-svn mailing list > Numpy-svn(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-svn >

2 1

effect of shape=None (the default) in format.open_memmap
by David Goldsmith 08 Jul '10

08 Jul '10

Hi, I'm trying to wrap my brain around the affect of leaving shape=None (the default) in format.open_memmap. First, I get that it's only even seen if the file is opened in write mode. Then, write_array_header_1_0 is called with dict d as second parameter, w/, as near as I can see, d['shape'] still = None. write_array_header_1_0 is a little opaque to me, but as near as I can tell, shape = None is then written as is to the file's header. Here's where things get a little worrisome/confusing. Looking ahead, the next function in the source is read_array_header_1_0, in which we see the following comment: "...The keys are strings 'shape' : tuple of int..." Then later in the code we see: # Sanity-check the values. if (not isinstance(d['shape'], tuple) or not numpy.all([isinstance(x, (int,long)) for x in d['shape']])): msg = "shape is not valid: %r" raise ValueError(msg % (d['shape'],)) Unless I'm missing something, if shape=None, this ValueError will be raised, correct? So it appears as if the default value for shape in the original function, open_memmap, will produce a header that would ultimately result in a "defective" file, at least as far as read_array_header_1_0 is concerned. A) Am I missing something (e.g., a numpy-wide default substitution for shape if it happens to equal None) that results in this conclusion being incorrect? B) If I am correct, "feature" or "bug"? DG

1 1

User-defined metadata in dtype
by Lluís 07 Jul '10

07 Jul '10

I've been trying to embed extra information into dtype, such that whether I have structured arrays or not, I can alwas have access to: * A name of the field. This is already present on structured arrays, but lost whenever you access a field. * A description of the field. Tried to build a dtype with 'titles', but there can be no repeated titles, which is not always possible. * Units of the field. By using __array_wrap__ I could always carry along the current units of the information on every field of the structured array (or the units of the array if it's a plain ndarray) with a package like 'units'. My current implementation uses __array_wrap__, __getitem__ and __setitem__ to maintain these fields, but I'm still not sure if that information should better reside inside dtype itself (I've tried to subclass and monkeypatch dtype from python with no success). My intent is to maintain all that information for later retrieval when plotting it into figures. Read you, Lluis -- "And it's much the same thing with knowledge, for whenever you learn something new, the whole world becomes that much richer." -- The Princess of Pure Reason, as told by Norton Juster in The Phantom Tollbooth

1 0

finfo.eps v. finfo.epsneg
by David Goldsmith 07 Jul '10

07 Jul '10

>>> np.finfo('float64').eps # returns a scalar 2.2204460492503131e-16 >>> np.finfo('float64').epsneg # returns an array array(1.1102230246251565e-16) Bug or feature? DG

2 1

OT? Distutils extension with shared libs
by Fabrice Silva 07 Jul '10

07 Jul '10

I know it is not directly related to numpy (even if it uses numpy.distutils), but I ask you folks how do you deal with code depending on other libs. In libIM7 projet ( https://launchpad.net/libim7 ), I wrap code from a device constructor with ctypes in order to read Particle Image Velocimetry (PIV) files stored by their software (format im7 and vc7). There is a dependency on zlib which is easy to solve in linux (installing zlib-dev package in debian). But as I want to use it also in windows (sharing the commercial dongle amongst various colleagues is a unconfortable solution), I am trying to configure the setup.py both for win and linux. But I am new to dev in windows... My questions are then: - how do you deal with dependencies in distutils? - what do you need to build against zlib (or another lib) in windows using distutils ? Thanks Fabricio

3 3

Re: [Numpy-discussion] reverse cumsum?
by Jonathan Stickel 07 Jul '10

07 Jul '10

On 7/6/10 10:42 , numpy-discussion-request(a)scipy.org wrote: > Date: Tue, 06 Jul 2010 10:02:57 -0400 > From: Alan G Isaac<aisaac(a)american.edu> > Subject: Re: [Numpy-discussion] reverse cumsum? > To: Discussion of Numerical Python<numpy-discussion(a)scipy.org> > Message-ID:<4C333791.5010505(a)american.edu> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > On 7/6/2010 9:56 AM, Ken Basye wrote: >> > Is there a simple way to get a cumsum in reverse order? >>>> >>> x = np.arange(10) >>>> >>> x[::-1].cumsum()[::-1] > array([45, 45, 44, 42, 39, 35, 30, 24, 17, 9]) > > Is that what you want? > > Alan Isaac Or, you can do: In [1]: a = np.arange(10) In [5]: np.sum(a) - np.cumsum(a) Out[5]: array([45, 44, 42, 39, 35, 30, 24, 17, 9, 0]) Jonathan

3 3

numpy on windows 64 bit
by Robin 06 Jul '10

06 Jul '10

Hi, I am having some problems with win64 with all my tests failing. I installed amd64 Python from Python.org and numpy and scipy from http://www.lfd.uci.edu/~gohlke/pythonlibs/ I noticed that on windows sys.maxint is the 32bit value (2147483647

3 4

[ANN] la 0.4, the labeled array
by Keith Goodman 06 Jul '10

06 Jul '10

The main class of the la package is a labeled array, larry. A larry consists of data and labels. The data is stored as a NumPy array and the labels as a list of lists (one list per dimension). Alignment by label is automatic when you add (or subtract, multiply, divide) two larrys. The focus of this release was binary operations between unaligned larrys with user control of the join method (five available) and the fill method. A general binary function, la.binaryop(), was added as were the convenience functions add, subtract, multiply, divide. Supporting functions such as la.align(), which aligns two larrys, were also added. download http://pypi.python.org/pypi/la doc http://larry.sourceforge.net code http://github.com/kwgoodman/la list1 http://groups.google.ca/group/pystatsmodels list2 http://groups.google.com/group/labeled-array RELEASE NOTES New larry methods - ismissing: A bool larry with element-wise marking of missing values - take: A copy of the specified elements of a larry along an axis New functions - rand: Random samples from a uniform distribution - randn: Random samples from a Gaussian distribution - missing_marker: Return missing value marker for the given larry - ismissing: A bool Numpy array with element-wise marking of missing values - correlation: Correlation of two Numpy arrays along the specified axis - split: Split into train and test data along given axis - listmap_fill: Index map a list onto another and index of unmappable elements - listmap_fill: Cython version of listmap_fill - align: Align two larrys using one of five join methods - info: la package information such as version number and HDF5 availability - binaryop: Binary operation on two larrys with given function and join method - add: Sum of two larrys using given join and fill methods - subtract: Difference of two larrys using given join and fill methods - multiply: Multiply two larrys element-wise using given join and fill methods - divide: Divide two larrys element-wise using given join and fill methods Enhancements - listmap now has option to ignore unmappable elements instead of KeyError - listmap.pyx now has option to ignore unmappable elements instead of KeyError - larry.morph() is much faster as are methods, such as merge, that use it Breakage from la 0.3 - Development moved from launchpad to github - func.py and afunc.py renamed flarry.py and farray.py to match new flabel.py. Broke: "from la.func import stack"; Did not break: "from la import stack" - Default binary operators (+, -, ...) no longer raise an error when no labels overlap Bug fixes - #590270 Index with 1d array bug: lar[1darray,:] worked; lar[1darray] crashed

1 0

Download Microsoft C/C++ compiler for use with Python 2.6/2.7 ASAP
by Sturla Molden 06 Jul '10

06 Jul '10

Microsoft has withdrawn VS2008 in favor of VS2010. The express version is also unavailable for download. We can still get a VC++ 2008 compiler required to build extensions for the official Python 2.6 and 2.7 binary installers here (Windows 7 SDK for .NET 3.5 SP1): http://www.microsoft.com/downloads/details.aspx?familyid=71DEB800-C591-4F97… <http://www.microsoft.com/downloads/details.aspx?familyid=71DEB800-C591-4F97…> Download today, before it goes away! It is possible to build C and Fortran extensions for Python 2.6/2.7 on x86 using mingw. Microsoft's compiler is required for C++ or amd64 though. (Intel's C/C++ compiler requires VS2008, which has now perished.) Microsoft has now published a download for Windows 7 SDK for .NET 4. It has the VC++ 2010 compiler. It can be a matter of days before the VC++ 2008 compiler is totally unavailable. Sturla Molden

1 0

reverse cumsum?
by Ken Basye 06 Jul '10

06 Jul '10

Hi, Is there a simple way to get a cumsum in reverse order? So far, the best I've come up with is to use fancy indexing twice to reverse things: >>> x = np.arange(10) >>> np.cumsum(x[np.arange(9, -1, -1)])[np.arange(9, -1, -1)] array([45, 45, 44, 42, 39, 35, 30, 24, 17, 9]) If it matters, I only care about the 1-d case at this point. Thanks, Ken

2 1

NumPy-Discussion July 2010