Mailman 3 March 2013 - NumPy-Discussion

ANN: NumPy 1.7.1rc1 release
by Ondřej Čertík 07 Apr '13

07 Apr '13

Hi, I'm pleased to announce the availability of the first release candidate of NumPy 1.7.1rc1. Sources and binary installers can be found at https://sourceforge.net/projects/numpy/files/NumPy/1.7.1rc1/ Please test it and report any bugs. It fixes a few bugs, listed below. I would like to thank everybody who contributed patches to this release: Nathaniel J. Smith, Sebastian Berg, Charles Harris, Bradley M. Froehle, Ralf Gommers, Christoph Gohlke, Mark Wiebe and Maximilian Albert. Cheers, Ondrej ========================= NumPy 1.7.1 Release Notes ========================= This is a bugfix only release in the 1.7.x series. Issues fixed ------------ gh-2973 Fix `1` is printed during numpy.test() gh-2983 BUG: gh-2969: Backport memory leak fix 80b3a34. gh-3007 Backport gh-3006 gh-2984 Backport fix complex polynomial fit gh-2982 BUG: Make nansum work with booleans. gh-2985 Backport large sort fixes gh-3039 Backport object take gh-3105 Backport nditer fix op axes initialization gh-3108 BUG: npy-pkg-config ini files were missing after Bento build. gh-3124 BUG: PyArray_LexSort allocates too much temporary memory. gh-3131 BUG: Exported f2py_size symbol prevents linking multiple f2py modules. gh-3117 Backport gh-2992 gh-3135 DOC: Add mention of PyArray_SetBaseObject stealing a reference gh-3134 DOC: Fix typo in fft docs (the indexing variable is 'm', not 'n'). gh-3136 Backport #3128 Checksums ========= 28c3f3e71b5eaa6bfab6e8340dbd35e7 release/installers/numpy-1.7.1rc1.tar.gz 436f416dee10d157314bd9da7ab95c9c release/installers/numpy-1.7.1rc1-win32-superpack-python2.7.exe a543c8cf69f66ff2b4c9565646105863 release/installers/numpy-1.7.1rc1-win32-superpack-python2.6.exe 6dfcbbd449b7fe4e841c5fd1bfa7af7c release/installers/numpy-1.7.1rc1-win32-superpack-python2.5.exe 22912792a1b6155ae2bdbc30bee8fadc release/installers/numpy-1.7.1rc1-win32-superpack-python3.2.exe 95bc5a5fcce9fcbc2717a774dccae31b release/installers/numpy-1.7.1rc1-win32-superpack-python3.3.exe 33cf283765a148846b49b89fb96d67d5 release/installers/numpy-1.7.1rc1-win32-superpack-python3.1.exe 9761de4b35493fed38c5d177da9c3b37 release/installers/numpy-1.7.1rc1.zip

4 5

Bug in np.records?
by Pierre Barbier de Reuille 06 Apr '13

06 Apr '13

Hey, I am trying to use titles for the record arrays. In the documentation, it is specified that any column can set to "None". However, trying this fails on numpy 1.6.2 because in np.core.records, on line 195, the "strip" method is called on the title object. This is really annoying. Could we fix this by replacing line 195 with: self._titles = [n.strip() if n is not None else None for n in titles[:self._nfields]] ? Thank you, -- Barbier de Reuille Pierre

2 1

Raveling, reshape order keyword unnecessarily confuses index and memory ordering
by Matthew Brett 05 Apr '13

05 Apr '13

Hi, We were teaching today, and found ourselves getting very confused about ravel and shape in numpy. Summary -------------- There are two separate ideas needed to understand ordering in ravel and reshape: Idea 1): ravel / reshape can proceed from the last axis to the first, or the first to the last. This is "ravel index ordering" Idea 2) The physical layout of the array (on disk or in memory) can be "C" or "F" contiguous or neither. This is "memory ordering" The index ordering is usually (but see below) orthogonal to the memory ordering. The 'ravel' and 'reshape' commands use "C" and "F" in the sense of index ordering, and this mixes the two ideas and is confusing. What the current situation looks like ---------------------------------------------------- Specifically, we've been rolling this around 4 experienced numpy users and we all predicted at least one of the results below wrongly. This was what we knew, or should have known: In [2]: import numpy as np In [3]: arr = np.arange(10).reshape((2, 5)) In [5]: arr.ravel() Out[5]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) So, the 'ravel' operation unravels over the last axis (1) first, followed by axis 0. So far so good (even if the opposite to MATLAB, Octave). Then we found the 'order' flag to ravel: In [10]: arr.flags Out[10]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [11]: arr.ravel('C') Out[11]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) But we soon got confused. How about this? In [12]: arr_F = np.array(arr, order='F') In [13]: arr_F.flags Out[13]: C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [16]: arr_F Out[16]: array([[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]]) In [17]: arr_F.ravel('C') Out[17]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) Right - so the flag 'C' to ravel, has got nothing to do with *memory* ordering, but is to do with *index* ordering. And in fact, we can ask for memory ordering specifically: In [22]: arr.ravel('K') Out[22]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) In [23]: arr_F.ravel('K') Out[23]: array([0, 5, 1, 6, 2, 7, 3, 8, 4, 9]) In [24]: arr.ravel('A') Out[24]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) In [25]: arr_F.ravel('A') Out[25]: array([0, 5, 1, 6, 2, 7, 3, 8, 4, 9]) There are some confusions to get into with the 'order' flag to reshape as well, of the same type. Ravel and reshape use the tems 'C' and 'F" in the sense of index ordering. This is very confusing. We think the index ordering and memory ordering ideas need to be separated, and specifically, we should avoid using "C" and "F" to refer to index ordering. Proposal ------------- * Deprecate the use of "C" and "F" meaning backwards and forwards index ordering for ravel, reshape * Prefer "Z" and "N", being graphical representations of unraveling in 2 dimensions, axis1 first and axis0 first respectively (excellent naming idea by Paul Ivanov) What do y'all think? Cheers, Matthew Paul Ivanov JB Poline

9 79

Re: [Numpy-discussion] Indexing bug
by Ivan Oseledets 05 Apr '13

05 Apr '13

Message: 2 Date: Sat, 30 Mar 2013 11:13:35 -0700 From: Jaime Fern?ndez del R?o <jaime.frio(a)gmail.com> Subject: Re: [Numpy-discussion] Indexing bug? To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> Message-ID: <CAPOWHWk+mL6KN6F2FHTPn5HTiU0UEQPj6KdxjNK_+T1E-YRiBg(a)mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1" On Sat, Mar 30, 2013 at 11:01 AM, Ivan Oseledets <ivan.oseledets(a)gmail.com>wrote: > I am using numpy 1.6.1, > and encountered a wierd fancy indexing bug: > > import numpy as np > c = np.random.randn(10,200,10); > > In [29]: print c[[0,1],:200,:2].shape > (2, 200, 2) > > In [30]: print c[[0,1],:200,[0,1]].shape > (2, 200) > > It means, that here fancy indexing is not working right for a 3d array. > On Sat, Mar 30, 2013 at 11:01 AM, Ivan Oseledets <ivan.oseledets(a)gmail.com>wrote: > I am using numpy 1.6.1, > and encountered a wierd fancy indexing bug: > > import numpy as np > c = np.random.randn(10,200,10); > > In [29]: print c[[0,1],:200,:2].shape > (2, 200, 2) > > In [30]: print c[[0,1],:200,[0,1]].shape > (2, 200) > > It means, that here fancy indexing is not working right for a 3d array. > --> It is working fine, review the docs: http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#advanced-ind… In your return, item [0, :] is c[0, :, 0] and item[1, :]is c[1, :, 1]. If you want a return of shape (2, 200, 2) where item [i, :, j] is c[i, :, j] you could use slicing: c[:2, :200, :2] or something more elaborate like: c[np.arange(2)[:, None, None], np.arange(200)[:, None], np.arange(2)] Jaime ---> Oh! So it is not a bug, it is a feature, which is completely incompatible with other array based languages (MATLAB and Fortran). To me, I can not find a single explanation why it is so in numpy. Taking submatrices from a matrix is a common operation and the syntax above is very natural to take submatrices, not a weird diagonal stuff. i.e., c = np.random.randn(100,100) d = c[[0,3],[2,3]] should NOT produce two numbers! (and you can not do it using slices!) In MATLAB and Fortran c(indi,indj) will produce a 2 x 2 matrix. How it can be done in numpy (and why the complications?) So, please consider this message as a feature request. Ivan

4 3

NumPy/SciPy participation in GSoC 2013
by Ralf Gommers 02 Apr '13

02 Apr '13

Hi all, It is the time of the year for Google Summer of Code applications. If we want to participate with Numpy and/or Scipy, we need two things: enough mentors and ideas for projects. If we get those, we'll apply under the PSF umbrella. They've outlined the timeline they're working by and guidelines at http://pyfound.blogspot.nl/2013/03/get-ready-for-google-summer-of-code.html. We should be able to come up with some interesting project ideas I'd think, let's put those at http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas. Preferably with enough detail to be understandable for people new to the projects and a proposed mentor. We need at least 3 people willing to mentor a student. Ideally we'd have enough mentors this week, so we can apply to the PSF on time. If you're willing to be a mentor, please send me the following: name, email address, phone nr, and what you're interested in mentoring. If you have time constaints and have doubts about being able to be a primary mentor, being a backup mentor would also be helpful. Cheers, Ralf P.S. as you can probably tell from the above, I'm happy to coordinate the GSoC applications for Numpy and Scipy

4 7

Indexing bug?
by Ivan Oseledets 30 Mar '13

30 Mar '13

I am using numpy 1.6.1, and encountered a wierd fancy indexing bug: import numpy as np c = np.random.randn(10,200,10); In [29]: print c[[0,1],:200,:2].shape (2, 200, 2) In [30]: print c[[0,1],:200,[0,1]].shape (2, 200) It means, that here fancy indexing is not working right for a 3d array. Is this bug fixed with higher versions of numpy? I do not check, since mine is from EPD and is compiled with MKL (and I can consider recompiling myself only under strong circumstances) Ivan

2 1

Polar/spherical coordinates handling
by Todd 29 Mar '13

29 Mar '13

>From what I can see, numpy doesn't have any functions for handling polar or spherical coordinate to/from cartesian coordinate conversion. I think such methods would be pretty useful. I am looking now and it doesn't look that hard to create functions to convert between n-dimensional cartesian and n-spherical coordinates. Would anyone be interested in me adding methods for this?

2 2

inheriting from recarray with nested dtypes
by Geoffrey Irving 28 Mar '13

28 Mar '13

I have the following two structured dtypes: rotation (quaternion) = dtype([('s','f8'),('v','3f8')]) frame = dtype([('t','3f8'),('r',rotation)]) For various reasons, I usually store rotation arrays in a class Rotations deriving from ndarray, and frames in a class Frames deriving from ndarray. Currently I am defining .s, .v, .t, and .r properties manually, and I'd like to switch to inheriting from recarray. However, the .r property should return an array of class Rotations. I.e., f = Frames(...) # f.dtype = frame r = f.r # r.dtype = rotation, type(r) = Rotations Is there a clean way to tell recarray to adjust the type returned? It already has a bit of intelligence there, since it returns ndarray vs. recarray based on whether the returned dtype has fields. The full code is here in case anyone is curious: https://github.com/otherlab/core/blob/master/vector/Frame.py https://github.com/otherlab/core/blob/master/vector/Rotation.py Thanks, Geoffrey

1 0

Implementing a "find first" style function
by Phil Elson 28 Mar '13

28 Mar '13

The ticket https://github.com/numpy/numpy/issues/2269 discusses the possibility of implementing a "find first" style function which can optimise the process of finding the first value(s) which match a predicate in a given 1D array. For example: >>> a = np.sin(np.linspace(0, np.pi, 200)) >>> print find_first(a, lambda a: a > 0.9) ((71, ), 0.900479032457) This has been discussed in several locations: https://github.com/numpy/numpy/issues/2269 https://github.com/numpy/numpy/issues/2333 http://stackoverflow.com/questions/7632963/numpy-array-how-to-find-index-of… *Rationale* For small arrays there is no real reason to avoid doing: >>> a = np.sin(np.linspace(0, np.pi, 200)) >>> ind = (a > 0.9).nonzero()[0][0] >>> print (ind, ), a[ind] (71,) 0.900479032457 But for larger arrays, this can lead to massive amounts of work even if the result is one of the first to be computed. Example: >>> a = np.arange(1e8) >>> print (a == 5).nonzero()[0][0] 5 So a function which terminates when the first matching value is found is desirable. As mentioned in #2269, it is possible to define a consistent ordering which allows this functionality for >1D arrays, but IMHO it overcomplicates the problem and was not a case that I personally needed, so I've limited the scope to 1D arrays only. *Implementation* My initial assumption was that to get any kind of performance I would need to write the *find* function in C, however after prototyping with some array chunking it became apparent that a trivial python function would be quick enough for my needs. The approach I've implemented in the code found in #2269 simply breaks the array into sub-arrays of maximum length *chunk_size* (2048 by default, though there is no real science to this number), applies the given predicating function, and yields the results from *nonzero()*. The given function should be a python function which operates on the whole of the sub-array element-wise (i.e. the function should be vectorized). Returning a generator also has the benefit of allowing users to get the first *n*matching values/indices. *Results* I timed the implementation of *find* found in my comment at https://github.com/numpy/numpy/issues/2269#issuecomment-14436725 with an obvious test: In [1]: from np_utils import find In [2]: import numpy as np In [3]: import numpy.random In [4]: np.random.seed(1) In [5]: a = np.random.randn(1e8) In [6]: a.min(), a.max() Out[6]: (-6.1194900990552776, 5.9632246301166321) In [7]: next(find(a, lambda a: np.abs(a) > 6)) Out[7]: ((33105441,), -6.1194900990552776) In [8]: (np.abs(a) > 6).nonzero() Out[8]: (array([33105441]),) In [9]: %timeit (np.abs(a) > 6).nonzero() 1 loops, best of 3: 1.51 s per loop In [10]: %timeit next(find(a, lambda a: np.abs(a) > 6)) 1 loops, best of 3: 912 ms per loop In [11]: %timeit next(find(a, lambda a: np.abs(a) > 6, chunk_size=100000)) 1 loops, best of 3: 470 ms per loop In [12]: %timeit next(find(a, lambda a: np.abs(a) > 6, chunk_size=1000000)) 1 loops, best of 3: 483 ms per loop This shows that picking a sensible *chunk_size* can yield massive speed-ups (nonzero is x3 slower in one case). A similar example with a much smaller 1D array shows similar promise: In [41]: a = np.random.randn(1e4) In [42]: %timeit next(find(a, lambda a: np.abs(a) > 3)) 10000 loops, best of 3: 35.8 us per loop In [43]: %timeit (np.abs(a) > 3).nonzero() 10000 loops, best of 3: 148 us per loop As I commented on the issue tracker, if you think this function is worth taking forward, I'd be happy to open up a pull request. Feedback greatfully received. Cheers, Phil

3 5

variables not defined in numpy.random __init.py__ ?
by Dinesh B Vadhia 27 Mar '13

27 Mar '13

Using PyInstaller, the following error occurs: Traceback (most recent call last): File "<string>", line 9, in <module> File "//usr/lib/python2.7/dist-packages/PIL/Image.py", line 355, in init __import__(f, globals(), locals(), []) File "//usr/lib/python2.7/dist-packages/PIL/IptcImagePlugin.py", line 23, in <module> import os, tempfile File "/usr/lib/python2.7/tempfile.py", line 34, in <module> from random import Random as _Random File "//usr/lib/python2.7/dist-packages/numpy/random/__init__.py", line 90, in <module> ranf = random = sample = random_sample NameError: name 'random_sample' is not defined Is line 90 in __init.py__ valid?

3 4