Mailman 3 April 2013 - NumPy-Discussion

dynamically choosing atlas libraries
by Edward Walter 07 Apr '13

07 Apr '13

Hello List, We're standing up a new computational cluster and going through the steps of building Numpy etc. We would like to be able to install multiple versions of ATLAS (with different build settings / tunings / etc) and have Numpy load the shared Atlas libraries dynamically based on the LD_LIBRARY_PATH. It's not clear to me how to do this given that the library path is hard coded in Numpy's site.cfg. It seems like other people have gotten this working though (i.e. RHEL/Centos with their collection of Atlas versions {atlas, atlas-sse3, etc}). Does anyone have any pointers on this? Thanks much. -Ed Walter Carnegie Mellon University

2 1

Fwd: Pull request #3188 - Void scalar pickling behavior.
by Alex Ford 06 Apr '13

06 Apr '13

Hello, Do any core developers or uses have guidance on how to resolve PR #3188 ( https://github.com/numpy/numpy/pull/3188) in relation to the pickling behavior of array scalar objects? To summarize, pickling array scalars with object fields, which are produced when indexing record arrays with object fields, stores the object address instead of pickling the referenced object. This behavior, obviously, results in invalid references on unpickling. Two current options are to: A) Raise an exception on pickling scalars with object fields. B) Transparently convert scalars to zero-rank arrays on pickling, which pickle properly. Unless there are objections or opinions on potential solutions, I am inclined to implement A. Thanks, Alex Ford

2 1

Bug in np.records?
by Pierre Barbier de Reuille 06 Apr '13

06 Apr '13

Hey, I am trying to use titles for the record arrays. In the documentation, it is specified that any column can set to "None". However, trying this fails on numpy 1.6.2 because in np.core.records, on line 195, the "strip" method is called on the title object. This is really annoying. Could we fix this by replacing line 195 with: self._titles = [n.strip() if n is not None else None for n in titles[:self._nfields]] ? Thank you, -- Barbier de Reuille Pierre

2 1

Raveling, reshape order keyword unnecessarily confuses index and memory ordering
by Matthew Brett 06 Apr '13

06 Apr '13

Hi, We were teaching today, and found ourselves getting very confused about ravel and shape in numpy. Summary -------------- There are two separate ideas needed to understand ordering in ravel and reshape: Idea 1): ravel / reshape can proceed from the last axis to the first, or the first to the last. This is "ravel index ordering" Idea 2) The physical layout of the array (on disk or in memory) can be "C" or "F" contiguous or neither. This is "memory ordering" The index ordering is usually (but see below) orthogonal to the memory ordering. The 'ravel' and 'reshape' commands use "C" and "F" in the sense of index ordering, and this mixes the two ideas and is confusing. What the current situation looks like ---------------------------------------------------- Specifically, we've been rolling this around 4 experienced numpy users and we all predicted at least one of the results below wrongly. This was what we knew, or should have known: In [2]: import numpy as np In [3]: arr = np.arange(10).reshape((2, 5)) In [5]: arr.ravel() Out[5]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) So, the 'ravel' operation unravels over the last axis (1) first, followed by axis 0. So far so good (even if the opposite to MATLAB, Octave). Then we found the 'order' flag to ravel: In [10]: arr.flags Out[10]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [11]: arr.ravel('C') Out[11]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) But we soon got confused. How about this? In [12]: arr_F = np.array(arr, order='F') In [13]: arr_F.flags Out[13]: C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [16]: arr_F Out[16]: array([[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]]) In [17]: arr_F.ravel('C') Out[17]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) Right - so the flag 'C' to ravel, has got nothing to do with *memory* ordering, but is to do with *index* ordering. And in fact, we can ask for memory ordering specifically: In [22]: arr.ravel('K') Out[22]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) In [23]: arr_F.ravel('K') Out[23]: array([0, 5, 1, 6, 2, 7, 3, 8, 4, 9]) In [24]: arr.ravel('A') Out[24]: array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]) In [25]: arr_F.ravel('A') Out[25]: array([0, 5, 1, 6, 2, 7, 3, 8, 4, 9]) There are some confusions to get into with the 'order' flag to reshape as well, of the same type. Ravel and reshape use the tems 'C' and 'F" in the sense of index ordering. This is very confusing. We think the index ordering and memory ordering ideas need to be separated, and specifically, we should avoid using "C" and "F" to refer to index ordering. Proposal ------------- * Deprecate the use of "C" and "F" meaning backwards and forwards index ordering for ravel, reshape * Prefer "Z" and "N", being graphical representations of unraveling in 2 dimensions, axis1 first and axis0 first respectively (excellent naming idea by Paul Ivanov) What do y'all think? Cheers, Matthew Paul Ivanov JB Poline

9 79

Re: [Numpy-discussion] Indexing bug
by Ivan Oseledets 05 Apr '13

05 Apr '13

Message: 2 Date: Sat, 30 Mar 2013 11:13:35 -0700 From: Jaime Fern?ndez del R?o <jaime.frio(a)gmail.com> Subject: Re: [Numpy-discussion] Indexing bug? To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> Message-ID: <CAPOWHWk+mL6KN6F2FHTPn5HTiU0UEQPj6KdxjNK_+T1E-YRiBg(a)mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1" On Sat, Mar 30, 2013 at 11:01 AM, Ivan Oseledets <ivan.oseledets(a)gmail.com>wrote: > I am using numpy 1.6.1, > and encountered a wierd fancy indexing bug: > > import numpy as np > c = np.random.randn(10,200,10); > > In [29]: print c[[0,1],:200,:2].shape > (2, 200, 2) > > In [30]: print c[[0,1],:200,[0,1]].shape > (2, 200) > > It means, that here fancy indexing is not working right for a 3d array. > On Sat, Mar 30, 2013 at 11:01 AM, Ivan Oseledets <ivan.oseledets(a)gmail.com>wrote: > I am using numpy 1.6.1, > and encountered a wierd fancy indexing bug: > > import numpy as np > c = np.random.randn(10,200,10); > > In [29]: print c[[0,1],:200,:2].shape > (2, 200, 2) > > In [30]: print c[[0,1],:200,[0,1]].shape > (2, 200) > > It means, that here fancy indexing is not working right for a 3d array. > --> It is working fine, review the docs: http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html#advanced-ind… In your return, item [0, :] is c[0, :, 0] and item[1, :]is c[1, :, 1]. If you want a return of shape (2, 200, 2) where item [i, :, j] is c[i, :, j] you could use slicing: c[:2, :200, :2] or something more elaborate like: c[np.arange(2)[:, None, None], np.arange(200)[:, None], np.arange(2)] Jaime ---> Oh! So it is not a bug, it is a feature, which is completely incompatible with other array based languages (MATLAB and Fortran). To me, I can not find a single explanation why it is so in numpy. Taking submatrices from a matrix is a common operation and the syntax above is very natural to take submatrices, not a weird diagonal stuff. i.e., c = np.random.randn(100,100) d = c[[0,3],[2,3]] should NOT produce two numbers! (and you can not do it using slices!) In MATLAB and Fortran c(indi,indj) will produce a 2 x 2 matrix. How it can be done in numpy (and why the complications?) So, please consider this message as a feature request. Ivan

4 3

einsum and broadcasting
by Jaakko Luttinen 05 Apr '13

05 Apr '13

I don't quite understand how einsum handles broadcasting. I get the following error, but I don't understand why: In [8]: import numpy as np In [9]: A = np.arange(12).reshape((4,3)) In [10]: B = np.arange(6).reshape((3,2)) In [11]: np.einsum('ik,k...->i...', A, B) --------------------------------------------------------------------------- ValueError: operand 0 did not have enough dimensions to match the broadcasting, and couldn't be extended because einstein sum subscripts were specified at both the start and end However, if I use explicit indexing, it works: In [12]: np.einsum('ik,kj->ij', A, B) Out[12]: array([[10, 13], [28, 40], [46, 67], [64, 94]]) It seems that it also works if I add '...' to the first operand: In [12]: np.einsum('ik...,k...->i...', A, B) Out[12]: array([[10, 13], [28, 40], [46, 67], [64, 94]]) However, as far as I understand, the syntax np.einsum('ik,k...->i...', A, B) should work. Have I misunderstood something or is there a bug? Thanks for your help! Jaakko

2 1

try to solve issue #2649 and revisit #473
by huangkandiy＠gmail.com 04 Apr '13

04 Apr '13

Hello, all I try to solve issue 2649 which is related to 473 on multiplication of a matrix and an array. As 2649 shows import numpy as np x = np.arange(5) I = np.asmatrix(np.identity(5)) print np.dot(I, x).shape # -> (1, 5) First of all I assume we expect that I.dot(x) and I * x behave the same, so I suggest add function dot to matrix, like def dot(self, other): return self * other Then the major issue is the constructor of array and matrix interpret a list differently. array([0,1]).shape = (2,) and matrix([0,1]).shape = (1, 2). It will throw error when run np.dot(I, x), because in __mul__, x will be converted to a 1*5 matrix first. It's not consistent with np.dot(np.identity(5), x), which returns x. To fix that, I suggest to check the dimension of array when convert it to matrix. If it's 1D array, then convert it to a vertical vector explicitly like this if isinstance(data, N.ndarray): + if len(data.shape) == 1: + data = data.reshape(data.shape[0], 1) if dtype is None: intype = data.dtype else: Any comments? -- Kan Huang Department of Applied math & Statistics Stony Brook University 917-767-8018

6 8

Moving linalg c code
by Charles R Harris 04 Apr '13

04 Apr '13

Hi All, There is a PR <https://github.com/numpy/numpy/pull/2954> that adds some blas and lapack functions to numpy. I'm thinking that if that PR is merged it would be good to move all of the blas and lapack functions, including the current ones in numpy/linalg into a single directory somewhere in numpy/core/src. So there are two questions here: should we be adding the new functions, and if so, should we consolidate all the blas and lapack C code into its own directory somewhere in numpy/core/src. Thoughts? Chuck

3 2

Sphinx question
by Charles R Harris 03 Apr '13

03 Apr '13

Making the numpy documentation generates hundreds of warnings like ...doc/source/reference/generated/numpy.memmap.argsort.rst:: WARNING: document isn't included in any toctree Note that argsort is a method inherited from ndarray. These warnings are repeated for all ndarray methods in all classes that subclass ndarray. Does anyone know the proper way to deal with this? Chuck

1 0

NumPy/SciPy participation in GSoC 2013
by Ralf Gommers 02 Apr '13

02 Apr '13

Hi all, It is the time of the year for Google Summer of Code applications. If we want to participate with Numpy and/or Scipy, we need two things: enough mentors and ideas for projects. If we get those, we'll apply under the PSF umbrella. They've outlined the timeline they're working by and guidelines at http://pyfound.blogspot.nl/2013/03/get-ready-for-google-summer-of-code.html. We should be able to come up with some interesting project ideas I'd think, let's put those at http://projects.scipy.org/scipy/wiki/SummerofCodeIdeas. Preferably with enough detail to be understandable for people new to the projects and a proposed mentor. We need at least 3 people willing to mentor a student. Ideally we'd have enough mentors this week, so we can apply to the PSF on time. If you're willing to be a mentor, please send me the following: name, email address, phone nr, and what you're interested in mentoring. If you have time constaints and have doubts about being able to be a primary mentor, being a backup mentor would also be helpful. Cheers, Ralf P.S. as you can probably tell from the above, I'm happy to coordinate the GSoC applications for Numpy and Scipy

4 7