Mailman 3 May 2015 - NumPy-Discussion

addition to numpy.i
by Tom Krauss 27 May '15

27 May '15

Hi folks, After some discussion with Bill Spotz I decided to try to submit my new typemap to numpy.i that allows in-place arrays of an arbitrary number of dimensions to be passed in as a "flat" array with a single "size". To that end I created my first pull request https://github.com/numpy/numpy/pull/5914 sorry if I missed any steps or procedures - I noticed only after I did the commit and pull request that I should have created a new feature branch, sorry about that. Anyway I noticed the pull request initiated a series of tests and one of them failed. How do I go about debugging and resolving the failure? Thanks for your help, Tom Krauss

2 2

Strategy for OpenBLAS
by Matthew Brett 27 May '15

27 May '15

Hi, This morning I was wondering whether we ought to plan to devote some resources to collaborating with the OpenBLAS team. Summary: we should explore ways of setting up numpy as a test engine for OpenBLAS development. Detail: I am getting the impression that OpenBLAS is looking like the most likely medium term solution for open-source stack builds of numpy and scipy on Linux and Windows at least. ATLAS has been our choice for this up until now, but it is designed for optimizing to a particular CPU configuration, which will likely make it slow on some or even most of the machines a general installer gets installed on. This is only likely to get more severe over time, because current ATLAS development is on multi-core optimization, where the number of cores may need to be set at compile time. The worry about OpenBLAS has always been that it is hard to maintain, and fixes don't always have tests. There might be other alternatives that are a better bet technically, but don't currently have OpenBLAS' dynamic selection features or CPU support. It is relatively easy to add tests using Python / numpy. We like tests. Why don't we propose a collaboration with OpenBLAS where we build and test numpy with every / most / some commits of OpenBLAS, and try to make it easy for the OpenBLAS team to add tests. Maybe we can use and add to the list of machines on which OpenBLAS is tested [1]? We Berkeley Pythonistas can certainly add the machines at our buildbot farm [2]. Maybe the Julia / R developers would be interested to help too? Cheers, Matthew [1] https://github.com/xianyi/OpenBLAS/wiki/Machine-List [2] http://nipy.bic.berkeley.edu/buildslaves

5 8

MPI: Sendrecv blocks
by Florian Lindner 27 May '15

27 May '15

Hello, I have this piece of code: comm = MPI.COMM_WORLD temp = np.zeros(blockSize*blockSize) PrintNB("Communicate A to", get_left_rank()) comm.Sendrecv(sendbuf=np.ascontiguousarray(lA), dest=get_left_rank(), recvbuf=temp) lA = np.reshape(temp, (blockSize, blockSize)) PrintNB("Finished sending") lA being a numpy array. Output is: [0] Communicate A to 1 [2] Communicate A to 3 [3] Communicate A to 2 [1] Communicate A to 0 [1] Finished sending # here it blocks [n] is the rank. I have a circular send. 0>1, 1>0 and 2>3, 3>2. I understood Sendrec so that it is made specifically for these cases, but still it blocks. What is the problem here? Thanks! Florian

1 0

Chaining apply_over_axis for multiple axes.
by Andrew Nelson 25 May '15

25 May '15

I have a function that operates over a 1D array, to return an array of a similar size. To use it in a 2D fashion I would have to do something like the following: for row in range(np.size(arr, 0): arr_out[row] = func(arr[row]) for col in range(np.size(arr, 1): arr_out[:, col] = func(arr[:, col]) I would like to generalise this to N dimensions. Does anyone have any suggestions of how to achieve this? Presumably what I need to do is build an iterator, and then remove an axis: # arr.shape=(2, 3, 4) it = np.nditer(arr, flags=['multi_index']) it.remove_axis(2) while not it.finished: arr_out[it.multi_index] = func(arr[it.multi_index]) it.iternext() If I have an array with shape (2, 3, 4) this would allow me to iterate over the 6 1D arrays that are 4 elements long. However, how do I then construct the iterator for the preceding axes?

3 2

ANN: Scipy 0.16.0 beta release 2
by Ralf Gommers 24 May '15

24 May '15

Hi all, The second beta for Scipy 0.16.0 is now available. After beta 1 a couple of critical issues on Windows were solved, and there are now also 32-bit Windows binaries (along with the sources and release notes) available on https://sourceforge.net/projects/scipy/files/scipy/0.16.0b2/. Please try this release and report any issues on the scipy-dev mailing list. Cheers, Ralf

1 0

Two questions about PEP 465 dot product
by Alexander Belopolsky 22 May '15

22 May '15

1. Is there a simple expression using existing numpy functions that implements PEP 465 semantics for @? 2. Suppose I have a function that takes two vectors x and y, and a matrix M and returns x.dot(M.dot(y)). I would like to "vectorize" this function so that it works with x and y of any ndim >= 1 and M of any ndim >= 2 treating multi-dimensional x and y as arrays of vectors and M as an array of matrices (broadcasting as necessary). The result should be an array of xMy products. How would I achieve that using PEP 465's @?

4 12

np.diag(np.dot(A, B))
by Mathieu Blondel 22 May '15

22 May '15

Hi, I often need to compute the equivalent of np.diag(np.dot(A, B)). Computing np.dot(A, B) is highly inefficient if you only need the diagonal entries. Two more efficient ways of computing the same thing are np.sum(A * B.T, axis=1) and np.einsum("ij,ji->i", A, B). The first can allocate quite a lot of temporary memory. The second can be quite cryptic for someone not familiar with einsum. I assume that einsum does not compute np.dot(A, B), but I haven't verified. Since this is is quite a recurrent pattern, I was wondering if it would be worth adding a dedicated function to NumPy and SciPy's sparse module. A possible name would be "diagdot". The best performance would be obtained when A is C-style and B fortran-style. Best, Mathieu

3 3

Re: [Numpy-discussion] np.diag(np.dot(A, B))
by Nadav Horesh 22 May '15

22 May '15

There was an idea on this list to provide a function the run multiple dot on several vectors/matrices. It seems to be a particular implementation of this proposed function. Nadav. On 22 May 2015 11:58, David Cournapeau <cournape(a)gmail.com> wrote: On Fri, May 22, 2015 at 5:39 PM, Mathieu Blondel <mathieu(a)mblondel.org<mailto:mathieu@mblondel.org>> wrote: Hi, I often need to compute the equivalent of np.diag(np.dot(A, B)). Computing np.dot(A, B) is highly inefficient if you only need the diagonal entries. Two more efficient ways of computing the same thing are np.sum(A * B.T, axis=1) and np.einsum("ij,ji->i", A, B). The first can allocate quite a lot of temporary memory. The second can be quite cryptic for someone not familiar with einsum. I assume that einsum does not compute np.dot(A, B), but I haven't verified. Since this is is quite a recurrent pattern, I was wondering if it would be worth adding a dedicated function to NumPy and SciPy's sparse module. A possible name would be "diagdot". The best performance would be obtained when A is C-style and B fortran-style. Does your implementation use BLAS, or is just a a wrapper around einsum ? David

1 0

binary wheels for numpy?
by Chris Barker 19 May '15

19 May '15

Hi folks., I did a little "intro to scipy" session as part of a larger Python class the other day, and was dismayed to find that "pip install numpy" still dosn't work on Windows. Thanks mostly to Matthew Brett's work, the whole scipy stack is pip-installable on OS-X, it would be really nice if we had that for Windows. And no, saying "you should go get Python(x,y) or Anaconda, or Canopy, or...) is really not a good solution. That is indeed the way to go if someone is primarily focusing on computational programming, but if you have a web developer, or someone new to Python for general use, they really should be able to just grab numpy and play around with it a bit without having to start all over again. My solution was to point folks to Chris Gohlke's site -- which is a Fabulous resource -- THANK YOU CHRISTOPH! But I still think that we should have the basic scipy stack on PyPi as Windows Wheels... IIRC, the last run through on this discussion got stuck on the "what hardware should it support" -- wheels do not allow a selection at install time, so we'd have to decide what instruction set to support, and just stick with that. Which would mean that: some folks would get a numpy/scipy that would run a bit slower than it might and some folks would get one that wouldn't run at all on their machine. But I don't see any reason that we can't find a compromise here -- do a build that supports most machines, and be done with it. Even now, people have to go get (one way or another) a MKL-based build to get optimum performance anyway -- so if we pick an instruction set support by, say (an arbitrary, and impossible to determine) 95% of machines out there -- we're good to go. I take it there are licensing issues that prevent us from putting Chris' Binaries up on PyPi? But are there technical issues I'm forgetting here, or do we just need to come to a consensus as to hardware version to support and do it? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

11 26

MaskedArray compatibility decorators
by J.Goutin 18 May '15

18 May '15

Hello, I created 2 decorators for improve compatibility of "numpy.ma.MaskedArray" with functions that don't support them. This simply convert MaskedArray to classical ndarray with masked value converted to NaN, use the function, and reconvert to MaskedArray. @MaArrayToNaNKeepMask : Re-use the source mask on the output. @MaArrayToNaNFixInvalid : Replace invalid values by mask on the output. Source: import numpy as np def MaArrayToNaNKeepMask(func): """ MaArray to ndArray with nan decorator. Keep mask from originale ndArray. """ def wrapper(MaArray, *args, **kwargs): try: Mask = MaArray.mask fill = MaArray.fill_value return np.ma.masked_array(func(MaArray.filled(np.NaN),*args, **kwargs), mask=Mask, fill_value=fill) except: return func(MaArray, *args, **kwargs) return wrapper def MaArrayToNaNFixInvalid(func): """ MaArray to ndArray with nan decorator. Recreate mask from invalid points. """ def wrapper(MaArray, *args, **kwargs): try: fill = MaArray.fill_value return np.ma.fix_invalid(func(MaArray.filled(np.NaN), *args, **kwargs), fill_value=fill) except: return func(MaArray, *args, **kwargs) return wrapper Exemple: import skimage.transform @MaArrayToNaNFixInvalid def maresize(image, output_shape, order=1, mode='constant', cval=0, clip=True, preserve_range=True): return skimage.transform.resize(image, output_shape, order, mode, cval, clip, preserve_range) I think it may be usefull to include this directly on numpy. There is a lot of functions that don't work directly with MaskedArray.

1 0

NumPy-Discussion May 2015