Mailman 3 May 2017 - NumPy-Discussion

failed to add routine to the core module
by marc 18 May '17

18 May '17

Dear Numpy developers, I'm trying to add a routine to calculate the sum of a product of two arrays (a dot product). But that would not increase the memory (from what I saw np.dot is increasing the memory while it should not be necessary). The idea is to avoid the use of the temporary array in the calculation of the variance ( numpy/numpy/core/_methods.py line 112). The routine that I want to implement look like this in python, |arr = np.random.rand(100000)| |mean = arr.mean()| |var = 0.0| |for ai in arr: var += (ai-mean)**2| I would like to implement it in the umath module. As a first step, I tried to reproduce the divmod function of umath, but I did not manage to do it, you can find my fork here <https://github.com/mbarbry/numpy/tree/looking_around> (the branch with the changes is call looking_around). During compilation I get the following error, |gcc: numpy/core/src/multiarray/number.c In file included from numpy/core/src/multiarray/number.c:17:0: numpy/core/src/multiarray/number.c: In function ‘array_sum_multiply’: numpy/core/src/private/binop_override.h:176:39: error: ‘PyNumberMethods {aka struct <anonymous>}’ has no member named ‘nb_sum_multiply’ (void*)(Py_TYPE(m2)->tp_as_number->SLOT_NAME) != (void*)(test_func)) ^ numpy/core/src/private/binop_override.h:180:13: note: in expansion of macro ‘BINOP_IS_FORWARD’ if (BINOP_IS_FORWARD(m1, m2, slot_expr, test_func) && \ ^ numpy/core/src/multiarray/number.c:363:5: note: in expansion of macro ‘BINOP_GIVE_UP_IF_NEEDED’ BINOP_GIVE_UP_IF_NEEDED(m1, m2, nb_sum_multiply, array_sum_multiply);| Sorry if my question seems basic, but I'm new in Numpy development. Any help? Thank you in advance, Marc Barbry PS: I opened an issues as well on the github repository https://github.com/numpy/numpy/issues/9130

3 3

Compare NumPy arrays with threshold and return the differences
by Nissim Derdiger 18 May '17

18 May '17

Hi, In my script, I need to compare big NumPy arrays (2D or 3D), and return a list of all cells with difference bigger than a defined threshold. The compare itself can be done easily done with "allclose" function, like that: Threshold = 0.1 if (np.allclose(Arr1, Arr2, Threshold, equal_nan=True)): Print('Same') But this compare does not return which cells are not the same. The easiest (yet naive) way to know which cells are not the same is to use a simple for loops code like this one: def CheckWhichCellsAreNotEqualInArrays(Arr1,Arr2,Threshold): if not Arr1.shape == Arr2.shape: return ['Arrays size not the same'] Dimensions = Arr1.shape Diff = [] for i in range(Dimensions [0]): for j in range(Dimensions [1]): if not np.allclose(Arr1[i][j], Arr2[i][j], Threshold, equal_nan=True): Diff.append(',' + str(i) + ',' + str(j) + ',' + str(Arr1[i,j]) + ',' + str(Arr2[i,j]) + ',' + str(Threshold) + ',Fail\n') return Diff (and same for 3D arrays - with 1 more for loop) This way is very slow when the Arrays are big and full of none-equal cells. Is there a fast straight forward way in case they are not the same - to get a list of the uneven cells? maybe some built-in function in the NumPy itself? Thanks! Nissim

4 3

Proposal: np.search() to complement np.searchsorted()
by Martin Spacek 16 May '17

16 May '17

Hello, I've opened up a pull request to add a function called np.search(), or something like it, to complement np.searchsorted(): https://github.com/numpy/numpy/pull/9055 There's also this issue I opened before starting the PR: https://github.com/numpy/numpy/issues/9052 Proposed API changes require discussion on the list, so here I am! This proposed function (and perhaps array method?) does the same as np.searchsorted(a, v), but doesn't require `a` to be sorted, and explicitly checks if all the values in `v` are a subset of those in `a`. If not, it currently raises an error, but that could be controlled via a kwarg. As I mentioned in the PR, I often find myself abusing np.searchsorted() by not explicitly checking these assumptions. The temptation to use it is great, because it's such a fast and convenient function, and most of the time that I use it, the assumptions are indeed valid. Explicitly checking those assumptions each and every time before I use np.searchsorted() is tedious, and easy to forget to do. I wouldn't be surprised if many others abuse np.searchsorted() in the same way. Looking at my own habits and uses, it seems to me that finding the indices of matching values of one array in another is a more common use case than finding insertion indices of one array into another sorted array. So, I propose that np.search(), or something like it, could be even more useful than np.searchsorted(). Thoughts? Martin

5 6

Re: [Numpy-discussion] Proposal: np.search() to complement np.searchsorted()
by Peter Creasey 16 May '17

16 May '17

> From: Stephan Hoyer <shoyer(a)gmail.com> > > I like the idea of a strategy keyword argument. strategy='auto' leaves the > door open for future improvements, e.g., if we ever add hash tables to > numpy. > > For the algorithm, I think we actually want to sort the needles array as > well in most (all?) cases. > > If haystack is also sorted, advancing thorough both arrays at once brings > down the cost of the actual search itself down to O(n+k). (Possibly this is > worth exposing as np.searchbothsorted or something similar?) I actually suggest reducing the scope of the problem to >>> search_unique(haystack, needles) which finds the index* in a haystack (of unique values) of each needle and can make the assumption (e.g. as Stephen Hoyer points out) that if len(needles)<<len(haystack), then you can just sort the needles and iterate through the haystack in O(H log(N)) steps (otherwise you sort the haystack and iterate through the needles instead). Whilst you could make this function work in the more general non-unique case (i.e. using stable sorts and finding the first occurrence) I notice this often an accident and users would be better served with an exception. I also prefer raising an exception if a needle is unfound (perhaps with an unfound_index=None kwarg). Finally I am a little wary of strategy=‘hashtable’ (though I love hash-tables!) since choosing the one hash that is great for everyone’s data does not seem an easy problem. Peter * ‘index_’ sounds more attractive than ‘search_’, but we’re stuck with the python connotation it would be the first occurrence and the numpy connotation of that you are giving rather than receiving indices.

1 0

How to concatenate 2 recarrays into a single recarray
by Isaac Gerg 13 May '17

13 May '17

I have 2 arrays, a and b which are rec arrays of length 10. Each array has 5 columns. I would like to combine all the columns into a single recarray with 10 columns and length 10. Thanks, Isaac

5 12

NumPy v1.13.0rc1 released.
by Charles R Harris 12 May '17

12 May '17

Hi All, I'm please to announce the NumPy 1.13.0rc1 release. This release supports Python 2.7 and 3.4-3.6 and contains many new features. It is one of the most ambitious releases in the last several years. Some of the highlights and new functions are *Highlights* - Operations like ``a + b + c`` will reuse temporaries on some platforms, resulting in less memory use and faster execution. - Inplace operations check if inputs overlap outputs and create temporaries to avoid problems. - New __array_ufunc__ attribute provides improved ability for classes to override default ufunc behavior. - New np.block function for creating blocked arrays. *New functions* - New ``np.positive`` ufunc. - New ``np.divmod`` ufunc provides more efficient divmod. - New ``np.isnat`` ufunc tests for NaT special values. - New ``np.heaviside`` ufunc computes the Heaviside function. - New ``np.isin`` function, improves on ``in1d``. - New ``np.block`` function for creating blocked arrays. - New ``PyArray_MapIterArrayCopyIfOverlap`` added to NumPy C-API. Wheels for the pre-release are available on PyPI. Source tarballs, zipfiles, release notes, and the Changelog are available on github <https://github.com/numpy/numpy/releases/tag/v1.13.0rc1>. A total of 100 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - A. Jesse Jiryu Davis + - Alessandro Pietro Bardelli + - Alex Rothberg + - Alexander Shadchin - Allan Haldane - Andres Guzman-Ballen + - Antoine Pitrou - Antony Lee - B R S Recht + - Baurzhan Muftakhidinov + - Ben Rowland - Benda Xu + - Blake Griffith - Bradley Wogsland + - Brandon Carter + - CJ Carey - Charles Harris - Danny Hermes + - Duke Vijitbenjaronk + - Egor Klenin + - Elliott Forney + - Elliott M Forney + - Endolith - Eric Wieser - Erik M. Bray - Eugene + - Evan Limanto + - Felix Berkenkamp + - François Bissey + - Frederic Bastien - Greg Young - Gregory R. Lee - Importance of Being Ernest + - Jaime Fernandez - Jakub Wilk + - James Cowgill + - James Sanders - Jean Utke + - Jesse Thoren + - Jim Crist + - Joerg Behrmann + - John Kirkham - Jonathan Helmus - Jonathan L Long - Jonathan Tammo Siebert + - Joseph Fox-Rabinovitz - Joshua Loyal + - Juan Nunez-Iglesias + - Julian Taylor - Kirill Balunov + - Likhith Chitneni + - Loïc Estève - Mads Ohm Larsen - Marein Könings + - Marten van Kerkwijk - Martin Thoma - Martino Sorbaro + - Marvin Schmidt + - Matthew Brett - Matthias Bussonnier + - Matthias C. M. Troffaes + - Matti Picus - Michael Seifert - Mikhail Pak + - Mortada Mehyar - Nathaniel J. Smith - Nick Papior - Oscar Villellas + - Pauli Virtanen - Pavel Potocek - Pete Peeradej Tanruangporn + - Philipp A + - Ralf Gommers - Robert Kern - Roland Kaufmann + - Ronan Lamy - Sami Salonen + - Sanchez Gonzalez Alvaro - Sebastian Berg - Shota Kawabuchi - Simon Gibbons - Stefan Otte - Stefan Peterson + - Stephan Hoyer - Søren Fuglede Jørgensen + - Takuya Akiba - Tom Boyd + - Ville Skyttä + - Warren Weckesser - Wendell Smith - Yu Feng - Zixu Zhao + - Zè Vinícius + - aha66 + - davidjn + - drabach + - drlvk + - jsh9 + - solarjoe + - zengi + Cheers, Chuck

11 10

Numpy 1.13.x branched
by Charles R Harris 11 May '17

11 May '17

Hi All, Just a quck note that 1.13.x has been branched and master is open for 1.14.0 development. Chuck

1 0

Ready to branch 1.13
by Charles R Harris 09 May '17

09 May '17

Hi All, I am getting ready to branch 1.13 after updating the release notes and .mailmap. If there is something that you think is essential to get into 1.13 that is not there, comment. Chuck

1 0