Mailman 3 October 2018 - NumPy-Discussion

Reminder: weekly status meeting
by Matti Picus 27 Oct '18

27 Oct '18

Hi everyone, The team at BIDS meets once a week to discuss progress, priorities, and roadblocks. While our priorities are broadly determined by the project roadmap [0], we would like to provide an opportunity for the community to give more regular and detailed feedback on our work. We therefore invite you to join us for our weekly calls, each **Wednesday from 12:00 to 13:00 Pacific Time**. Detail of the next meeting (2018-10-24) is given in the agenda [1], which is a living document. Feel free to add topics you wish to discuss. We hope to see you there! Best regards, Stéfan, Tyler, Matti [0]https://www.numpy.org/neps/index.html [1]https://hackmd.io/5WZ6VwQKSbSR_4Ng65pUFw?both

5 11

NumPy 1.15.3 release
by Charles R Harris 22 Oct '18

22 Oct '18

Hi All, On behalf of the NumPy team, I am pleased to announce the release of NumPy 1.15.3. This is a bugfix release for bugs and regressions reported following the 1.15.2 release. The most noticeable fix is probably for the memory leak <https://github.com/numpy/numpy/issues/12037> encountered when slicing classes derived from Numpy. The Python versions supported by this release are 2.7, 3.4-3.7. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.15.3/>, source archives are available from Github <https://github.com/numpy/numpy/releases/tag/v1.15.3>. Compatibility Note ================== The NumPy 1.15.x OS X wheels released on PyPI no longer contain 32-bit binaries. That will also be the case in future releases. See `#11625 <https://github.com/numpy/numpy/issues/11625>`__ for the related discussion. Those needing 32-bit support should look elsewhere or build from source. Contributors ============ A total of 7 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Allan Haldane * Charles Harris * Jeroen Demeyer * Kevin Sheppard * Matthew Bowden + * Matti Picus * Tyler Reddy Pull requests merged ==================== A total of 12 pull requests were merged for this release. * `#12080 <https://github.com/numpy/numpy/pull/12080>`__: MAINT: Blacklist some MSVC complex functions. * `#12083 <https://github.com/numpy/numpy/pull/12083>`__: TST: Add azure CI testing to 1.15.x branch. * `#12084 <https://github.com/numpy/numpy/pull/12084>`__: BUG: test_path() now uses Path.resolve() * `#12085 <https://github.com/numpy/numpy/pull/12085>`__: TST, MAINT: Fix some failing tests on azure-pipelines mac and... * `#12187 <https://github.com/numpy/numpy/pull/12187>`__: BUG: Fix memory leak in mapping.c * `#12188 <https://github.com/numpy/numpy/pull/12188>`__: BUG: Allow boolean subtract in histogram * `#12189 <https://github.com/numpy/numpy/pull/12189>`__: BUG: Fix in-place permutation * `#12190 <https://github.com/numpy/numpy/pull/12190>`__: BUG: limit default for get_num_build_jobs() to 8 * `#12191 <https://github.com/numpy/numpy/pull/12191>`__: BUG: OBJECT_to_* should check for errors * `#12192 <https://github.com/numpy/numpy/pull/12192>`__: DOC: Prepare for NumPy 1.15.3 release. * `#12237 <https://github.com/numpy/numpy/pull/12237>`__: BUG: Fix MaskedArray fill_value type conversion. * `#12238 <https://github.com/numpy/numpy/pull/12238>`__: TST: Backport azure-pipeline testing fixes for Mac Cheers, Charles Harris

1 0

Removing priority labels from github
by Matti Picus 20 Oct '18

20 Oct '18

We currently have highest, high, normal, low, and lowest priority labels for github issues/PRs. At the recent status meeting, we proposed consolidating these to a single "high" priority label. Anything "low" priority should be merged or closed since it will be quickly forgotten, and no "normal" tag is needed. With that, we (the BIDS team) would like to encourage reviewers to use the "high" priority tag to indicate things we should be working on. Any objections or thoughts? Matti (in the names of Tyler and Stefan)

5 5

BIDS/NumPy dev meetings, Wednesdays 12pm Pacific
by Stefan van der Walt 19 Oct '18

19 Oct '18

Hi everyone, The team at BIDS meets once a week to discuss progress, priorities, and roadblocks. While our priorities are broadly determined by the project roadmap [0], we would like to provide an opportunity for the community to give more regular and detailed feedback on our work. We therefore invite you to join us for our weekly calls, each **Wednesday from 12:00 to 13:00 Pacific Time**. Detail of the next meeting (2018-10-17) is given in the agenda [1], which is a growing document—feel free to add topics you wish to discuss. We hope to see you there! I will send another reminder next week. Best regards, Stéfan [0] https://www.numpy.org/neps/index.html [1] https://hackmd.io/YZfpGn5BSu6acAFLBaRjtw#

5 6

Approving NEP 27 - Historical discussion of 0-D arrays
by Matti Picus 18 Oct '18

18 Oct '18

In PR 12166 https://github.com/numpy/numpy/pull/12166 we revived an old wiki document discussing the implementation of 0-dimensional arrays. This became informational NEP-27 http://www.numpy.org/neps/nep-0027-zero-rank-arrarys.html. There was fruitful discussion of the NEP and the need for both 0-D arrays and scalars on the PR comments. The NEP itself is informational and freezes the information to the 2006 discussion, noting that "some of the information here is dated, for instance indexing of 0-D arrays now is now implemented and does not error." I would like to submit the NEP for discussion and approval. Matti

4 4

random.choice(replace=False) very slow
by Matthew Brett 17 Oct '18

17 Oct '18

Hi, I noticed that numpy.random.choice was very slow, with the replace=False option, and then I noticed it can (for most cases) be made many hundreds of times faster in Python code: In [18]: sample = np.random.uniform(size=1000000) In [19]: timeit np.random.choice(sample, 500, replace=False) 42.1 ms ± 214 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) IIn [22]: def rc(x, size): ...: n = np.prod(size) ...: n_plus = n * 2 ...: inds = np.unique(np.random.randint(0, n_plus+1, size=n_plus))[:n] ...: return x[inds].reshape(size) In [23]: timeit rc(sample, 500) 86.5 µs ± 421 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)each) Is there a reason why it's so slow in C? Could something more intelligent than the above be used to speed it up? Cheers, Matthew

2 1

summary of NumFOCUS Summit & roadmap presentations
by Ralf Gommers 16 Oct '18

16 Oct '18

Hi all, At the end of September the NumFOCUS Summit was held; Allan and I both attended on behalf of NumPy. I've written up a summary of the event from a NumPy perspective: https://rgommers.github.io/2018/10/2018-numfocus-summit---a-summary/. I also link in that post to both of the presentations I gave on the NumPy roadmap. I suspect that what I wrote raises more questions than it answers, so questions/ideas/criticism very welcome! Cheers, Ralf

1 0

A zeros_like implementation based on calloc instead of copyto
by Mark Harfouche 15 Oct '18

15 Oct '18

Hello, Currently, `zeros_like` is based `copyto` as opposed to `calloc`. This causes inconsistencies in the amount of time it takes to create an array with `zeros` + `shape` and `zeros_like` for large arrays. This was first raised https://github.com/numpy/numpy/issues/9909 It seems to me that a memory copy can be avoided by using `PyArray_NewFromDescr_int` in C. I propose creating a new C_API function `PyArray_NewZerosLikeArray` that behaves much like the `PyArray_NewLikeArray` with the exception that it calls `PyArray_NewFromDescr_int` instead of `PyArray_NewFromDescr` to initialize the array to zeros with calloc. An all C implementation of `zeros_like` is also possible by adapting the `empty_like` function. A draft implementation is viewable https://github.com/hmaarrfk/numpy/pull/2/files for those looking for more details about my proposed implementation. Thank you for considering. Mark

1 0

Exact semantics of ufunc.reduce
by Hameer Abbasi 12 Oct '18

12 Oct '18

Hello! I’m trying to investigate the exact way ufunc.reduce works when given a custom dtype. Does it cast before or after the operation, or somewhere in between? How does this differ from ufunc.reduceat, for example? We ran into this issue in pydata/sparse#191 (https://github.com/pydata/sparse/issues/191) when trying to match the two where the only thing differing is the number of zeros for sum, which shouldn’t change the result. Best Regards, Hameer Abbasi

2 1

Re: [Numpy-discussion] ndrange, like range but multidimensiontal
by Mark Harfouche 12 Oct '18

12 Oct '18

Allan, Sorry for the delay. I had my mailing list preferences set to digest. I changed them for now. (I hope this message continues that thread). Thank you for your feedback. You are correct in identifying that the real feature is expanding the `ndindex` API to support slicing. See comments about the separate points you raised below ## Expanding the API of ndindex > Better rather to add the slicing functionality to ndindex, than create a whole new nearly-identical function. This is a very important point. I should have included a note about it. My [first attempt]( https://github.com/hmaarrfk/numpy/pull/1/files#diff-1bd953557a98073031ce66d…) did try that approach. I ran into 2 issues: 1. Getting around the catch-all positional argument is annoying, and logic to do that will likely be error prone. Peculiarities about how we implement it might cause some very strange for `tuple-like` inputs that we don't expect. 2. `ndindex` is an iterator itself. As proposed, `ndrange`, like `range`, is not an iterator. Changing this behaviour would likely lead to breaking code that uses that assumption. For example anybody using introspection or code like: ``` indx = np.ndindex(5, 5) next(indx) # Don't look at the (0, 0) coordinate for i in indx: print(i) ``` would break if `ndindex` becomes "not an iterator" For these two reasons, I thought it was easier to simply have a new class, that seems like a close sibling to `ndindex`. I personally don't care about point 1 so much. In my mind, start, stop and step is confusing in ND. but maybe some might find it useful? Point 1 also makes it harder to make `ndrange` more familiar to `range` users. > I don't like adding more obscure functions Hopefully the name `ndrange` makes it easier to find? ## Writing vectorized code > np.ndindex is already a somewhat obscure and discouraged method since it is usually better to find a vectorized numpy operation instead of a for loop I understand that this kind of function is not focused on `numerical` operations on the elements of the matrix itself. It really is there to help fill the void of any useful multi-dimensional python container. I think `ndrange`/`ndindex` is there to be used like `np.vectorized`. I've tried to use `np.vectorize` in my own code, but quickly found that making logic fit into vectorize's requirements was often more complicated than writing my own loop multi-nested loops. In my opinion, nested `range` loops or `ndrange`/`ndindex` is a much more natural way to loop over collections compared to `np.vectorized`. I'm glad to add warnings to the docs. ## Implementation detail: itertools.product + range vs nditer > Maybe there is a way to use the same optimization tricks as in the current implementation of ndindex but allow different stop/step? My primary goal here is to make `ndrange` behave much like `range`. By implementing it on top of `range`, it makes it obvious to me how to enforce that behaviour as the API of range gets expanded (though it seems to have settled since Python 3.3). Whatever we decide to call `ndrange`/`ndindex`, the tests I wrote can help ensure we have good range-API coverage (for now). itertools.product + range seems to be much faster than the current implementation of ndindex (python 3.6) ``` %%timeit for i in np.ndindex(100, 100): pass 3.94 ms ± 19.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) %%timeit import itertools for i in itertools.product(range(100), range(100)): pass 231 µs ± 1.09 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each) ```

4 14