Mailman 3 January 2016 - NumPy-Discussion

Re: [Numpy-discussion] ENH: Add the function 'expand_view'
by John Kirkham 09 Jan '16

09 Jan '16

First, off sorry for the long turnaround on responding to these questions. Below I have tried to respond to everyone's questions and comments. I have restructured the order of the messages so that my responses are a little more structured. If anybody has more thoughts or questions, please let me know. >>>> Takes an array and tacks on arbitrary dimensions on either side, which >>> is returned as a view always. Here are the relevant features: >>>> >>>> * Creates a view of the array that has the dimensions before and after >>> tacked on to it. >>>> * Takes the before and after arguments independent of each other and >>> the current shape. >>>> * Allows for read and write access to the underlying array. >>> >>> Can you expand this with some discussion of why you want this function, >>> and why you chose these specific features? (E.g. as mentioned in the PR >>> comments already, the reason broadcast_to returns a read-only array is that >>> it was decided that this was less confusing for users, not because of any >>> technical issue.) Sometimes I find that broadcasting is insufficient or gets confused with some more complex cases. Even adding the extra dimensions using `np.newaxis`/`None` doesn't always cut it. So, being able to add the dimensions with the right shape and without copying is very nice. However, I wanted to do this in a way where this would always be a view onto the original data to avoid any copying penalty. It also came in handy when I didn't know how many dimensions I would be dealing with. The alternative would be to do something doing things like `a[..., (b.ndim-1)*(None,)]` and possibly something similar with `b`, which end up being pretty gross and unreadable as opposed to doing `expand_view(a, reps_after=b.shape[1:])`, which is quite clear. The latter also conveys to reader what those dimensions are doing. Finally in cases where `None` or `np.newaxis` work fine, it is possible for you to combine them in some operation you are doing in way which you did not intend. If broadcasting would have worked in this case, you won't get an error, but you will get the wrong answer, which you may discover later than you would like. When using `expand_view` an explicit shape can be set, which when mismatched will give you an error allowing you discover the problem as soon as you run the code as opposed after when you are trying to figure out your answer is wrong. The write option I never actually use so making it readonly is fine. I just never blocked that behavior. Perhaps it would be preferable as has been suggested. Ensuring a view is nice because this remains performant for large arrays. Using a view seems reasonable here based on the fact that we are never trying to restructure the data here. Instead, we are trying to make it appear as if its shape were different. > How is this different from using np.newaxis and broadcasting? Or am I > misunderstanding this? I hope the description above also helps clarify this a bit. To be more succinct and explicit: * This can always be used even when broadcasting may get confused. * Works cleanly and easily with variable numbers of dimensions unlike `np.newaxis`/`None`. * Using explicit shapes more tightly constrains behavior with the resulting view (helping catch bugs). * Provides the reader more information about how the other dimensions should work. >> Why is this a stride_trick? >> >> I thought this looks similar to expand_dims and could maybe be implemented >> with some extra options there. So, `expand_dims` adds a single dimension with a length of 1 where specified, but not multiple. Though I am going to infer by extra options you mean change it some way that it can add multiple axes. Though I suppose you could do this and it may even be worth pursing, it ends up being a similar idea to using `np.newaxis`/`None` a few times and so I have basically the same points to make here as I made in the last section's response. Also, I feel like using `expand_dims` with more axes specified might make it a little harder to follow than just using `np.newaxis`/`None`, but it could have its merrits. >From a performance standpoint, `expand_dims` using `reshape` to add these extra dimensions. So, it has the possibility of not returning a view, but a copy, which could take some time to build if the array is large. By using our method, we guarantee a view will always be returned; so, this allocation will never be encountered.

2 1

ANN: second release candidate for scipy 0.17.0
by Evgeni Burovski 07 Jan '16

07 Jan '16

Hi, I'm pleased to announce the availability of the second release candidate for Scipy 0.17.0. It's two days ahead of the original schedule: based on typical development patterns, I'd like to have two weekends and a full working week before January 17th, when this rc2 is supposed to become the final release. Please try this rc and report any issues on Github tracker or scipy-dev mailing list. Source tarballs and full release notes are available from Github Releases: https://github.com/scipy/scipy/releases/tag/v0.17.0rc2 Compared to rc1, the following PRs were merged: - - `#5624 <https://github.com/scipy/scipy/pull/5624>`__: FIX: Fix interpolate - - `#5625 <https://github.com/scipy/scipy/pull/5625>`__: BUG: msvc9 binaries crash when indexing std::vector of size 0 - - `#5635 <https://github.com/scipy/scipy/pull/5635>`__: BUG: misspelled __dealloc__ in cKDTree. - - `#5642 <https://github.com/scipy/scipy/pull/5642>`__: STY: minor fixup of formatting of 0.17.0 release notes. - - `#5643 <https://github.com/scipy/scipy/pull/5643>`__: BLD: fix a build issue in special/Faddeeva.cc with isnan. - - `#5661 <https://github.com/scipy/scipy/pull/5661>`__: TST: linalg tests used stdlib random instead of numpy.random. There is a bit of an unusual change in this rc. In short, in testing rc1 we found that 1) interpolate.interp1d had an undocumented feature of allowing an array-valued fill_value to be broadcast against the data being interpolated, and 2) we broke it in 0.17.0rc1. (See PR 5624 for details.) We now *think* we fixed it so that no existing code is broken. Nevertheless, I would like to encourage everyone to test their code against this rc2, especially all code which uses interp1d. Cheers, Evgeni

2 1

Numpy 1.10.4 release.
by Charles R Harris 07 Jan '16

07 Jan '16

Hi All, I'm pleased the release of the Numpy 1.10.4 (bugs stomped) release. This release was motivated by a reported segfault, but a few additional fixes were made: * gh-6922 BUG: numpy.recarray.sort segfaults on Windows, * gh-6937 BUG: busday_offset does the wrong thing with modifiedpreceding roll, * gh-6949 BUG: Type is lost when slicing a subclass of recarray, together with one minor enhancement * gh-6950 BUG trace is not subclass aware, np.trace(ma) != ma.trace(). The sources and documentation are available on Sourceforge, but no binaries. The usual windows binaries were ommitted due to problems with the toolchain. We hope that will be remedied in the future. Mac binaries can be installed from pypi. "Where is numpy 1.10.3?", you may ask. There were glitches with the uploads to pypi for that version that necessitated a version upgrade. A release manager's life is not a happy one. Many thanks to Marten van Kerkwijk, Sebastian Seberg, and Mark Wiebe for the quick bug fixes. Chuck

2 2

Building master issues on Windows
by G Young 07 Jan '16

07 Jan '16

Hello all, I've recently encountered issues building numpy off master on Windows in which setup.py complains that it can't find the Advapi library in any directories (which is an empty list). I scanned the DLL under System32 and ran /sfc scannow as Administrator, and both came up clean. Is anyone else encountering (or can reproduce) this issue, or is it just me? Any suggestions about how to resolve this problem? Thanks! Greg

2 3

Subclass awareness of outer, inner, dot, ...
by Marten van Kerkwijk 06 Jan '16

06 Jan '16

Hi All, Having just had a look at functions where astropy's quantities are silently converted to ndarray, I came across some that, in principle, are easy to solve, yet for which, as always, there is a worry about backward compatibility. So, the question arises what to do. As a specific example, consider `np.outer` which is defined in `numeric.py` and boils down to multiply(a.ravel()[:, newaxis], b.ravel()[newaxis,:], out) Since multiply is fine with subclasses, all should be well, except that before it is called, there are `a = asarray(a)` and `b = asarray(b)` statements, which mean subclasses are lost. Obviously, in this case, a simple fix would be to use `asanyarray` instead, but ideally, similar routines behave similarly and hence one would also want to change `np.inner` and `np.dot` (and perhaps more)... Hence, before doing anything, I thought I would ask whether: 1. Such changes are or are not too risky for backward compatibility; 2. If so, whether, given that `np.dot` can be caught using `__numpy_ufunc__`, one should perhaps allow functions such as `outer` also to be passed through that? All the best, Marten

1 0

Numpy 1.10.3 release.
by Charles R Harris 06 Jan '16

06 Jan '16

Hi All, A significant segfault problem has been reported against Numpy 1.10.2 and I want to make a quick 1.10.3 release to get it fixed. Two questions - What exactly is the release process that has been decided on? AFAIK, I should just do a source release on Sourceforge, ping Matthew to produce wheels for Mac and wait for him to put them on pypi, and then upload the sources to pypi. No windows binaries are to be produced. - Is there anything else that needs fixing for 1.10.3? Chuck

4 8

1.10.3 release tomorrow, 1.11.x branch this month.
by Charles R Harris 05 Jan '16

05 Jan '16

Hi All, I'm going to attempt a 1.10.3 release tomorrow to address the segfault reported in #6922 <https://github.com/numpy/numpy/issues/6922>. I'd also like to branch 1.11.x sometime this month, hopefully around Jan 18 (two weeks from now). There are some unresolved issues, in particular __numpy_ufunc__, but there is plenty of accumulated material and I'd like to get something out before the tinder buildup constitutes a fire hazard. Releases generate plenty of sparks and it would be good if the 1.11.0 release was less flamable than the 1.10 release was. I will take a look through the current bug fix PRs with intent to merge as many as possible before the branch, but enhancements will not be a high priority except for a couple that have been sitting ready in the queue for awhile. If there are some enhancements that you think need to be in the release, mention them here. Chuck

4 4

Re: [Numpy-discussion] deprecate random.random_integers
by G Young 04 Jan '16

04 Jan '16

Hello all, In light of the discussion in #6910 <https://github.com/numpy/numpy/pull/6910>, I have gone ahead and deprecated random_integers in my most recent PR here <https://github.com/numpy/numpy/pull/6910>. As this is an API change (sort of), what are people's thoughts on this deprecation? Thanks! Greg

2 1

Re: [Numpy-discussion] dtype random.rand
by G Young 03 Jan '16

03 Jan '16

Hello, Issue #6790 <https://github.com/numpy/numpy/issues/6790> had requested the enhancement of adding the dtype argument to both random.randint and random.rand. With #6910 <https://github.com/numpy/numpy/pull/6910> merged in, that addresses the first half of the request. What do people think of adding a dtype argument to random.rand? Thanks! Greg

1 0

Proposal for a new function: np.moveaxis
by Stephan Hoyer 02 Jan '16

02 Jan '16

I've put up a pull request implementing a new function, np.moveaxis, as an alternative to np.transpose and np.rollaxis: https://github.com/numpy/numpy/pull/6630 This functionality has been discussed (even the exact function name) several times over the years, but it never made it into a pull request. The most pressing issue is that the behavior of np.rollaxis is not intuitive to most users: https://mail.scipy.org/pipermail/numpy-discussion/2010-September/052882.html https://github.com/numpy/numpy/issues/2039 http://stackoverflow.com/questions/29891583/reason-why-numpy-rollaxis-is-so… In this pull request, I also allow the source and destination axes to be sequences as well as scalars. This does not add much complexity to the code, solves some additional use cases and makes np.moveaxis a proper generalization of the other axes manipulation routines (see the pull requests for details). Best of all, it already works on ndarray duck types (like masked array and dask.array), because they have already implemented transpose. I think np.moveaxis would be a useful addition to NumPy -- I've found myself writing helper functions with a subset of its functionality several times over the past few years. What do you think? Cheers, Stephan

4 3