Mailman 3 November 2018 - NumPy-Discussion

Vectorized version of numpy.linspace
by Hameer Abbasi Nov. 14, 2018

Nov. 14, 2018

> On Wednesday, Nov 14, 2018 at 11:46 PM, Stephan Hoyer <shoyer(a)gmail.com (mailto:shoyer@gmail.com)> wrote: > > > On Wed, Nov 14, 2018 at 2:35 PM Sebastian Berg <sebastian(a)sipsolutions.net (mailto:sebastian@sipsolutions.net)> wrote: > > On Wed, 2018-11-14 at 14:32 -0500, Marten van Kerkwijk wrote: > > > Code being better than words: see > > > https://github.com/numpy/numpy/pull/12388 for an implementation. The > > > change in the code proper is very small, though it is worrying that > > > it causes two rather unrelated tests too fail (even if arguably both > > > tests were wrong). > > > > > > Note that this does not give flexibility to put the axis where one > > > wants; as written, the code made putting it at the start the obvious > > > solution, as it avoids doing anything with the shapes of start and > > > stop. > > > > Hehe, my first gut feeling was the last axis to be the obvious one ;). > > This has been discussed before (but what hasn't) I believe, probably > > some old issue or even PR somewhere. > > I am mildly in favor, just because there is probably not much reason > > against an easy vectorization. Doesn't need to be advertised much in > > the docs anyway. > > Although it might be good to settle the "obvious" part in case I am not > > alone in first thinking of -1 being the obvious default. I would > > probably skip the axis argument for now, unless someone actually has a > > use case. > > Indeed -- I think the best argument for adding an "axis" argument is that it allows people to be explicit about where the axis ends up, e.g., both np.linspace(start, stop, num=5, axis=0) and np.linspace(start, stop, num=5, axis=-1) make their intent quite clear. > > To me, axis=0 feels like the right default, matching np.concatenate and np.stack. But NumPy already has split conventions for this sort of thing (e.g., gufuncs add axes at the end), so I like the explicit option. I’d like to have another vote for axis=-1 by default. Stack and concatenate are different because we are concatenating/stacking complete arrays, so it makes sense to “compose” them along the first axis to maintain C-contiguous-ness. I actually think of this as the reverse, we are “composing/combining” lots of 1D arrays over all the other dimensions, so to preserve C-contiguous-ness, it’s better to have axis=-1. > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)python.org > https://mail.python.org/mailman/listinfo/numpy-discussion Best Regards, Hameer Abbasi

4 3

Weekly status meeting Nov 14 at 12:00 Pacific time
by Matti Picus Nov. 13, 2018

Nov. 13, 2018

We will be holding our weekly BIDS NumPy status meeting on Wed Nov 14 at noon Pacific time. Please join us. The draft agenda, along with details of how to join, is up at https://hackmd.io/H0x6z5uYSgex2FA6p5nlvw?both Previous sessions' notes are available at https://github.com/BIDS-numpy/docs/tree/master/status_meetings Matti, Tyler and Stefan

1 0

Problem with libgfortran installed with pip install numpy
by David Grote Nov. 12, 2018

Nov. 12, 2018

Hi - I have recently come across this problem. On my mac, I build a Fortran code, producing a shared object, that I import into Python along with numpy. This had been working fine until recently when I started seeing sag faults deep inside the Fortran code, usually in Fortran print statements. I tracked this down to a gfortran version issue. I use the brew installation of Python and gcc (using the most recent version, 8.2.0). gcc of course installs a version of libgfortran.dylib. Doing a lsof of a running Python, I see that it finds that copy of libgfortran, and also a copy that was downloaded with numpy (/usr/local/lib/python3.7/site-packages/numpy/.dylibs/libgfortran.3.dylib). Looking at numpy's copy of libgfortran, I see that it is version 4.9.0, much older. Since my code is importing numpy first, the OS seems be using numpy's version of libgfortran to link when importing my code. I know from other experience that older versions of libgfortran are not compatible with code compiled using a new version of gfortran and so therefore segfaults happen. If I download the numpy source and do python setup.py install, I don't have this problem. After this long description, my question is why is such an old version of gcc used to build the distribution of numpy that gets installed from pypi? gcc version 4.9.0 is from 2014. Can a newer version be used? Thanks! Dave

3 4

asarray/anyarray; matrix/subclass
by Hameer Abbasi Nov. 11, 2018

Nov. 11, 2018

> Begin forwarded message: > > From: Stephan Hoyer > Date: Friday, Nov 09, 2018 at 3:19 PM > To: Hameer Abbasi > Cc: Stefan van der Walt , Marten van Kerkwijk > Subject: asarray/anyarray; matrix/subclass > > This is a great discussion, but let's try to have it in public (e.g., on the NumPy mailing list). > On Fri, Nov 9, 2018 at 8:42 AM Hameer Abbasi <einstein.edison(a)gmail.com (mailto:einstein.edison@gmail.com)> wrote: > > Hi Stephan, > > > > The issue I have with writing another function is that asarray/asanyarray are so widely used that it’d be a huge maintenance task to update them throughout NumPy, not to mention other codebases, not to mention other codebases having to rely on newer NumPy versions for this. In short, it would dramatically reduce adaptability of this function. > > > > One path we can take is to allow asarray/asanyarray to be overridable via __array_function__ (the former is debatable). This solves most of our duck-array related issues without introducing another protocol. > > > > Regardless of what path we choose, I would recommend changing asanyarray to not pass through np.matrix regardless, instead passing through mat.view(type=np.ndarray) instead, which has O(1) cost and memory. In the vast majority of contexts, it’s used to ensure an array-ish structure for another operation, and usually there’s no guarantee that what comes out will be a matrix anyway. I suggest we raise a FutureWarning and then change this behaviour. > > > > There have been a number of discussions about deprecating np.matrix (and a few about MaskedArray as well, though there are less compelling reasons for that one). I suggest we start down that path as soon as possible. The biggest (only?) user I know of blocking that is scipy.sparse, and we’re on our way to replacing that with PyData/Sparse. > > > > Best Regards, > > Hameer Abbasi > > > > > > > On Friday, Nov 09, 2018 at 1:26 AM, Stephan Hoyer <shoyer(a)gmail.com (mailto:shoyer@gmail.com)> wrote: > > > Hi Hameer, > > > > > > I'd love to talk about this in more detail. I agree that something like this is needed. > > > > > > The challenge with reusing an existing function like asanyarray() is that there is at least one (somewhat?) widely used ndarray subclass that badly violates the Liskov Substitution Principle: np.matrix. > > > > > > NumPy can't really use np.asanyarray() widely for internal purposes until we don't have to worry about np matrix. We might special case np.matrix in some way, but then asanyarray() would do totally opposite things on different versions of NumPy. It's almost certainly a better idea to just write a new function with the desired semantics, and "soft deprecate" asanyarray(). The new function can explicitly black list np.matrix, as well as any other subclasses we know of that badly violate LSP. > > > > > > Cheers, > > > Stephan > > > On Thu, Nov 8, 2018 at 5:06 PM Hameer Abbasi <einstein.edison(a)gmail.com (mailto:einstein.edison@gmail.com)> wrote: > > > > No, Stefan, I’ll do that now. Putting you in the cc. > > > > > > > > It slipped my mind among the million other things I had in mind — Namely: My job visa. It was only done this Monday. > > > > > > > > Hi, Marten, Stephan: > > > > > > > > Stefan wants me to write up a NEP that allows a given object to specify that it is a duck array — Namely, that it follows duck-array semantics. > > > > > > > > We were thinking of switching asanyarray to switch to passing through anything that implements the duck-array protocol along with ndarray subclasses. I’m sure this would help XArray and Quantity work better with existing codebases, along with PyData/Sparse arrays. > > > > > > > > Would you be interested? > > > > > > > > Best Regards, > > > > Hameer Abbasi > > > > > > > > > > > > > On Thursday, Nov 08, 2018 at 9:09 PM, Stefan van der Walt <stefanv(a)berkeley.edu (mailto:stefanv@berkeley.edu)> wrote: > > > > > Hi Hameer, > > > > > > > > > > In last week's meeting, we had the following in the notes: > > > > > > > > > > > Hameer is contacting Marten & Stephan and write up a draft NEP for > > > > > > clarifying the asarray/asanyarray and matrix/subclass path forward. > > > > > > > > > > Did any of that happen that you could share? > > > > > > > > > > Thanks and best regards, > > > > > Stéfan Hello, everyone, Me, Stefan van der Walt, Stephan Hoyer and Marten van Kerkwijk were having a discussion about the state of matrix, asarray and asanyarray. Our thoughts are summarised above (in the quoted text that I’m forwarding) Basically, this grew out of a discussion relating to asanyarray/asarray inconsistencies in NumPy about which to use where. Historically, asarray was used in many libraries/places instead of asanyarray usually because np.matrix caused problems due to its special behaviour with regard to indexing (it always returns a 2-D object when eliminating one dimension, but a 0-D one when eliminating both), its behaviour regarding __mul__ (the multiplication operator represents matrix multiplication rather than element-wise multiplication) and its fixed dimensionality (matrix is 2D only). Because of these three things, as Stephan accurately pointed out, it violates the Liskov Substitution Principle. Because of this behaviour, many libraries switched from using asanyarray to asarray, as np.matrix wouldn’t work with their code. This shut out other matrix subclasses from being used as well, such as MaskedArray and astropy.Quantity. Even if asanyarray is used, there is usually no guarantee that a matrix will be returned instead of an array. The changes I’m proposing are twofold, but simple: asanyarray should return mat.view(type=np.ndarray) instead of matrices, after an appropriate time with a FutureWarning. This allows us to preserve the performance (Creating a view is O(1) both in memory and time), and the mutability of the original matrix. This change should happen after a FutureWarning and the usual grace period. In the spirit of allowing duck-arrays to work with existing NumPy code, asanyarray should be overridable via __array_function__, so that duck arrays can decide whether to pass themselves through. If subclasses are allowed, so should ducka-arrays as well. This is a part of a larger effort to deprecate np.matrix. As far as I’m aware, it has one big customer (scipy.sparse). The effort to replace that is already underway at PyData/Sparse. Best Regards, Hameer Abbasi

8 15

Weekly status meeting 8.11 at 12:00 pacific time
by Matti Picus Nov. 11, 2018

Nov. 11, 2018

We will be holding our weekly BIDS NumPy status meeting on Thurs Nov 8 at noon pacific time. We moved the meeting to Thursday because of a scheduling conflict. Please join us. The draft agenda, along with details of how to join, is up at https://hackmd.io/TTurMvviSkarcxf8vURq-Q?both Previous sessions' notes are available at https://github.com/BIDS-numpy/docs/tree/master/status_meetings Matti, Tyler and Stefan

2 1

Developer Meeting, Berkeley, 30 Nov / 1 Dec
by Stefan van der Walt Nov. 11, 2018

Nov. 11, 2018

Hi everyone, On Friday 30 November & Saturday 1 December we will host a NumPy Development Meeting at the Berkeley Institute for Data Science. We will discuss the work being done on dtypes, review NEPs under implementation, and solicit feedback for updating the community roadmap. Please get in touch if you would like to attend, so that we can tally the numbers and work out travel support. Best regards, Stéfan

1 0

Should unique types of all arguments be passed on in __array_function__?
by Marten van Kerkwijk Nov. 10, 2018

Nov. 10, 2018

Hi All, While thinking about implementations using __array_function__, I wondered whether the "types" argument passed on is not defined too narrowly. Currently, it only contains the types of arguments that provide __array_ufunc__, but wouldn't it make more sense to provide the unique types of all arguments, independently of whether those types have defined __array_ufunc__? It would seem quite useful for any override to know, e.g., whether a string or an integer is passed on. I thought of this partially as I was wondering how an implementation for ndarray itself would look like. For that, it is definitely useful to know all unique types, since if it is only ndarray, no casting whatsoever needs to be done, while if there are integers, lists, etc, an attempt has to be made to turn these into arrays (i.e., the `as[any]array` calls currently present in the implementations, which really more logically are part of `ndarray.__array_function__` dispatch). Should we change this? It is quite trivially done, but perhaps I am missing a reason for omitting the non-override types. All the best, Marten

3 9

Prep for NumPy 1.16.0 branch
by Charles R Harris Nov. 10, 2018

Nov. 10, 2018

Hi All, Time to begin looking forward to the NumPy 1.16.x branch. I think there are three main topics to address: 1. current PRs that need review and merging, 2. critical fixes that need to be made, 3. status of `__array_function__`. The last probably needs some discussion. `__array_fuction__` seems to be working at this point, but does cause noticeable slowdowns in some function calls. I don't know if those slowdowns are significant in practice, the only way to discover that may be to make the release, or at least thorough testing of the release candidates, but we should at least discuss them and possible workarounds if needed. Trying to have things in good shape is important because 1.16.x will be the last release that supports Python 2.7, and even though we will be maintaining it for the next year, that will be easier if it is stable. As to the first two topics, I think we should try to be conservative at this point and look mostly for bug fixes and documentation updates. Thoughts? Chuck

6 9

numpy pprint?
by Foad Sojoodi Farimani Nov. 7, 2018

Nov. 7, 2018

Hello everyone, Following this question <https://stackoverflow.com/q/53126305/4999991>, I'm convinced that numpy ndarrays are not MATLAB/mathematical multidimentional matrices and I should stop expecting them to be. However I still think it would have a lot of benefit to have a function like sympy's pprint to pretty print. something like pandas .head and .tail method plus .left .right .UpLeft .UpRight .DownLeft .DownRight methods. when nothing mentioned it would show 4 corners and put dots in the middle if the array is to big for the terminal. Best, Foad

7 19

out parameter for np.fromfile
by Mark Harfouche Nov. 5, 2018

Nov. 5, 2018

I was wondering what would your thoughts be on adding an output parameter to np.fromfile? The advantage would be when interfacing with executables like ffmpeg which are arguably easier to use by calling them as a subprocess compared to a shared library in python. Having the output parameter in np.fromfile would enable pre-allocation of large arrays that are reused during the computation of new image frames when decoding large video files. Thoughts are appreciated! Mark

2 2