Mailman 3 November 2017 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker 11 Jul '23

11 Jul '23

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

Type annotations for NumPy
by Stephan Hoyer 05 Dec '17

05 Dec '17

There's been growing interest in supporting PEP-484 style type annotations in NumPy: https://github.com/numpy/numpy/issues/7370 This would allow NumPy users to add type-annotations to their code that uses NumPy, which they could check with mypy, pycharm or pytype. For example: def f(x: np.ndarray) -> np.ndarray: """Identity function on a NumPy array.""" return x Eventually, we could include data types and potentially array shapes as part of the type. This gets quite a bit more complicated, and to do in a really satisfying way would require new features in Python's typing system. To help guide discussion, I wrote a doc describing use-cases and needs for typing array shapes in more detail: https://docs.google.com/document/d/1vpMse4c6DrWH5rq2tQSx3qwP_m_0lyn-Ij4WHqQ… Nathaniel Smith and I recently met with group in San Francisco interested in this topic, including several mypy/typeshed developers (Jelle Zijlstra and Ethan Smith). We discussed and came up with a plan for moving forward: 1. Release basic type stubs for numpy.ndarray without dtypes or shapes, as separate "numpy_stubs" package on PyPI per PEP 561. This will let us iterate rapidly on (experimental) type annotations without coupling to NumPy's release cycle. 2. Add support for dtypes in ndarray type-annotations. This might be as simple as writing np.ndarray[np.float64], but will need a decision about appropriate syntax for shape typing to ensure that this is forwards compatible with typing shapes. Note: this will likely require minor changes to NumPy itself, e.g., to add __class_getitem__ per PEP 560. 3. Add support for shapes in ndarray type-annotations, and define a broader standard for typing array shapes. This will require collaboration with type-checker developers on the required typing features (for details, see my doc above). Eventually, this may entail writing a PEP. I'm writing to gauge support for this general plan, and specifically to get support for step 1. Cheers, Stephan

11 20

Deprecate matrices in 1.15 and remove in 1.17?
by Marten van Kerkwijk 03 Dec '17

03 Dec '17

Hi All, I wondered if the move to python3-only starting with numpy 1.17 would be a good reason to act on what we all seem to agree: that the matrix class was a bad idea, with its overriding of multiplication and lack of support for stacks of matrices. For 1.17, minimum python supposedly is >=3.5, so we will be guaranteed to have the matrix multiply operator @ available, and hence there is arguably even less of a case for keeping the matrix class; removing it would allow taking out quite a bit of accumulated special-casing (the immediate reasons for writing this were gh-10123 and 10132). What do people think? If we do go in this direction, we might want to add PendingDeprecationWarning for 1.15 (maybe DeprecationWarning for python3; for python2 matrix would never disappear). All the best, Marten

15 25

Re: [Numpy-discussion] Is there a way that indexing a matrix of data with a matrix of indices?
by Eric Hermes 30 Nov '17

30 Nov '17

On Wed, 2017-11-29 at 12:00 -0500, numpy-discussion-request(a)python.org wrote: > Date: Wed, 29 Nov 2017 14:56:28 +0000 (UTC) > From: "ZHUO QL (KDr2)" <zhuoql(a)yahoo.com> > To: Discussion of Numerical Python <numpy-discussion(a)python.org> > Subject: [Numpy-discussion] Is there a way that indexing a matrix of > data with a matrix of indices? > Message-ID: <360382279.3966234.1511967388050(a)mail.yahoo.com> > Content-Type: text/plain; charset="utf-8" > > Hi, all > suppose: > - D, is the data matrix, its shape is? M x N- I, is the indices > matrix, its shape is M x K,? K<=N > Is there a efficient way to get a Matrix R with the same shape of I > so that R[x,y] = D[x, I[x,y]] ? > A nested for-loop or list-comprehension is too slow for me.?? > Thanks. I don't know if this will be substantially faster, but you can try the following: I += np.array(range(M))[:, np.newaxis] * N R = D.ravel()[I.ravel()].reshape((M, K)) Eric > ---- > ZHUO QL (KDr2) http://kdr2.com > -------------- next part -------------- > An HTML attachment was scrubbed... > URL: <http://mail.python.org/pipermail/numpy- > discussion/attachments/20171129/baeaddc0/attachment-0001.html>

2 1

Is there a way that indexing a matrix of data with a matrix of indices?
by ZHUO QL (KDr2) 29 Nov '17

29 Nov '17

Hi, all suppose: - D, is the data matrix, its shape is M x N- I, is the indices matrix, its shape is M x K, K<=N Is there a efficient way to get a Matrix R with the same shape of I so that R[x,y] = D[x, I[x,y]] ? A nested for-loop or list-comprehension is too slow for me. Thanks. ---- ZHUO QL (KDr2) http://kdr2.com

2 1

Re: [Numpy-discussion] Proposal of timeline for dropping Python 2.7 support
by Nathaniel Smith 17 Nov '17

17 Nov '17

See Thomas's reply quoted below (it was rejected by the mailing list since he's not subscribed): On Nov 9, 2017 01:24, "Thomas Kluyver" <thomas(a)kluyver.me.uk> wrote: On Thu, Nov 9, 2017, at 08:52 AM, Nathaniel Smith wrote: On Nov 8, 2017 23:59, "Ralf Gommers" <ralf.gommers(a)gmail.com> wrote: Regarding http://www.python3statement.org/: I'd say that as long as there are people who want to spend their energy on the LTS release (contributors *and* enough maintainer power to review/merge/release), we should not actively prevent them from doing that. Yeah, agreed. I don't feel like this is incompatible with the spirit of python3statement.org, though looking at the text I can see how it's not clear. My guess is they'd be happy to adjust the text, especially if it lets them add numpy :-). CC'ing Thomas and Matthias. Thanks Nathaniel. We have (IMO) left a degree of deliberate ambiguity around precisely what 'drop support' means, because it's not going to be the same for all projects. The nature of open source also means that there can be ambiguity over what 'support' entails and who is considered part of the project. I would say that the idea of the statement is compatible with an LTS release series receiving critical bugfixes beyond 2020, while the main energy of the project is focused on Py3-only feature releases. [If numpy-discussion doesn't allow non-member posts, feel free to pass this on or quote it in on-list messages] Thomas

11 16

PyArray_GETITEM and PyArray_SETITEM
by Mmanu Chaturvedi 17 Nov '17

17 Nov '17

Hello All, I need to make use of the limited numpy API access Pybind11 gives, in order to add a feature to it. It seems to give access to functions from numpy_api.py [1]. I need to use PyArray_GETITEM and PyArray_SETITEM in order to get and set array elements [2], these functions / macros are not exposed via numpy_api.py, but are in `numpy/ndarraytypes.h`. We were wondering why aren't PyArray_GETITEM and PyArray_SETITEM exposed like the rest of numpy API? Is it possible to replicate the behavior using the members exposed in numpy_api.py ? Any help would be appreciated. Mmanu [1] https://github.com/numpy/numpy/blob/1368cbb696ae27b849eed67b4fd31c 550a55dad5/numpy/core/code_generators/numpy_api.py [2] https://github.com/pybind/pybind11/pull/1152/files#diff- 52f1945d779be1e60903590907bb9326R241

3 2

Upcoming revision of the BLAS standard
by Nathaniel Smith 15 Nov '17

15 Nov '17

Hi NumPy and SciPy developers, Apparently there is some work afoot to update the BLAS standard, with a working document here: https://docs.google.com/document/d/1DY4ImZT1coqri2382GusXgBTTTVdBDvtD5I14QH… This seems like something where we might want to get involved in, so that the new standard works for us, and James Demmel (the first author on that proposal and a professor here at Berkeley) suggested they'd be interested to hear our thoughts. I'm not sure exactly what the process is here -- apparently there have been some workshops, and there was going to be a BoF today at Supercomputing, but I don't know what the schedule is or how they'll be making decisions. It's possible for anyone interested to click on that google doc above and make "suggestions", but it seems like maybe it would be useful for the NumPy/SciPy teams to come up with some sort of shared document on what we want? I'm really, really not the biggest linear algebra expert on these lists, so I'm hoping those with more experience will jump in, but to get started here are some initial ideas for things we might want to ask for: - Support for arbitrary strided memory layout - Replacing xerbla with proper error codes (already in that proposal) - There's some discussion about NaN handling where I think we might have opinions. (Am I remember right that currently we have to check for NaNs ourselves all the time because there are libraries that blow up if we don't, and we don't know which ones those are?) - Where the spec ends up giving implementors flexibility, some way to detect at compile time what options they chose. -n -- Nathaniel J. Smith -- https://vorpus.org

2 1