Mailman 3 June 2019 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker July 11, 2023

July 11, 2023

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem Feb. 18, 2023

Feb. 18, 2023

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell July 22, 2021

July 22, 2021

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

NumPy 1.17.0rc1 released
by Charles R Harris July 26, 2019

July 26, 2019

Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.17.0rc1. The 1.17 release contains a number of new features that should substantially improve its performance and usefulness. The Python versions supported are 3.5-3.7, note that Python 2.7 has been dropped. Python 3.8b1 should work with the released source packages, but there are no guarantees about future releases. Highlights of this release are: - A new extensible random module along with four selectable random numbe5 generators and improved seeding designed for use in parallel processes has been added. The currently available bit generators are MT19937, PCG64, Philox, and SFC64. - NumPy's FFT implementation was changed from fftpack to pocketfft, resulting in faster, more accurate transforms and better handling of datasets of prime length. - New radix sort and timsort sorting methods. It is currently not possible to choose which will be used, but they are hardwired to the datatype and used when either ``stable`` or ``mergesort`` is passed as the method. - Overriding numpy functions is now possible by default Downstream developers should use Cython >= 0.29.10 for Python 3.8 support and OpenBLAS >= 3.7 (not currently out) to avoid problems on the Skylake architecture. The NumPy wheels on PyPI are built from the OpenBLAS development branch in order to avoid those problems. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.17.0rc1/>, source archives and release notes are available from Github <https://github.com/numpy/numpy/releases/tag/v1.17.0rc1>. *Contributors* A total of 142 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Aaron Voelker + - Abdur Rehman + - Abdur-Rahmaan Janhangeer + - Abhinav Sagar + - Adam J. Stewart + - Adam Orr + - Albert Thomas + - Alex Watt + - Alexander Blinne + - Alexander Shadchin - Allan Haldane - Ander Ustarroz + - Andras Deak - Andreas Schwab - Andrew Naguib + - Andy Scholand + - Ankit Shukla + - Anthony Sottile - Antoine Pitrou - Antony Lee - Arcesio Castaneda Medina + - Assem + - Bernardt Duvenhage + - Bharat Raghunathan + - Bharat123rox + - Bran + - Bruce Merry + - Charles Harris - Chirag Nighut + - Christoph Gohlke - Christopher Whelan + - Chuanzhu Xu + - Daniel Hrisca - Daniel Lawrence + - Debsankha Manik + - Dennis Zollo + - Dieter Werthmüller + - Dominic Jack + - EelcoPeacs + - Eric Larson - Eric Wieser - Fabrice Fontaine + - Gary Gurlaskie + - Gregory Lee + - Gregory R. Lee - Hameer Abbasi - Haoyu Sun + - He Jia + - Hunter Damron + - Ian Sanders + - Ilja + - Isaac Virshup + - Isaiah Norton + - Jaime Fernandez - Jakub Wilk - Jan S. (Milania1) + - Jarrod Millman - Javier Dehesa + - Jeremy Lay + - Jim Turner + - Jingbei Li + - Joachim Hereth + - John Belmonte + - John Kirkham - John Law + - Jonas Jensen - Joseph Fox-Rabinovitz - Joseph Martinot-Lagarde - Josh Wilson - Juan Luis Cano Rodríguez - Julian Taylor - Jérémie du Boisberranger + - Kai Striega + - Katharine Hyatt + - Kevin Sheppard - Kexuan Sun - Kiko Correoso + - Kriti Singh + - Lars Grueter + - Maksim Shabunin + - Manvi07 + - Mark Harfouche - Marten van Kerkwijk - Martin Reinecke + - Matthew Brett - Matthias Bussonnier - Matti Picus - Michel Fruchart + - Mike Lui + - Mike Taves + - Min ho Kim + - Mircea Akos Bruma - Nick Minkyu Lee - Nick Papior - Nick R. Papior + - Nicola Soranzo + - Nimish Telang + - OBATA Akio + - Oleksandr Pavlyk - Ori Broda + - Paul Ivanov - Pauli Virtanen - Peter Andreas Entschev + - Peter Bell + - Pierre de Buyl - Piyush Jaipuriayar + - Prithvi MK + - Raghuveer Devulapalli + - Ralf Gommers - Richard Harris + - Rishabh Chakrabarti + - Riya Sharma + - Robert Kern - Roman Yurchak - Ryan Levy + - Sebastian Berg - Sergei Lebedev + - Shekhar Prasad Rajak + - Stefan van der Walt - Stephan Hoyer - SuryaChand P + - Søren Rasmussen + - Thibault Hallouin + - Thomas A Caswell - Tobias Uelwer + - Tony LaTorre + - Toshiki Kataoka - Tyler Moncur + - Tyler Reddy - Valentin Haenel - Vrinda Narayan + - Warren Weckesser - Weitang Li - Wojtek Ruszczewski - Yu Feng - Yu Kobayashi + - Yury Kirienko + - @aashuli + - @euronion + - @luzpaz - @parul + - @spacescientist + Cheers, Charles Harris

4 13

defining a NumPy API standard?
by Ralf Gommers July 15, 2019

July 15, 2019

Hi all, I have an idea that I've discussed with a few people in person, and the feedback has generally been positive. So I'd like to bring it up here, to get a sense of if this is going to fly. Note that this is NOT a proposal at this point. Idea in five words: define a NumPy API standard Observations ------------ - Many libraries, both in Python and other languages, have APIs copied from or inspired by NumPy. - All of those APIs are incomplete, and many deviate from NumPy either by accident or on purpose. - The NumPy API is very large and ill-defined. Libraries with a NumPy-like API ------------------------------- In Python: - GPU: Tensorflow, PyTorch, CuPy, MXNet - distributed: Dask - sparse: pydata/sparse - other: tensorly, uarray/unumpy, ... In other languages: - JavaScript: numjs - Go: Gonum - Rust: rust-ndarray, rust-numpy - C++: xtensor - C: XND - Java: ND4J - C#: NumSharp, numpy.net - Ruby: Narray, xnd-ruby - R: Rray This is an incomplete list. Xtensor and XND aim for multi-language support. These libraries are of varying completeness, size and quality - everything from one-person efforts that have just started, to large code bases that go beyond NumPy in features or performance. Idea ---- Define a standard for "the NumPy API" (or "NumPy core API", or .... - it's just a name for now), that other libraries can use as a guide on what to implement and when to say they are NumPy compatible. In scope: - Define a NumPy API standard, containing an N-dimensional array object and a set of functions. - List of functions and ndarray methods to include. - Recommendations about where to deviate from NumPy (e.g. leave out array scalars) Out of scope, or to be treated separately: - dtypes and casting - (g)ufuncs - function behavior (e.g. returning views vs. copies, which keyword arguments to include) - indexing behavior - submodules (fft, random, linalg) Who cares and why? - Library authors: this saves them work and helps them make decisions. - End users: consistency between libraries/languages, helps transfer knowledge and understand code - NumPy developers: gives them a vocabulary for "the NumPy API", "compatible with NumPy", etc. Risks: - If not done well, we just add to the confusion rather than make things better. - Opportunity for endless amount of bikeshedding - ? Some more rationale: We (NumPy devs) mostly have a shared understanding of what is "core NumPy functionality", what we'd like to remove but are stuck with, what's not used a whole lot, etc. Examples: financial functions don't belong, array creation methods with weird names like np.r_ were a mistake. We are not communicating this in any way though. Doing so would be helpful. Perhaps this API standard could even have layers, to indicate what's really core, what are secondary sets of functionality to include in other libraries, etc. Discussion and next steps ------------------------- What I'd like to get a sense of is: - Is this a good idea to begin with? - What should the scope be? - What should the format be (a NEP, some other doc, defining in code)? If this idea is well-received, I can try to draft a proposal during the next month (help/volunteers welcome!). It can then be discussed at SciPy'19 - high-bandwidth communication may help to get a set of people on the same page and hash out a lot of details. Thoughts? Ralf

15 39

NumPy master open for 1.18 development.
by Charles R Harris June 30, 2019

June 30, 2019

Hi All, NumPy 1.17.x has been branched. Chuck

1 0

1.17. 0 Release
by Charles R Harris June 29, 2019

June 29, 2019

Hi All, I've put up a partially edited version of the 1.17.0 release notes <https://github.com/numpy/numpy/pull/13869>, review would be much appreciated. The use of ` and ' can be especially tricky. Once the preparation is finished, I will make a branch and put out the first release candidate. Chuck

1 0

Syntax Improvement for Array Transpose
by Stewart Clelland June 27, 2019

June 27, 2019

Hi All, Based on discussion with Marten on github <https://github.com/numpy/numpy/issues/13797>, I have a couple of suggestions on syntax improvements on array transpose operations. First, introducing a shorthand for the Hermitian Transpose operator. I thought "A.HT" might be a viable candidate. Second, the adding an array method that operates like a normal transpose. To my understanding, "A.tranpose()" currently inverts the usual order of all dimensions. This may be useful in some applications involving tensors, but is not what I would usually assume a transpose on a multi-dimensional array would entail. I suggest a syntax of "A.MT" to indicate a transpose of the last two dimensions by default, maybe with optional arguments (i,j) to indicate which two dimensions to transpose. I'm new to this mailing list format, hopefully I'm doing this right :) Thanks, Stew

16 64

new MaskedArray class
by Allan Haldane June 25, 2019

June 25, 2019

Hi all, Chuck suggested we think about a MaskedArray replacement for 1.18. A few months ago I did some work on a MaskedArray replacement using `__array_function__`, which I got mostly working. It seems like a good time to bring it up for discussion now. See it at: https://github.com/ahaldane/ndarray_ducktypes It should be very usable, it has docs you can read, and it passes a pytest-suite with 1024 tests adapted from numpy's MaskedArray tests. What is missing? It needs even more tests for new functionality, and a couple numpy-API functions are missing, in particular `np.median`, `np.percentile`, `np.cov`, and `np.corrcoef`. I'm sure other devs could also find many things to improve too. Besides fixing many little annoyances from MaskedArray, and simplifying the logic by always storing the mask in full, it also has new features. For instance it allows the use of a "X" variable to mark masked locations during array construction, and I solve the issue of how to mask individual fields of a structured array differently. At this point I would by happy to get some feedback on the design and what seems good or bad. If it seems like a good start, I'd be happy to move it into a numpy repo of some sort for further collaboration & discussion, and maybe into 1.18. At the least I hope it can serve as a design study of what we could do. Let me also drop here two more interesting detailed issues: First, the issue of what to do about .real and .imag of complex arrays, and similarly about field-assignment of structured arrays. The problem is that we have a single mask bool per element of a complex array, but what if someone does `arr.imag = MaskedArray([1,X,1])`? How should the mask of the original array change? Should we make .real and .imag readonly? Second, a more general issue of how to ducktype scalars when using `__array_function__` which I think many ducktype implementors will have to face. For MaskedArray, I created an associated "MaskedScalar" type. However, MaskedScalar has to behave differently from normal numpy scalars in a number of ways: It is not part of the numpy scalar hierarchy, it fails checks `isinstance(var, np.floating)`, and np.isscalar returns false. Numpy scalar types cannot be subclassed. We have discussed before the need to have distinction between 0d-arrays and scalars, so we shouldn't just use a 0d (in fact, this makes printing very difficult). This leads me to think that in future dtype-overhaul plans, we should consider creating a subclassable `np.scalar` base type to wrap all numpy scalar variables, and code like `isinstance(var, np.floating)` should be replaced by `isinstance(var.dtype.type, np.floating)` or similar. That is, the numeric dtype of the scalar is no longer encoded in `type(var)` but in `var.dtype`: The fact that the variable is a numpy scalar is decoupled from its numeric dtype. This is useful because there are many "associated" properties of scalars in common with arrays which have nothing to do with the dtype, which ducktype implementors want to touch. I imagine this will come up a lot: In that repo I also have an "ArrayCollection" ducktype which required a "CollectionScalar" scalar, and similarly I imagine people implementing units want the units attached to the scalar, independently of the dtype. Cheers, Allan

9 37

NumPy Community Meeting Thursday, June 26
by Sebastian Berg June 24, 2019

June 24, 2019

Hi all, There will be a NumPy Community meeting on _Thursday_ June 26 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in- progress meeting notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian PS: We decided to move to Thursday, since at least Matti is traveling. If you have something to discuss and Thursday is not good, we can do Wednesday as well.

1 1