Mailman 3 January 2021 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker 11 Jul '23

11 Jul '23

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Documentation Team meeting - Monday June 8th
by Melissa Mendonça 03 Dec '22

03 Dec '22

Hi all! A reminder that on Monday, June 8, we have another documentation team meeting at 3PM UTC**. If you wish to join on Zoom, you need to use this link https://zoom.us/j/420005230 Here's the permanent hackmd document with the meeting notes: https://hackmd.io/oB_boakvRqKR-_2jRV-Qjg <https://www.google.com/url?q=https%3A%2F%2Fhackmd.io%2FoB_boakvRqKR-_2jRV-Q…> Hope to see you around (especially if you want to introduce yourself or discuss ideas for Google Season of Docs). ** You can click this link to get the correct time at your timezone: https://www.timeanddate.com/worldclock/fixedtime.html?msg=NumPy+Documentati… - Melissa

6 74

[Feature Request] Add alias of np.concatenate as np.concat
by Iordanis Fostiropoulos 10 May '22

10 May '22

In regard to Feature Request: https://github.com/numpy/numpy/issues/16469 It was suggested to sent to the mailing list. I think I can make a strong point as to why the support for this naming convention would make sense. Such as it would follow other frameworks that often work alongside numpy such as tensorflow. For backward compatibility, it can simply be an alias to np.concatenate I often convert portions of code from tf to np, it is as simple as changing the base module from tf to np. e.g. np.expand_dims -> tf.expand_dims. This is done either in debugging (e.g. converting tf to np without eager execution to debug portion of the code), or during prototyping, e.g. develop in numpy and convert in tf. I find myself more than at one occasion to getting syntax errors because of this particular function np.concatenate. It is unnecessarily long. I imagine there are more people that also run into the same problems. Pandas uses concat (torch on the other extreme uses simply cat, which I don't think is as descriptive).

7 6

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

Unreliable crash when converting using numpy.asarray via C buffer interface
by Friedrich Romstedt 29 Mar '21

29 Mar '21

Hi, This is with Python 3.8.2 64-bit and numpy 1.19.2 on Windows 10. I'd like to be able to convert some C++ extension type to a numpy array by using ``numpy.asarray``. The extension type implements the Python buffer interface to support this. The extension type, called "Image" here, holds some chunk of ``double``, C order, contiguous, 2 dimensions. It "owns" the buffer; the buffer is not shared with other objects. The following Python code crashes:: image = <... Image production ...> ar = numpy.asarray(image) However, when I say:: image = <... Image production ...> print("---") ar = numpy.asarray(image) the entire program is executing properly with correct data in the numpy ndarray produced using the buffer interface. The extension type permits reading the pixel values by a method; copying them over by a Python loop works fine. I am ``Py_INCREF``-ing the producer in the C++ buffer view creation function properly. The shapes and strides of the buffer view are ``delete[]``-ed upon releasing the buffer; avoiding this does not prevent the crash. I am catching ``std::exception`` in the view creation function; no such exception occurs. The shapes and strides are allocated by ``new Py_ssize_t[2]``, so they will survive the view creation function. I spent some hours trying to figure out what I am doing wrong. Maybe someone has an idea about this? I double-checked each line of code related to this problem and couldn't find any mistake. Probabaly I am not looking at the right aspect. Best, Friedrich

7 16

On the use of adding an expfit class in or next to the polynomial module
by Swan BOSC 30 Jan '21

30 Jan '21

Hello people ! First-of-all, thank you for your great service to the Python community. I'm working for a project on porting some Octave code base towards Python. As you can imagine, it is a lot of fun, but I sometimes come across functions that are not (yet ?) available in the mighty NumPy package. Namely, the expfit function, as presented [here](https://octave.sourceforge.io/optim/function/expfit.html) (https://octave.sourceforge.io/optim/function/expfit.html) doesn't seem to exist. I'm currently building a replacement for it in my words (keywords I mean ;) ) but maybe a better solution would be to contribute some code into NumPy. AFAIK, the preferred way to fit a Polynomial nowadays is to call numpy.polynomial.polynomial.Polynomial. Do you think that a class for `Exponential` based on that one would be an interesting addition to NumPy ? Thanks in advance, Swan BOSC.

2 2

NumPy 1.20.0 released
by Charles R Harris 30 Jan '21

30 Jan '21

Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.20.0rc2. This NumPy release is the largest to date, containing some 684 merged pull requests contributed by 184 people. See the list of highlights below. The Python versions supported for this release are 3.7-3.9, support for Python 3.6 has been dropped. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.20.0/>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.20.0>. Linux users will need pip >= 0.19.3 in order to install manylinux2010 and manylinux2014 wheels. *Highlights* - Annotations for NumPy functions. This work is ongoing and improvements can be expected pending feedback from users. - Wider use of SIMD to increase execution speed of ufuncs. Much work has been done in introducing universal functions that will ease use of modern features across different hardware platforms. This work is ongoing. - Preliminary work in changing the dtype and casting implementations in order to provide an easier path to extending dtypes. This work is ongoing but enough has been done to allow experimentation and feedback. - Extensive documentation improvements comprising some 185 PR merges. This work is ongoing and part of the larger project to improve NumPy's online presence and usefulness to new users. - Further cleanups related to removing Python 2.7. This improves code readability and removes technical debt. - Preliminary support for the upcoming Cython 3.0. *Contributors* A total of 184 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Aaron Meurer + * Abhilash Barigidad + * Abhinav Reddy + * Abhishek Singh + * Al-Baraa El-Hag + * Albert Villanova del Moral + * Alex Leontiev + * Alex Rockhill + * Alex Rogozhnikov * Alexander Belopolsky * Alexander Kuhn-Regnier + * Allen Downey + * Andras Deak * Andrea Olivo + * Andrew Eckart + * Anirudh Subramanian * Anthony Byuraev + * Antonio Larrosa + * Ashutosh Singh + * Bangcheng Yang + * Bas van Beek + * Ben Derrett + * Ben Elliston + * Ben Nathanson + * Bernie Gray + * Bharat Medasani + * Bharat Raghunathan * Bijesh Mohan + * Bradley Dice + * Brandon David + * Brandt Bucher * Brian Soto + * Brigitta Sipocz * Cameron Blocker + * Carl Leake + * Charles Harris * Chris Brown + * Chris Vavaliaris + * Christoph Gohlke * Chunlin Fang * CloseChoice + * Daniel G. A. Smith + * Daniel Hrisca * Daniel Vanzo + * David Pitchford + * Davide Dal Bosco + * Derek Homeier * Dima Kogan + * Dmitry Kutlenkov + * Douglas Fenstermacher + * Dustin Spicuzza + * E. Madison Bray + * Elia Franzella + * Enrique Matías Sánchez + * Erfan Nariman | Veneficus + * Eric Larson * Eric Moore * Eric Wieser * Erik M. Bray * EthanCJ-git + * Etienne Guesnet + * FX Coudert + * Felix Divo * Frankie Robertson + * Ganesh Kathiresan * Gengxin Xie * Gerry Manoim + * Guilherme Leobas * Hassan Kibirige * Hugo Mendes + * Hugo van Kemenade * Ian Thomas + * InessaPawson + * Isabela Presedo-Floyd + * Isuru Fernando * Jakob Jakobson + * Jakub Wilk * James Myatt + * Jesse Li + * John Hagen + * John Zwinck * Joseph Fox-Rabinovitz * Josh Wilson * Jovial Joe Jayarson + * Julia Signell + * Jun Kudo + * Karan Dhir + * Kaspar Thommen + * Kerem Hallaç * Kevin Moore + * Kevin Sheppard * Klaus Zimmermann + * LSchroefl + * Laurie + * Laurie Stephey + * Levi Stovall + * Lisa Schwetlick + * Lukas Geiger + * Madhulika Jain Chambers + * Matthias Bussonnier * Matti Picus * Melissa Weber Mendonça * Michael Hirsch * Nick R. Papior * Nikola Forró * Noman Arshad + * Paul YS Lee + * Pauli Virtanen * Paweł Redzyński + * Peter Andreas Entschev * Peter Bell * Philippe Ombredanne + * Phoenix Meadowlark + * Piotr Gaiński * Raghav Khanna + * Raghuveer Devulapalli * Rajas Rade + * Rakesh Vasudevan * Ralf Gommers * Raphael Kruse + * Rashmi K A + * Robert Kern * Rohit Sanjay + * Roman Yurchak * Ross Barnowski * Royston E Tauro + * Ryan C Cooper + * Ryan Soklaski * Safouane Chergui + * Sahil Siddiq + * Sarthak Vineet Kumar + * Sayed Adel * Sebastian Berg * Sergei Vorfolomeev + * Seth Troisi * Sidhant Bansal + * Simon Gasse * Simon Graham + * Stefan Appelhoff + * Stefan Behnel + * Stefan van der Walt * Steve Dower * Steve Joachim + * Steven Pitman + * Stuart Archibald * Sturla Molden * Susan Chang + * Takanori H + * Tapajyoti Bose + * Thomas A Caswell * Tina Oberoi * Tirth Patel * Tobias Pitters + * Tomoki, Karatsu + * Tyler Reddy * Veniamin Petrenko + * Wansoo Kim + * Warren Weckesser * Wei Yang + * Wojciech Rzadkowski * Yang Hau + * Yogesh Raisinghani + * Yu Feng * Yuya Unno + * Zac Hatfield-Dodds * Zuhair Ali-Khan + * @abhilash42 + * @danbeibei + * @dojafrat * @dpitch40 + * @forfun + * @iamsoto + * @jbrockmendel + * @leeyspaul + * @mitch + * @prateek arora + * @serge-sans-paille + * @skywalker + * @stphnlyd + * @xoviat * @谭九鼎 + * @JMFT + * @Jack + * @Neal C + Cheers, Charles Harris

1 0

PSF Scientific WG grants
by Stefan van der Walt 27 Jan '21

27 Jan '21

Hi all, I'd like to make you aware of this call for funding from the PSF: https://pyfound.blogspot.com/2020/12/psf-scientific-working-group-announces… Funding is for up to 4000 USD. The deadline has been extended (the blog post will soon be updated accordingly); I recommend submitting requests within the next two weeks. Stéfan

1 0

Value based promotion and user DTypes
by Sebastian Berg 27 Jan '21

27 Jan '21

Hi all, does anyone have a thought about how user DTypes (i.e. DTypes not currently part of NumPy) should interact with the "value based promotion" logic we currently have? For now I can just do anything, and we will find out later. And I will have to do something for now, basically with the hope that it all turns out all-right. But there are multiple options for both what to offer to user DTypes and where we want to move (I am using `bfloat16` as a potential DType here). 1. The "weak" dtype option (this is what JAX does), where: np.array([1], dtype=bfloat16) + 4. returns a bfloat16, because 4. is "lower" than all floating point types. In this scheme the user defined `bfloat16` knows that the input is a Python float, but it does not know its value (if an overflow occurs during conversion, it could warn or error but not upcast). For example `np.array([1], dtype=uint4) + 2**5` will try `uint4(2**5)` assuming it works. NumPy is different `2.**300` would ensure the result is a `float64`. If a DType does not make use of this, it would get the behaviour of option 2. 2. The "default" DType option: np.array([1], dtype=bfloat16) + 4. is always the same as `bfloat16 + float64 -> float64`. 3. Use whatever NumPy considers the "smallest appropriate dtype". This will not always work correctly for unsigned integers, and for floats this would be float16, which doesn't help with bfloat16. 4. Try to expose the actual value. (I do not want to do this, but it is probably a plausible extension with most other options, since the other options can be the "default".) Within these options, there is one more difficulty. NumPy currently applies the same logic for: np.array([1], dtype=bfloat16) + np.array(4., dtype=np.float64) which in my opinion is wrong (the second array is typed). We do have the same issue with deciding what to do in the future for NumPy itself. Right now I feel that new (user) DTypes should live in the future (whatever that future is). I have said previously, that we could distinguish this for universal functions. But calls like `np.asarray(4.)` are common, and they would lose the information that `4.` was originally a Python float. So, recently, I was considering that a better option may be to limit this to math Python operators: +, -, /, **, ... Those are the places where it may make a difference to write: arr + 4. vs. arr + bfloat16(4.) int8_arr + 1 vs. int8_arr + np.int8(1) arr += 4. (in-place may be the most significant use-case) while: np.add(int8_arr, 1) vs. np.add(int8_arr, np.int8(1)) is maybe less significant. On the other hand, it would add a subtle difference between operators vs. direct ufunc calls... In general, it may not matter: We can choose option 1 (which the bfloat16 does not have to use), and modify it if we ever change the logic in NumPy itself. Basically, I will probably pick option 1 for now and press on, and we can reconsider later. And hope that it does not make things even more complicated than it is now. Or maybe better just limit it completely to always use the default for user DTypes? But I would be interested if the "limit to Python operators" is something we should aim for here. This does make a small difference, because user DTypes could "live" in the future if we have an idea of how that future may look like. Cheers, Sebastian

3 7