Mailman 3 May 2019 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker 11 Jul '23

11 Jul '23

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

scientific Python featured in GitHub keynote
by Ralf Gommers 02 Jun '19

02 Jun '19

Hi all, On Thursday I had the pleasure to be at GitHub Satellite, together with quite a few other maintainers from projects throughout our ecosystem, and see NumPy, Matplotlib, AstroPy and other projects highlighted prominently in Nat Friedman's keynote. It included the story of the black hole image, and the open source software that enabled that image. It's the first 21 minutes of https://www.youtube.com/watch?v=xAbJkn4uRL4. Also, we now have "used by" for each repo and the dependency graph ( https://github.com/numpy/numpy/network/dependents): right now there are 205,240 repos and 13,877 packages on GitHub that depend on NumPy. Those numbers were not easy to get before, so very useful to have them in the UI now. Cheers, Ralf

2 3

Histgrom density estimation (density kwarg) with out of range values
by Sebastian Berg 31 May '19

31 May '19

Hi all, unfortunately it was noticed in Issue 13604 [0] that when histogram is given used with a specified range and the `density=True` keyword argument out of bound values are simply discarded [1]. Discarding out of bound values makes sense when the density/normed option is not used, since in that case event counts are reported. However, when out of bound values exist, the probability density should arguably not sum up to 1 anymore. We seem to have three ways possible to continue here: 1. Call it an outright bug and fix it. 2. Add a FutureWarning, and change it later (unfortunately noisy) 3. Add a new kwarg to control what happens and a FutureWarning which can be silenced using the new kwarg. (No change will ever happen if `range` or manual bin edges were not specified.) If all agree that there is no reasonable use case for the current implementation, it would be tempting to simply change it, or use a FutureWarning (unfortunately forcing users to manually calculate the density). If there is any half-decent use case, the kwarg may be the nicer option. Personally, if no one finds has a use case, I am slightly tending towards the "bug fix" option right now. All the Best, Sebastian [0] https://github.com/numpy/numpy/issues/13604 [1] As mentioned in the documentation.

1 0

Community Call -- May 29/ 2019
by Tyler Reddy 29 May '19

29 May '19

Hi, There will be a NumPy Community Call on May 29/ 2019 at 11 am Pacific Time. Anyone may join and edit the work-in-progress meeting notes: https://hackmd.io/Au2YB5QpQjyFUcCfdT1efw?view I think we still need to sort out a link / medium for the call--hopefully that will get added to the document in time. Best wishes, Tyler

1 1

NumPy 1.16.4 released.
by Charles R Harris 28 May '19

28 May '19

Charles R Harris <charlesr.harris(a)gmail.com> Apr 21, 2019, 8:39 PM to numpy-discussion, SciPy, SciPy-User, bcc: python-announce-list Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.16.4 which contains several fixes for newly reported bugs. The Python versions supported in this release are 2.7 and 3.5-3.7. Downstream developers building this release should use Cython >= 0.29.2 and, if using OpenBLAS, OpenBLAS > v0.3.7. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.16.4/>, source archives and release notes are available from Github <https://github.com/numpy/numpy/releases/tag/v1.16.4>. If you are installing using pip, you may encounter a problem with older installed versions of NumPy that pip did not delete becoming mixed with the current version, resulting in an ``ImportError``. That problem is particularly common on Debian derived distributions due to a modified pip. The fix is to make sure all previous NumPy versions installed by pip have been removed. See #12736 <https://github.com/numpy/numpy/issues/12736> for discussion of the issue. *Contributors* A total of 10 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Charles Harris - Eric Wieser - Dennis Zollo + - Hunter Damron + - Jingbei Li + - Kevin Sheppard - Matti Picus - Nicola Soranzo + - Sebastian Berg - Tyler Reddy *Pull requests merged* A total of 16 pull requests were merged for this release. - gh-13392: BUG: Some PyPy versions lack PyStructSequence_InitType2. - gh-13394: MAINT, DEP: Fix deprecated ``assertEquals()`` - gh-13396: BUG: Fix structured_to_unstructured on single-field types (backport) - gh-13549: BLD: Make CI pass again with pytest 4.5 - gh-13552: TST: Register markers in conftest.py. - gh-13559: BUG: Removes ValueError for empty kwargs in arraymultiter_new - gh-13560: BUG: Add TypeError to accepted exceptions in crackfortran. - gh-13561: BUG: Handle subarrays in descr_to_dtype - gh-13562: BUG: Protect generators from log(0.0) - gh-13563: BUG: Always return views from structured_to_unstructured when... - gh-13564: BUG: Catch stderr when checking compiler version - gh-13565: BUG: longdouble(int) does not work - gh-13587: BUG: distutils/system_info.py fix missing subprocess import (#13523) - gh-13620: BUG,DEP: Fix writeable flag setting for arrays without base - gh-13641: MAINT: Prepare for the 1.16.4 release. - gh-13644: BUG: special case object arrays when printing rel-, abs-error Cheers, Charles Harris

2 1

numpy.org/neps not refreshing
by Matti Picus 27 May '19

27 May '19

When we merge a PR, a CI job on circleCI updates both https://github.com/numpy/neps and https://github.com/numpy/devdocs. These are meant to be served as github pages at http://www.numpy.org/neps/ and http://www.numpy.org/devdocs respectively. For some reason the devdocs is updating but the neps site is not, http://www.numpy.org/neps is still showing content from May 22 (a week ago). Could someone who is an admin to the numpy organization try to work out what is going on? It would be nice to be able to point to the updated roadmap and neps. Matti

2 1

acknowledging sponsorship and institutional partners
by Ralf Gommers 27 May '19

27 May '19

Hi all, In [1] I am adding institutional partners and sponsor logos to the numpy.org website. Sebastian made some good comments there, and I think it would be very helpful if we had some guidelines on how we acknowledge sponsorship. Our governance doc has clear text on what an Institutional Partner is, see [2]. We don't have anything written down about sponsorship though. In my open PR I followed the example of Jupyter (see [3]), which lists Institutional Partners first, followed by Sponsors. For sponsors I think we will want to define some minimum level of sponsorship for which we will put a logo somewhere (main page, or about page). Jupyter seems to just put everything together. Scikit-learn and NumFOCUS do the same on their front pages. NumFOCUS has tiered levels as well with different benefits, and displays the tiers at [4]. Page 17 of the NumFOCUS sponsorship brochure [5] spells out the sponsorship levels very clearly: from platinum at $100k to bronze at $10k, and a special level for "emerging leader" (startups) below that. I think that following the NumFOCUS model would be the most straightforward thing to do, because (a) we're part of NumFOCUS, and (b) it's very well documented. And also fairest in a way - it gives some recognition proportionally to the contribution. My PR right now lists Moore, Sloan and Tidelift as the 3 sponsors. The first two contributed on the order of $500k each (spread out over 2-3 years), while Tidelift currently contributes $1000/month. So I propose: - acknowledging all active sponsorship (within the last 12 months) by logo placement on numpy.org - acknowledging past sponsorship as well on numpy.org, but on a separate page and perhaps just as a listing rather than prominent logo placement - adopting the NumFOCUS tiered sponsorship model - listing institutional partners and sponsors in the same place, with partners first (following what Jupyter does). Thoughts? Cheers, Ralf [1] https://github.com/numpy/numpy.org/pull/21 [2] https://www.numpy.org/devdocs/dev/governance/governance.html#institutional-… [3] https://jupyter.org/about [4] https://numfocus.org/sponsors [5] https://numfocus.org/wp-content/uploads/2018/07/NumFOCUS-Corporate-Sponsors…

3 4

__skip_array_function__ discussion summary
by Sebastian Berg 25 May '19

25 May '19

Hi all, This is an attempt from me to wrap up the discussion a bit so that others can chime in if they want to. NumPy 1.17 will ship with `__array_function__` a way for array like projects (dask, cupy) to override almost all numpy functions [0]. This addition is uncontroversial. NumPy 1.17 will _not_ ship with the `__skip_array_funciton__` following a longer dicussion. For those interested, I tried to give an very short overview over the topic below. The discussion here is around the addition of `__skip_array_function__` which would allow code to use: np.ones_like.__skip_array_function__(*args) to reuse the current implementation in numpy (i.e. directly call the current code). This can simplify things drastically for some array- likes, since they do not have to provide an alternative implementation. However, PR-13585 [1] sparked a more detailed discussion, since it was going to add the use of `__skip_array_function__` internally in numpy [2]. The issue is exposure of implementation details. If we do not use it internally, a user may implement their own `np.empty_like` and rely on `np.ones_like` to use `np.empty_like` [3] internally. Thus, `np.ones_like(my_array_like)` can work without `my_array_like` having any special code for `np.ones_like`. The PR exposes the issue that if `np.ones_like` is changed to call `np.empty_like.__skip_array_function__` internally, this will break the users `my_array_like` (it will not call their own `np.empty_like` implementation. We could expect users to fix up such breaking changes, but it exposes how fragile the interaction of user types using `__skip_array_function__` and changes in the specific implementation used by numpy can be in some cases. The second option would be to make sure we use `__skip_array_function__` internally, so that users cannot expect `np.ones_like` to work because they made `np.empty_like` work in the above example (does not increase the "API surface" of NumPy). Plus it increases the issue that the numpy code itself is less readable if we use `__skip_array_function__` internally in many/all places. Those two options further have very different goals in mind for the final usage of the protocol. So that right now the solution is to step back, not include the addition and rather gain experience with the NumPy 1.17 release that includes `__array_function__` but not `__skip_array_function`. I hope this may help those interested who did not follow the full discussion, can't say I feel I am very good at summarizing. For details I encourage you to have a look at the PR discussion and the recent mails to the list. Best, Sebastian [0] http://www.numpy.org/neps/nep-0018-array-function-protocol.html#implementat… [1] https://github.com/numpy/numpy/pull/13585 [2] Mostly for slight optimization. [3] It also uses `np.copyto` which can be overridden as well.

4 3