Mailman 3 September 2018 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker 11 Jul '23

11 Jul '23

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

Problem with libgfortran installed with pip install numpy
by David Grote 13 Nov '18

13 Nov '18

Hi - I have recently come across this problem. On my mac, I build a Fortran code, producing a shared object, that I import into Python along with numpy. This had been working fine until recently when I started seeing sag faults deep inside the Fortran code, usually in Fortran print statements. I tracked this down to a gfortran version issue. I use the brew installation of Python and gcc (using the most recent version, 8.2.0). gcc of course installs a version of libgfortran.dylib. Doing a lsof of a running Python, I see that it finds that copy of libgfortran, and also a copy that was downloaded with numpy (/usr/local/lib/python3.7/site-packages/numpy/.dylibs/libgfortran.3.dylib). Looking at numpy's copy of libgfortran, I see that it is version 4.9.0, much older. Since my code is importing numpy first, the OS seems be using numpy's version of libgfortran to link when importing my code. I know from other experience that older versions of libgfortran are not compatible with code compiled using a new version of gfortran and so therefore segfaults happen. If I download the numpy source and do python setup.py install, I don't have this problem. After this long description, my question is why is such an old version of gcc used to build the distribution of numpy that gets installed from pypi? gcc version 4.9.0 is from 2014. Can a newer version be used? Thanks! Dave

3 4

PR Cleanup
by Charles R Harris 27 Sep '18

27 Sep '18

Hi All, As usual, the top of the PR stack is getting all the attention. As a start on cleaning up the old PRs, I'd like to suggest that all the maintainers look at their (long) pending PRs and decide which they want to keep, close those they don't want to pursue, and rebase the others. Might also help if they would post here the PRs that they think we should finish up. Chuck

12 17

scaling of the covariance matrix in np.polyfit
by Andreas Nußbaumer 27 Sep '18

27 Sep '18

Hi, a while ago I tried to figure out what holds back the PR 11197. Could anyone have a look? Thanks for your help. Andreas

1 0

Proposal to accept NEP-18, __array_function__ protocol
by Stephan Hoyer 27 Sep '18

27 Sep '18

I propose to accept NEP-18, "A dispatch mechanism for NumPy’s high level array functions": http://www.numpy.org/neps/nep-0018-array-function-protocol.html Since the last round of discussion, we added a new section on "Callable objects generated at runtime" clarifying that to handle such objects is out of scope for the initial proposal in the NEP. If there are no substantive objections within 7 days from this email, then the NEP will be accepted; see NEP 0 for more details. Cheers, Stpehan

12 61

Numpy 1.14.6 release
by Charles R Harris 23 Sep '18

23 Sep '18

Hi All, On behalf of the NumPy team, I am pleased to announce the release of NumPy 1.14.6. This is a bug fix release for bugs and regressions reported following the 1.14.5 release. The significant fixes are: - Fix for behavior change in ``ma.masked_values(shrink=True)`` - Fix the new cached allocations machinery to be thread safe. The Python versions supported by this release are 2.7, 3.4-3.7. The wheels are linked with OpenBLAS v0.3.0, which should fix some of the linalg problems reported for earlier releases of NumPy 1.14. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.14.6/>, source archives are available from Github <https://github.com/numpy/numpy/releases/tag/v1.14.6>. Compatibility Note ================== The NumPy 1.14.6 OS X wheels released on PyPI no longer contain 32-bit binaries. See #11625 <https://github.com/numpy/numpy/issues/11625> for the related discussion. Those needing 32-bit support should look elsewhere or build from source. Contributors ============ A total of 4 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Charles Harris - Eric Wieser - Julian Taylor - Matti Picus Pull requests merged ==================== A total of 4 pull requests were merged for this release. - #11985: BUG: fix cached allocations without the GIL - #11986: BUG: Undo behavior change in ma.masked_values(shrink=True) - #11987: BUG: fix refcount leak in PyArray_AdaptFlexibleDType - #11995: TST: Add Python 3.7 testing to NumPy 1.14. Cheers, Charles Harris

1 0

NumPy 1.15.2 released
by Charles R Harris 23 Sep '18

23 Sep '18

Hi All, On behalf of the NumPy team, I am pleased to announce the release of NumPy 1.15.2. This is a bug fix release for bugs and regressions reported following the 1.15.2 release. The fixes are: - the matrix PendingDeprecationWarning is now suppressed in pytest 3.8, - the new cached allocations machinery has been fixed to be thread safe, - the boolean indexing of subclasses now works correctly, - a small memory leak in PyArray_AdaptFlexibleDType has been fixed. The Python versions supported by this release are 2.7, 3.4-3.7. The wheels are linked with OpenBLAS v0.3.0, which should fix some of the linalg problems reported for NumPy 1.14. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.15.2/>, source archives are available from Github <https://github.com/numpy/numpy/releases/tag/v1.15.2> Compatibility Note ================== The NumPy 1.15.x OS X wheels released on PyPI no longer contain 32-bit binaries. That will also be the case in future releases. See `#11625 <https://github.com/numpy/numpy/issues/11625>`__ for the related discussion. Those needing 32-bit support should look elsewhere or build from source. Contributors ============ A total of 4 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Charles Harris * Julian Taylor * Marten van Kerkwijk * Matti Picus Pull requests merged ==================== A total of 4 pull requests were merged for this release. * `#11902 <https://github.com/numpy/numpy/pull/11902>`__: BUG: Fix matrix PendingDeprecationWarning suppression for pytest... * `#11981 <https://github.com/numpy/numpy/pull/11981>`__: BUG: fix cached allocations without the GIL for 1.15.x * `#11982 <https://github.com/numpy/numpy/pull/11982>`__: BUG: fix refcount leak in PyArray_AdaptFlexibleDType * `#11992 <https://github.com/numpy/numpy/pull/11992>`__: BUG: Ensure boolean indexing of subclasses sets base correctly. Cheers, Charles Harris

1 0

rescuing missing data documents
by Ralf Gommers 22 Sep '18

22 Sep '18

Hi all, Recently a few people (Tyler, Matti, myself, maybe others?) were discussing the history of missing data in the context of custom dtypes. I mentioned that there was a good write-up, but couldn't find it at the time. I just stumbled across it, it's hidden in the numpy.org repo: http://www.numpy.org/NA-overview. So hope this helps. That document further references two alternative NEPs that didn't make it into our repo: 1. https://gist.github.com/njsmith/1056379 (authors Matthew Brett, Nathaniel Smith), 2. https://gist.github.com/njsmith/1068264 (author Nathaniel Smith), as well as another summary: https://github.com/njsmith/numpy/wiki/NA-discussion-status I'd like to see the NEPs rescued, and the summary moved from the numpy.org repo (maybe into yet another NEP?). It would be best if the original authors do this, but otherwise I'm happy to do it. Thoughts? Cheers, Ralf

3 3