Mailman 3 April 2018 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker 11 Jul '23

11 Jul '23

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

Where to discuss NEPs (was: Re: new NEP: np.AbstractArray and np.asabstractarray)
by Nathaniel Smith 31 May '18

31 May '18

On Thu, Mar 8, 2018 at 7:06 AM, Marten van Kerkwijk <m.h.vankerkwijk(a)gmail.com> wrote: > Hi Nathaniel, > > Overall, hugely in favour! For detailed comments, it would be good to > have a link to a PR; could you put that up? Well, there's a PR here: https://github.com/numpy/numpy/pull/10706 But, this raises a question :-). (One which also came up here: https://github.com/numpy/numpy/pull/10704#issuecomment-371684170) There are sensible two workflows we could use (or at least, two that I can think of): 1. We merge updates to the NEPs as we go, so that whatever's in the repo is the current draft. Anyone can go to the NEP webpage at http://numpy.org/neps (WIP, see #10702) to see the latest version of all NEPs, whether accepted, rejected, or in progress. Discussion happens on the mailing list, and line-by-line feedback can be done by quote-replying and commenting on individual lines. From time to time, the NEP author takes all the accumulated feedback, updates the document, and makes a new post to the list to let people know about the updated version. This is how python-dev handles PEPs. 2. We use Github itself to manage the review. The repo only contains "accepted" NEPs; draft NEPs are represented by open PRs, and rejected NEPs are represented by PRs that were closed-without-merging. Discussion uses Github's commenting/review tools, and happens in the PR itself. This is roughly how Rust handles their RFC process, for example: https://github.com/rust-lang/rfcs Trying to do some hybrid version of these seems like it would be pretty painful, so we should pick one. Given that historically we've tried to use the mailing list for substantive features/planning discussions, and that our NEP process has been much closer to workflow 1 than workflow 2 (e.g., there are already a bunch of old NEPs already in the repo that are effectively rejected/withdrawn), I think we should maybe continue that way, and keep discussions here? So my suggestion is discussion should happen on the list, and NEP updates should be merged promptly, or just self-merged. Sound good? -n -- Nathaniel J. Smith -- https://vorpus.org

7 15

Adding fweights and aweights to numpy.corrcoef
by Corin Hoad 12 May '18

12 May '18

Hello, Would it be possible to add the fweights and aweights keyword arguments from np.cov to np.corrcoef? They would retain their meaning from np.cov as frequency- or importance-based weightings respectively. Yours, Corin Hoad

3 6

Extending ufunc signature syntax for matmul, frozen dimensions
by Matti Picus 02 May '18

02 May '18

In looking to solve issue #9028 "no way to override matmul/@ if __array_ufunc__ is set", it seems there is consensus around the idea of making matmul a true gufunc, but matmul can behave differently for different combinations of array and vector: (n,k),(k,m)->(n,m) (n,k),(k) -> (n) (k),(k,m)->(m) Currently there is no way to express that in the ufunc signature. The proposed solution to issue #9029 is to extend the meaning of a signature so "syntax like (n?,k),(k,m?)->(n?,m?) could mean that n and m are optional dimensions; if missing in the input, they're treated as 1, and then dropped from the output" Additionally, there is an open pull request #5015 "Add frozen dimensions to gufunc signatures" to allow signatures like '(3),(3)->(3)'. I would like extending ufunc signature handling to implement both these ideas, in a way that would be backwardly-compatible with the publicly exposed PyUFuncObject. PyUFunc_FromFuncAndDataAndSignature is used to allocate and initialize a PyUFuncObject, are there downstream projects that allocate their own PyUFuncObject not via PyUFunc_FromFuncAndDataAndSignature? If so, we could use one of the "reserved" fields, or extend the meaning of the "identity" field to allow version detection. Any thoughts? Any other thoughts about extending the signature syntax? Thanks, Matti

6 14

summary of "office Hours" open discusison April 25
by Matti Picus 01 May '18

01 May '18

Office Hours 25April 2018 12:00 -13:00 PDT Present: Matti Picus, Allan Haldane, Ralf Gommers, Matthew Brett, Tyler Reddy, Stéfan van der Walt, Hameer Abbasi Some of the people were not present for the entire discussion, audio was a little flaky at times. Topics: Grant background overview Matti has been browsing through issues and pull-requests to try to get a handle on common themes and community pain points. - Policy questions: - Do we close duplicate issues? (answer - Yes, referencing the other issue, as long as they are true duplicates ) - Do we close tutorial-like issues that are documented?(answer - Yes, maybe improving documentation) - Common theme - there are many issues about overflow, mainly about int32. Maybe add a mode or command switch for warning on int32 overflow? - Requested topic for discussion - improving CI and MacOS testing - How to filter CI issues on github? There is a component:build label but it is not CI specific - What about MacOS testing - should it be sending notices? (answer - Probably) - Running ASV benchmarking (https://asv.readthedocs.io/en/latest/). It is done with SciPy, but it is fragile, not done nightly; need ability to run branches more robustly documentation on SciPy site https://github.com/scipy/scipy/tree/master/benchmarks - Hameer: f2py during testing is the system one, not the internal one Most of the remaining discussion was a meta-discussion about how the community will continue to decide priorities and influence how the full-time developers spend their time. - Setting up a community-driven roadmap would be useful - Be aware of the risks of having devoted developer time on a community project - Influence can be subtle: ideally, community writes roadmap, instead of simply commenting on proposal - Can we distill past lessons to inform future decisions? - In general, how to determine community priorities? - Constant communication paramount, looks like things are going in the right direction. Furher resources to consider: - How did Jupyter organize their roadmap (ask Brian Granger)? - How did Pandas run the project with a full time maintainer (Jeff Reback)? - Can we copy other projects' management guidelines? We did not set a time for another online discussion, since it was felt that maybe near/during the sprint in May would be appropriate. I apologize for any misrepresentation. Matti Picus

3 2

axis and keepdims arguments for generalized ufuncs
by Marten van Kerkwijk 30 Apr '18

30 Apr '18

Hi All, When introducing the ``axes`` argument for generalized ufuncs, the plan was to eventually also add ``axis`` and ``keepdims`` for reduction-like gufuncs. I have now attempted to do so in https://github.com/numpy/numpy/pull/11018 It is not completely feature-compatible with reductions in that one cannot (yet) pass in a tuple or None to ``axis``. Comments most welcome. All the best, Marten

1 0

numpy.pad -- problem?
by Virgil Stokes 29 Apr '18

29 Apr '18

Here is a python code snippet: # python vers. 3.6.5 (v3.6.5:f59c0932b4, Mar 28 2018, 17:00:18) [MSC v.1900 64 bit (AMD64)] import numpy as np # numpy vers. 1.14.3 #import matplotlib.pyplot as plt N = 21 amp = 10 t = np.linspace(0.0,N-1,N) arg = 2.0*np.pi/(N-1) y = amp*np.sin(arg*t) print('y:\n',y) print('mean(y): ',np.mean(y)) #plt.plot(t,y) #plt.show() ypad = np.pad(y, (3,2),'mean') print('ypad:\n',ypad) When I execute this the outputs are: y: [ 0.00000000e+00 3.09016994e+00 5.87785252e+00 8.09016994e+00 9.51056516e+00 1.00000000e+01 9.51056516e+00 8.09016994e+00 5.87785252e+00 3.09016994e+00 1.22464680e-15 -3.09016994e+00 -5.87785252e+00 -8.09016994e+00 -9.51056516e+00 -1.00000000e+01 -9.51056516e+00 -8.09016994e+00 -5.87785252e+00 -3.09016994e+00 -2.44929360e-15] mean(y): -1.3778013372117948e-16 ypad: [-1.37780134e-16 -1.37780134e-16 -1.37780134e-16 0.00000000e+00 3.09016994e+00 5.87785252e+00 8.09016994e+00 9.51056516e+00 1.00000000e+01 9.51056516e+00 8.09016994e+00 5.87785252e+00 3.09016994e+00 1.22464680e-15 -3.09016994e+00 -5.87785252e+00 -8.09016994e+00 -9.51056516e+00 -1.00000000e+01 -9.51056516e+00 -8.09016994e+00 -5.87785252e+00 -3.09016994e+00 -2.44929360e-15 -7.40148683e-17 -7.40148683e-17] The left pad is correct, but the right pad is different and not the mean of y) --- why?

3 4

NumPy 1.14.3 released
by Allan Haldane 28 Apr '18

28 Apr '18

Hi All, I am pleased to announce the release of NumPy 1.14.3. This is a bugfix release for a few bugs reported following the 1.14.2 release: * np.lib.recfunctions.fromrecords accepts a list-of-lists, until 1.15 * In python2, float types use the new print style when printing to a file * style arg in "legacy" print mode now works for 0d arrays The Python versions supported in this release are 2.7 and 3.4 - 3.6. The Python 3.6 wheels available from PIP are built with Python 3.6.2 and should be compatible with all previous versions of Python 3.6. The source releases were cythonized with Cython 0.28.2. Contributors ============ A total of 6 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Allan Haldane * Charles Harris * Jonathan March + * Malcolm Smith + * Matti Picus * Pauli Virtanen Pull requests merged ==================== A total of 8 pull requests were merged for this release. * `#10862 <https://github.com/numpy/numpy/pull/10862>`__: BUG: floating types should override tp_print (1.14 backport) * `#10905 <https://github.com/numpy/numpy/pull/10905>`__: BUG: for 1.14 back-compat, accept list-of-lists in fromrecords * `#10947 <https://github.com/numpy/numpy/pull/10947>`__: BUG: 'style' arg to array2string broken in legacy mode (1.14... * `#10959 <https://github.com/numpy/numpy/pull/10959>`__: BUG: test, fix for missing flags['WRITEBACKIFCOPY'] key * `#10960 <https://github.com/numpy/numpy/pull/10960>`__: BUG: Add missing underscore to prototype in check_embedded_lapack * `#10961 <https://github.com/numpy/numpy/pull/10961>`__: BUG: Fix encoding regression in ma/bench.py (Issue #10868) * `#10962 <https://github.com/numpy/numpy/pull/10962>`__: BUG: core: fix NPY_TITLE_KEY macro on pypy * `#10974 <https://github.com/numpy/numpy/pull/10974>`__: BUG: test, fix PyArray_DiscardWritebackIfCopy... Cheers, Allan Haldane

2 1