Mailman 3 June 2013 - NumPy-Discussion

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 19 Feb '23

19 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 23 Jul '21

23 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

C-API: multidimensional array indexing?
by Johann Bauer 01 Apr '16

01 Apr '16

Dear experts, is there a C-API function for numpy which implements Python's multidimensional indexing? Say, I have a 2d-array PyArrayObject * M; and an index int i; how do I extract the i-th row or column M[i,:] respectively M[:,i]? I am looking for a function which gives again a PyArrayObject * and which is a view to M (no copied data; the result should be another PyArrayObject whose data and strides points to the correct memory portion of M). I searched the API documentation, Google and mailing lists for quite a long time but didn't find anything. Can you help me? Thanks, Johann

3 3

taking a 2D uneven surface slice
by Bryan Woods 04 Dec '14

04 Dec '14

I'm trying to do something that at first glance I think should be simple but I can't quite figure out how to do it. The problem is as follows: I have a 3D grid Values[Nx, Ny, Nz] I want to slice Values at a 2D surface in the Z dimension specified by Z_index[Nx, Ny] and return a 2D slice[Nx, Ny]. It is not as simple as Values[:,:,Z_index]. I tried this: >>> values.shape (4, 5, 6) >>> coords.shape (4, 5) >>> slice = values[:,:,coords] >>> slice.shape (4, 5, 4, 5) >>> slice = np.take(values, coords, axis=2) >>> slice.shape (4, 5, 4, 5) >>> Obviously I could create an empty 2D slice and then fill it by using np.ndenumerate to fill it point by point by selecting values[i, j, Z_index[i, j]]. This just seems too inefficient and not very pythonic.

3 3

Really cruel draft of vbench setup for NumPy (.add.reduce benchmarks since 2011)
by Yaroslav Halchenko 08 Sep '13

08 Sep '13

On Wed, 01 May 2013, Sebastian Berg wrote: > > btw -- is there something like panda's vbench for numpy? i.e. where > > it would be possible to track/visualize such performance > > improvements/hits? > Sorry if it seemed harsh, but only skimmed mails and it seemed a bit > like the an obvious piece was missing... There are no benchmark tests I > am aware of. You can try: > a = np.random.random((1000, 1000)) > and then time a.sum(1) and a.sum(0), on 1.7. the fast axis (1), is only > slightly faster then the sum over the slow axis. On earlier numpy > versions you will probably see something like half the speed for the > slow axis (only got ancient or 1.7 numpy right now, so reluctant to give > exact timings). FWIW -- just as a cruel first attempt look at http://www.onerussian.com/tmp/numpy-vbench-20130506/vb_vb_reduce.html why float16 case is so special? I have pushed this really coarse setup (based on some elderly copy of pandas' vbench) to https://github.com/yarikoptic/numpy-vbench if you care to tune it up/extend and then I could fire it up again on that box (which doesn't do anything else ATM AFAIK). Since majority of time is spent actually building it (did it with ccache though) it would be neat if you come up with more of benchmarks to run which you might think could be interesting/important. -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik

9 26

quickselect
by Charles R Harris 13 Aug '13

13 Aug '13

Hi All, There is a PR <https://github.com/numpy/numpy/pull/3360> adding quickselect to numpy as a function `partition`. Comments on name and exposure in the numpy API are welcome. Chuck

5 9

numpy.filled, again
by Nathaniel Smith 01 Jul '13

01 Jul '13

Hi all, It looks like we've gotten a bit confused and need to untangle something. There's a PR to add new functions 'np.filled' and 'np.filled_like': https://github.com/numpy/numpy/pull/2875 And there was a discussion about this on the list back in January: http://thread.gmane.org/gmane.comp.python.numeric.general/52763 I think a reasonable summary of the opinions in the thread are: - This functionality is great, ... - ...but we can't call it 'np.filled' because there's also 'np.ma.filled' which does something else... - ...but there really aren't any better names... - ...so we should overload np.empty, like: 'np.empty(shape, fill=value)' In the mean time the original submitter has continued puttering along polishing the original patch, and it's ready to merge... except it's still the original interface, somehow the thread discussion and the PR discussion never met up. So, we have to decide what to do. Personally I think that overloading np.empty is horribly ugly, will continue confusing newbies and everyone else indefinitely, and I'm 100% convinced that we'll regret implementing such a warty interface for something that should be so idiomatic. (Unfortunately I got busy and didn't actually say this in the previous thread though.) So I think we should just merge the PR as is. The only downside is the np.ma inconsistency, but, np.ma is already inconsistent (cf. masked_array.fill versus masked_array.filled!), somewhat deprecated, and AFAICT there are far more people who will benefit from a clean np.filled idiom than who actually use np.ma (and in particular its fill-value functionality). So there would be two bad-but-IMHO-acceptable options: either live with an inconsistency between np.filled and np.ma.filled, or deprecate np.ma.filled in favor of masked_array.filled (which does exactly the same thing) and eventually switch np.ma.filled to be consistent with the new np.filled. But, that's just my opinion. -n

17 61

Automatic custom dtype
by Richard Hattersley 29 Jun '13

29 Jun '13

Hi all, In my continuing adventures in the Land of Custom Dtypes I've come across some rather disappointing behaviour in 1.7 & 1.8. I've defined my own class `Time360`, and a corresponding dtype `time360` which references Time360 as its scalar type. Now with 1.6.2 I can do: >>> t = Time360(2013, 6, 29) >>> np.array([t]).dtype dtype('Time360') And since all the instances supplied to the function were instances of the scalar type for my dtype, numpy automatically created an array using my dtype. Happy days! But in 1.7 and 1.8 I get: >>> np.array([t]).dtype dtype('O') So now I just get a plain old object array. Boo! Hiss! Is this expected? Desirable? An unwanted regression? Richard

2 9

NumPy sprints @ scipy 2013
by David Cournapeau 28 Jun '13

28 Jun '13

Hi there, It is very last minute, but I have set up a page to coordinate a bit scipy 2013's numpy sprints (Friday 28 and Saturday 29th, although I may not be there the second day). Depending on the audience, I will also look for low hanging fruits. Feel free to edit the wiki at will cheers, David

2 2

low level optimization in NumPy and minivect
by Frédéric Bastien 27 Jun '13

27 Jun '13

Hi, I saw that recently Julian Taylor is doing many low level optimization like using SSE instruction. I think it is great. Last year, Mark Florisson released the minivect[1] project that he worked on during is master thesis. minivect is a compiler for element-wise expression that do some of the same low level optimization that Julian is doing in NumPy right now. Mark did minivect in a way that allow it to be reused by other project. It is used now by Cython and Numba I think. I had plan to reuse it in Theano, but I didn't got the time to integrate it up to now. What about reusing it in NumPy? I think that some of Julian optimization aren't in minivect (I didn't check to confirm). But from I heard, minivect don't implement reduction and there is a pull request to optimize this in NumPy. The advantage to concentrate some functionality in one common package is that more project benefit from optimization done to it. (after the work to use it first!) How this could be done in NumPy? NumPy have its own code generator for many dtype. We could call minivect code generator to replace some of it. What do you think of this idea? Sadly, I won't be able to spent time on the code for this, but I wanted to raise the idea while people are working on that, in case it is helpful. Frédéric [1] https://github.com/markflorisson88/minivect

7 13

NumPy-Discussion June 2013