Mailman 3 July 2013 - NumPy-Discussion

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 19 Feb '23

19 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 23 Jul '21

23 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

C-API: multidimensional array indexing?
by Johann Bauer 01 Apr '16

01 Apr '16

Dear experts, is there a C-API function for numpy which implements Python's multidimensional indexing? Say, I have a 2d-array PyArrayObject * M; and an index int i; how do I extract the i-th row or column M[i,:] respectively M[:,i]? I am looking for a function which gives again a PyArrayObject * and which is a view to M (no copied data; the result should be another PyArrayObject whose data and strides points to the correct memory portion of M). I searched the API documentation, Google and mailing lists for quite a long time but didn't find anything. Can you help me? Thanks, Johann

3 3

taking a 2D uneven surface slice
by Bryan Woods 04 Dec '14

04 Dec '14

I'm trying to do something that at first glance I think should be simple but I can't quite figure out how to do it. The problem is as follows: I have a 3D grid Values[Nx, Ny, Nz] I want to slice Values at a 2D surface in the Z dimension specified by Z_index[Nx, Ny] and return a 2D slice[Nx, Ny]. It is not as simple as Values[:,:,Z_index]. I tried this: >>> values.shape (4, 5, 6) >>> coords.shape (4, 5) >>> slice = values[:,:,coords] >>> slice.shape (4, 5, 4, 5) >>> slice = np.take(values, coords, axis=2) >>> slice.shape (4, 5, 4, 5) >>> Obviously I could create an empty 2D slice and then fill it by using np.ndenumerate to fill it point by point by selecting values[i, j, Z_index[i, j]]. This just seems too inefficient and not very pythonic.

3 3

Speedup by avoiding memory alloc twice in scalar array
by Arink Verma 09 Jan '14

09 Jan '14

Hi, I am working on performance parity between numpy scalar/small array and python array as GSOC mentored By Charles. Currently I am looking at PyArray_Return, which allocate separate memory just for scalar return. Unlike python which allocate memory once for returning result of scalar operations; numpy calls malloc twice once for the array object itself, and a second time for the array data. These memory allocations are happening in PyArray_NewFromDescr and PyArray_Scalar. Stashing both within a single allocation would be more efficient. In, PyArray_Scalar, new struct (PyLongScalarObject) need allocation in case of scalar arrays. Instead, can we just some how convert/cast PyArrayObject to PyLongScalarObject.?? -- Arink Verma www.arinkverma.in

5 12

Really cruel draft of vbench setup for NumPy (.add.reduce benchmarks since 2011)
by Yaroslav Halchenko 07 Sep '13

07 Sep '13

On Wed, 01 May 2013, Sebastian Berg wrote: > > btw -- is there something like panda's vbench for numpy? i.e. where > > it would be possible to track/visualize such performance > > improvements/hits? > Sorry if it seemed harsh, but only skimmed mails and it seemed a bit > like the an obvious piece was missing... There are no benchmark tests I > am aware of. You can try: > a = np.random.random((1000, 1000)) > and then time a.sum(1) and a.sum(0), on 1.7. the fast axis (1), is only > slightly faster then the sum over the slow axis. On earlier numpy > versions you will probably see something like half the speed for the > slow axis (only got ancient or 1.7 numpy right now, so reluctant to give > exact timings). FWIW -- just as a cruel first attempt look at http://www.onerussian.com/tmp/numpy-vbench-20130506/vb_vb_reduce.html why float16 case is so special? I have pushed this really coarse setup (based on some elderly copy of pandas' vbench) to https://github.com/yarikoptic/numpy-vbench if you care to tune it up/extend and then I could fire it up again on that box (which doesn't do anything else ATM AFAIK). Since majority of time is spent actually building it (did it with ccache though) it would be neat if you come up with more of benchmarks to run which you might think could be interesting/important. -- Yaroslav O. Halchenko, Ph.D. http://neuro.debian.net http://www.pymvpa.org http://www.fail2ban.org Senior Research Associate, Psychological and Brain Sciences Dept. Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik

9 26

quickselect
by Charles R Harris 13 Aug '13

13 Aug '13

Hi All, There is a PR <https://github.com/numpy/numpy/pull/3360> adding quickselect to numpy as a function `partition`. Comments on name and exposure in the numpy API are welcome. Chuck

5 9

Re: [Numpy-discussion] unit tests / developing numpy (Pauli Virtanen)
by Graeme B. Bell 03 Aug '13

03 Aug '13

Hi Pauli, Thanks for looking into this. Apologies for mangling the email subject line in my previous reply. Answers are below, inline: > That is interesting, as it has worked for me on all configurations. > You can check under 'build/testenv/' --- does your Python version by > chance install it to a `lib64` directory instead of `lib`? $ ls build/testenv/ bin lib64 > (i) > > What does > > import os > from distutils.sysconfig import get_python_lib > get_python_lib(prefix=os.path.abspath('build/testenv')) > > report? Is there a 'numpy' directory below the reported directory after > running runtests.py? >>> import os >>> from distutils.sysconfig import get_python_lib >>> get_python_lib(prefix=os.path.abspath('build/testenv')) '/ssd-space/home/X/github/numpy/build/testenv/lib/python2.7/site-packages' I'm running this in a fresh python client in the directory 'github/numpy'. > (ii) > > Start a fresh Python interpreter and run > > import sys > print sys.modules.get('numpy') > > (Note: no "import numpy" above). Does it print `None` or something else? >>> import sys >>> print sys.modules.get('numpy') None Thanks, Pauli and Stefan. Graeme.

2 1

Search array in array
by Nicola Creati 30 Jul '13

30 Jul '13

Hello, I'm wondering if there is a fast way to solve the following problem. I have two arrays: A = [[ 4, 9, 10], [ 7, 4, 17], [12, 21, 14], [12, 24, 11], [18, 21, 3], [16, 3, 7], [17, 21, 5], [24, 3, 14]] B = [[17, 5], [14, 21]] I need to search rows of A that contain elements of each row of B regardless of the order of the elements in B. The searched results is: [2, 6] . Thanks. Nicola -- _____________________________________________________________________ Nicola Creati Istituto Nazionale di Oceanografia e di Geofisica Sperimentale - OGS IRI (Ricerca Teconologica e Infrastrutture) Department B.go Grotta Gigante - Brisciki 42/c 34010 Sgonico - Zgonik (TS) - Italy Tel. +39-0402140213 Fax +39-040327307

2 2

Re: [Numpy-discussion] unit tests / developing numpy (Pauli Virtanen)
by Graeme B. Bell 26 Jul '13

26 Jul '13

> > No, runtests builds the code and sets PYTHONPATH accordingly. > > -- > Pauli Virtanen Hello Pauli, Thanks again for writing back. I agree that may be what runtests.py is intended to do, but it is unfortunately not what it actually does in its default configuration, at least on my computer. I got burned by this a few nights ago. $ pwd /home/X/github/numpy $ more numpy/version.py # THIS FILE IS GENERATED FROM NUMPY SETUP.PY short_version = '1.8.0' $ echo $PYTHONPATH $ python runtests.py Building, see build.log... Build OK Running unit tests for numpy NumPy version 1.7.1 NumPy is installed in /usr/lib64/python2.7/site-packages/numpy Python version 2.7.3 (default, Aug 9 2012, 17:23:57) [GCC 4.7.1 20120720 (Red Hat 4.7.1-5)] nose version 1.3.0 *note the version number and directory used by runtests.py* It reported 100% tests passed (unsurprising since it was testing the release version!). In reality, the current directory at that time failed a test when I pushed it to the main repository. Can you suggest anything that I may be doing wrong here? Graeme On Jul 25, 2013, at 7:00 PM, numpy-discussion-request(a)scipy.org wrote: > Send NumPy-Discussion mailing list submissions to > numpy-discussion(a)scipy.org > > To subscribe or unsubscribe via the World Wide Web, visit > http://mail.scipy.org/mailman/listinfo/numpy-discussion > or, via email, send a message with subject or body 'help' to > numpy-discussion-request(a)scipy.org > > You can reach the person managing the list at > numpy-discussion-owner(a)scipy.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of NumPy-Discussion digest..." > > > Today's Topics: > > 1. Re: add .H attribute? (Chris Barker - NOAA Federal) > 2. Re: unit tests / developing numpy (Graeme B. Bell) > 3. Re: unit tests / developing numpy (Pauli Virtanen) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Thu, 25 Jul 2013 07:49:44 -0700 > From: Chris Barker - NOAA Federal <chris.barker(a)noaa.gov> > Subject: Re: [Numpy-discussion] add .H attribute? > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: <-29565752093052626@unknownmsgid> > Content-Type: text/plain; charset=UTF-8 > > On Jul 23, 2013, at 4:57 PM, Alan G Isaac <alan.isaac(a)gmail.com> wrote: > >> Finally, I think (?) everyone (proponents and opponents) >> would be happy if .H could provide access to an iterative >> view of the conjugate transpose. > > Except those of us that don't think numpy needs it at all. > > But I'll call it a -0 > > -Chris > > > ------------------------------ > > Message: 2 > Date: Thu, 25 Jul 2013 14:52:07 +0000 > From: "Graeme B. Bell" <grb(a)skogoglandskap.no> > Subject: Re: [Numpy-discussion] unit tests / developing numpy > To: "<numpy-discussion(a)scipy.org>" <numpy-discussion(a)scipy.org> > Message-ID: <20D22199-8C71-49A0-8292-CEBBBB70B902(a)skogoglandskap.no> > Content-Type: text/plain; charset="us-ascii" > > Nathaniel, Pauli: > > Thanks for the suggestions! > > = runtests.py is a nice solution, but unless you also set up your PYTHONPATH and install the code you've been working on, you're going to run with whichever version of numpy you have installed normally rather than the code you've just been working on (e.g. 1.7.1 rather than 1.8dev). > > Unfortunately, this caused me to submit a buggy set of commits, thinking they had passed the tests. > > > = env approach: thanks, I'll give that a try and compare it with the /tmp install approach. > > > Can we add this to the dev workflow web pages? > > Graeme. > > ------------------------------ > > Message: 3 > Date: Thu, 25 Jul 2013 18:38:11 +0300 > From: Pauli Virtanen <pav(a)iki.fi> > Subject: Re: [Numpy-discussion] unit tests / developing numpy > To: numpy-discussion(a)scipy.org > Message-ID: <ksrgos$60t$1(a)ger.gmane.org> > Content-Type: text/plain; charset=ISO-8859-1; format=flowed > > 25.07.2013 17:52, Graeme B. Bell kirjoitti: > [clip] >> = runtests.py is a nice solution, but unless you also set up your >> PYTHONPATH and install the code you've been working on, you're going >> to run with whichever version of numpy you have installed normally >> rather than the code you've just been working on (e.g. 1.7.1 rather > than 1.8dev). > > No, runtests builds the code and sets PYTHONPATH accordingly. > > -- > Pauli Virtanen > > > > ------------------------------ > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > End of NumPy-Discussion Digest, Vol 82, Issue 51 > ************************************************

2 1

NumPy-Discussion July 2013