Mailman 3 November 2013 - NumPy-Discussion

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

C-API: multidimensional array indexing?
by Johann Bauer 31 Mar '16

31 Mar '16

Dear experts, is there a C-API function for numpy which implements Python's multidimensional indexing? Say, I have a 2d-array PyArrayObject * M; and an index int i; how do I extract the i-th row or column M[i,:] respectively M[:,i]? I am looking for a function which gives again a PyArrayObject * and which is a view to M (no copied data; the result should be another PyArrayObject whose data and strides points to the correct memory portion of M). I searched the API documentation, Google and mailing lists for quite a long time but didn't find anything. Can you help me? Thanks, Johann

3 3

taking a 2D uneven surface slice
by Bryan Woods 04 Dec '14

04 Dec '14

I'm trying to do something that at first glance I think should be simple but I can't quite figure out how to do it. The problem is as follows: I have a 3D grid Values[Nx, Ny, Nz] I want to slice Values at a 2D surface in the Z dimension specified by Z_index[Nx, Ny] and return a 2D slice[Nx, Ny]. It is not as simple as Values[:,:,Z_index]. I tried this: >>> values.shape (4, 5, 6) >>> coords.shape (4, 5) >>> slice = values[:,:,coords] >>> slice.shape (4, 5, 4, 5) >>> slice = np.take(values, coords, axis=2) >>> slice.shape (4, 5, 4, 5) >>> Obviously I could create an empty 2D slice and then fill it by using np.ndenumerate to fill it point by point by selecting values[i, j, Z_index[i, j]]. This just seems too inefficient and not very pythonic.

3 3

Silencing NumPy output
by Frédéric Bastien 27 Feb '14

27 Feb '14

Hi, NumPy 1.8 removed the private NumPy interface numpy.distutils.__config__. So a Theano user make a PR to make Theano use the official interface: numpy.distutils.system_info.get_info("blas_opt") But this output many stuff to the output. I can silence part of it by silencing warnings, but I'm not able to silence this output: Found executable /usr/bin/gfortran ATLAS version 3.8.3 built by mockbuild on Wed Jul 28 02:12:34 UTC 2010: UNAME : Linux x86-15.phx2.fedoraproject.org 2.6.32-44.el6.x86_64 #1 SMP Wed Jul 7 15:47:50 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux INSTFLG : -1 0 -a 1 ARCHDEFS : -DATL_OS_Linux -DATL_ARCH_Corei7 -DATL_CPUMHZ=1596 -DATL_SSE3 -DATL_SSE2 -DATL_SSE1 -DATL_USE64BITS -DATL_GAS_x8664 F2CDEFS : -DAdd_ -DF77_INTEGER=int -DStringSunStyle CACHEEDGE: 524288 F77 : gfortran, version GNU Fortran (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) F77FLAGS : -O -g -Wa,--noexecstack -fPIC -m64 SMC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SMCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64 SKC : gcc, version gcc (GCC) 4.5.0 20100716 (Red Hat 4.5.0-3) SKCFLAGS : -fomit-frame-pointer -mfpmath=sse -msse3 -O2 -fno-schedule-insns2 -g -Wa,--noexecstack -fPIC -m64 -L/opt/lisa/os_v2/canopy/appdata/canopy-1.1.0.1371.rh5-x86_64/../../appdata/canopy-1.1.0.1371.rh5-x86_64/lib -lptf77blas -lptcblas -latlas I tried to redirect the stdout and stderr, but it don't work. I looked into NumPy code and I don't see a way to change that from a library that use NumPy. Is there a way to access to silence that output? Is there a new place of the old interface: numpy.distutils.__config__ that I can reuse? It don't need to be a public interface. thanks Frédéric

5 13

Speedup by avoiding memory alloc twice in scalar array
by Arink Verma 08 Jan '14

08 Jan '14

Hi, I am working on performance parity between numpy scalar/small array and python array as GSOC mentored By Charles. Currently I am looking at PyArray_Return, which allocate separate memory just for scalar return. Unlike python which allocate memory once for returning result of scalar operations; numpy calls malloc twice once for the array object itself, and a second time for the array data. These memory allocations are happening in PyArray_NewFromDescr and PyArray_Scalar. Stashing both within a single allocation would be more efficient. In, PyArray_Scalar, new struct (PyLongScalarObject) need allocation in case of scalar arrays. Instead, can we just some how convert/cast PyArrayObject to PyLongScalarObject.?? -- Arink Verma www.arinkverma.in

5 12

-ffast-math
by Dan Goodman 03 Dec '13

03 Dec '13

Hi, Is it possible to get access to versions of ufuncs like sin and cos but compiled with the -ffast-math compiler switch? I recently noticed that my weave.inline code was much faster for some fairly simple operations than my pure numpy code, and realised after some fiddling around that it was due to using this switch. The speed difference was enormous in my test example. I checked the difference in accuracy and over the range of values I'm interested in the errors are not really significant. If there's currently no way of using these faster versions of these functions in numpy, maybe it would be worth adding as a feature for future versions? Thanks, Dan Goodman

4 9

array aligned on a different byte-boundary than reported by dtype.alignment (travis-ci)
by Henry Gomersall 30 Nov '13

30 Nov '13

I'm running some test code on travis-ci, which is currently failing, but passing locally. I've identified the problem as being that my code tests internally for the alignment of an array being its "natural alignment", which I establish by checking "data_pointer%test_array.dtype.alignment" (I do this actually on the pointer in Cython, but the same is true if I check .ctypes.data). The issue I'm having is that the array that is being created inside the travis-ci environment is _not_ naturally aligned. I don't do anything funny in creating the array - it comes from a .copy() of another array. Specifically, the problem is that .dtype.alignment is reporting 32 on a complex256 array, but the actual alignment is 16. My machine (running a much more up to date Ubuntu installation) reports 16 for the .dtype.alignment attribute on a similar array. Is this something that can be expected in some situation? Is there an explanation? Is this a bug? Cheers, Henry p.s. The error is raised here: https://github.com/hgomersall/pyFFTW/blob/travis-debug/pyfftw/pyfftw.pyx#L7… through https://github.com/hgomersall/pyFFTW/blob/travis-debug/pyfftw/builders/_uti… in which the problematic input array is simply created as in https://github.com/hgomersall/pyFFTW/blob/travis-debug/test/test_pyfftw_bui… With output: https://travis-ci.org/hgomersall/pyFFTW/jobs/14739340 (the last line)

1 0

sum of array for masked area
by questions anon 29 Nov '13

29 Nov '13

Hi All, I just posted this on the SciPy forum but realised it might be more appropriate here? I have a separate text file for daily rainfall data that covers the whole country. I would like to calculate the monthly mean, min, max and the mean of the sum for one state. The mean, max and min are just the mean, max and min for all data in that month however the sum data needs to work out the total for the month across the array and then sum that value. I use gdal tools to mask out the rest of the country and I use numpy tools for the summary stats. I can get the max, min and mean for the state, but the mean of the sum keeps giving me a result for the whole country rather than just the state, even though I am performing the analysis on the state only data. I am not sure if this is a masking issue or a numpy calculation issue. The mask works fine for the other summary statistics. Any feedback will be greatly appreciated! import numpy as np import matplotlib.pyplot as plt from numpy import ma as MA from mpl_toolkits.basemap import Basemap from datetime import datetime as dt from datetime import timedelta import os from StringIO import StringIO from osgeo import gdal, gdalnumeric, ogr, osr import glob import matplotlib.dates as mdates import sys shapefile=r"/Users/state.shp" ## Create masked array from shapefile xmin,ymin,xmax,ymax=[111.975,- 9.975, 156.275,-44.525] ncols,nrows=[886, 691] #Your rows/cols maskvalue = 1 xres=(xmax-xmin)/float(ncols) yres=(ymax-ymin)/float(nrows) geotransform=(xmin,xres,0,ymax,0, -yres) 0 src_ds = ogr.Open(shapefile) src_lyr=src_ds.GetLayer() dst_ds = gdal.GetDriverByName('MEM').Create('',ncols, nrows, 1 ,gdal.GDT_Byte) dst_rb = dst_ds.GetRasterBand(1) dst_rb.Fill(0) #initialise raster with zeros dst_rb.SetNoDataValue(0) dst_ds.SetGeoTransform(geotransform) err = gdal.RasterizeLayer(dst_ds, [maskvalue], src_lyr) dst_ds.FlushCache() mask_arr=dst_ds.GetRasterBand(1).ReadAsArray() np.set_printoptions(threshold='nan') mask_arr[mask_arr == 255] = 1 newmask=MA.masked_equal(mask_arr,0) ### calculate monthly summary stats for state Only rainmax=[] rainmin=[] rainmean=[] rainsum=[] yearmonthlist=[] yearmonth_int=[] errors=[] OutputFolder=r"/outputfolder" GLOBTEMPLATE = r"/daily-rainfall/combined/rainfall-{year}/r{year}{month:02}??.txt" def accumulate_month(year, month): files = glob.glob(GLOBTEMPLATE.format(year=year, month=month)) monthlyrain=[] for ifile in files: try: f=np.genfromtxt(ifile,skip_header=6) except: print "ERROR with file:", ifile errors.append(ifile) f=np.flipud(f) stateonly_f=np.ma.masked_array(f, mask=newmask.mask) # this masks data to state print "stateonly_f:", stateonly_f.max(), stateonly_f.mean(), stateonly_f.sum() monthlyrain.append(stateonly_f) yearmonth=dt(year,month,1) yearmonthlist.append(yearmonth) yearmonthint=str(year)+str(month) d=dt.strptime(yearmonthint, '%Y%m') print d date_string=dt.strftime(d,'%Y%m') yearmonthint=int(date_string) yearmonth_int.append(yearmonthint) r_sum=np.sum(monthlyrain, axis=0) r_mean_of_sum=MA.mean(r_sum) r_max, r_mean, r_min=MA.max(monthlyrain), MA.mean(monthlyrain), MA.min(monthlyrain) rainmax.append(r_max) rainmean.append(r_mean) rainmin.append(r_min) rainsum.append(r_mean_of_sum) print " state only:", yearmonthint,r_max, r_mean, r_min, r_mean_of_sum

2 2

numpy datetime64 NaT string conversion bug & patch
by Charles G. Waldman 28 Nov '13

28 Nov '13

If you convert an array of strings to datetime64s and 'NaT' (or one of its variants) appears in the string, all subsequent values are rendered as NaT: (this is in 1.7.1 but the problem is present in current dev version as well) >>> import numpy as np >>> a = np.array(['2010', 'nat', '2030']) >>> a.astype(np.datetime64) array(['2010', 'NaT', 'NaT'], dtype='datetime64[Y]') The fix is to re-initalize 'dt' inside the loop in _strided_to_strided_string_to_datetime (patch attached) Correct behavior (with patch): >>> import numpy as np >>> a=np.array(['2010', 'nat', '2020']) >>> a.astype(np.datetime64) array(['2010', 'NaT', '2020'], dtype='datetime64[Y]') >>>

2 1

NumPy-Discussion November 2013