Mailman 3 December 2011 - NumPy-Discussion

idea of optimisation?
by Xavier Barthelemy Dec. 7, 2011

Dec. 7, 2011

Hi everyone I was wondering if there is a more optimal way to write what follows: I am studying waves, so I have an array of wave crests positions, Xcrest and the positions of the ZeroCrossings, Xzeros. The goal is to find between which Xzeros my xcrest are. XXX1=XCrest CrestZerosNeighbour=np.zeros([len(XCrest),2], dtype='d') for nn in range(len(Xzeros)-1): X1=Xzeros[nn] X2=Xzeros[nn+1] indexxx1=np.where((X1<=XXX1) & (XXX1 < X2)) try: CrestZerosNeighbour[indexxx1[0]]=np.array([X1,X2]) except: pass Someone has an idea? in the spirit of (numpy.ma.masked_outside) which does exactly the opposite I want: it masks an array outside an interval. I would like to mask everything except the interval that contains my value. I do this operation a large number of times , and a loop is time consuming. thanks Xavier -- « Quand le gouvernement viole les droits du peuple, l'insurrection est, pour le peuple et pour chaque portion du peuple, le plus sacré des droits et le plus indispensable des devoirs » Déclaration des droits de l'homme et du citoyen, article 35, 1793

3 6

Slow Numpy/MKL vs Matlab/MKL
by Oleg Mikulya Dec. 7, 2011

Dec. 7, 2011

Hi, How to make Numpy to match Matlab in term of performance ? I have tryied with different options, using different MKL libraries and ICC versions, still Numpy is below Matalb for certain basic tasks by ~2x. About 5 years ago I was able to get about same speed, not anymore. Matlab suppose to use same MKL, what it the reason of such Numpy slowness (beside one, yet fundamental, task) ? My conditions: *site.cfg*: <<<->>> [DEFAULT] library_dirs = /opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64 include_dirs = /opt/intel/composer_xe_2011_sp1.7.256/mkl/include mkl_libs = mkl_mc3, mkl_intel_thread, mkl_intel_lp64, mkl_core blas_libs = mkl_blas95_lp64 lapack_libs = mkl_lapack95_lp64 [lapack_opt] library_dirs = /opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64 include_dirs = /opt/intel/composer_xe_2011_sp1.7.256/mkl/include libraries = mkl_lapack95_lp64 [blas_opt] library_dirs = /opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64 include_dirs = /opt/intel/composer_xe_2011_sp1.7.256/mkl/include libraries = mkl_blas95_lp64 <<<->>> *intelccompiler.py*: <<<->>> ... linker_flags = '-O3 -openmp -lpthread -xHOST -fPIC -parallel -m64' compiler_opt_flags = '-static -xHOST -O3 -fPIC -mkl=parallel -ipo -parallel -m64' icc_run_string = 'icc ' + compiler_opt_flags icpc_run_string = 'icpc ' + compiler_opt_flags linker_run_string = 'icc ' + linker_flags + ' -shared ' ... <<<->>> *test.py* <<<->>> import numpy from numpy import random import time n=10000 m=10000 A=random.rand(n,m) b=numpy.ones((n,1)) tic = time.time() x=numpy.linalg.solve(A,b) toc = time.time() dt=toc-tic print "lin eq %7.1f " %dt tic = time.time() x=numpy.linalg.eig(A) toc= time.time() dt=toc-tic print "eig %7.1f " %dt tic = time.time() x=numpy.linalg.svd(A) toc= time.time() dt=toc-tic print "svd %7.1f " %dt <<<->>> *test.m* <<<->>> n=10000; m=10000; A=rand(n,m); b=ones(n,1); disp('lin eq'); tic; x=A\b; toc; disp('Eig'); tic; x=eig(A); toc; disp('SVD'); tic; x=svd(A); toc; <<<->>> Results: Linux_2.6.32...x86_64, Scientific Linux 6.1, i7, 12 GB RAM, python 2.7, Numpy 1.6, Matlab R2011b: Test Python/Numpy Matlab lin eq 14.6 14.9 eig 750.8 312.8 svd 431.9 271.2

4 4

loop through values in a array and find maximum as looping
by questions anon Dec. 7, 2011

Dec. 7, 2011

I would like to produce an array with the maximum values out of many (10000s) of arrays. I need to loop through many multidimentional arrays and if a value is larger (in the same place as the previous array) then I would like that value to replace it. e.g. a=[1,1,2,2 11,2,2 1,1,2,2] b=[1,1,3,2 2,1,0,0 1,1,2,0] where b>a replace with value in b, so the new a should be : a=[1,1,3,2] 2,1,2,2 1,1,2,2] and then keep looping through many arrays and replace whenever value is larger. I have tried numpy.putmask but that results in TypeError: putmask() argument 1 must be numpy.ndarray, not list Any other ideas? Thanks

6 21

What does fftn take as parameters?
by Roger Binns Dec. 6, 2011

Dec. 6, 2011

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 (Note I'm a programmer type, not a math type and am doing coding directed by a matlab user.) I'm trying to do an fft on multiple columns of data at once (ultimately feeding into a correlation calculation). I can use fft() to work on one column: data=[23, 43, 53, 54, 0, 10] powtwo=8 # nearest power of two size numpy.fft.fft(data, powtwo) I want to do that but using fftn (the matlab user said it is the right function) but I can't work out from the docs or experimentation how the input data should be formatted. eg is it row major or column major. For example the above could be: data=[ [23, 43, 53, 54, 0, 10] ] or data=[ [23], [43], [53], [54], [0], [10] ] All the examples in the docs use square inputs (ie x and y axes are the same length) so that doesn't help. The documentation shows examples of the output, but not the input. I found code passing in a single int (not a list of int) as the s parameter, but that also gives me an error. Roger -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iEYEARECAAYFAk7dN2YACgkQmOOfHg372QQ4YQCg4sKmtx8UAoEOuosWzUofw/KZ B5AAoKeHzP8HgpvDrXDANj0wqll5L9MO =iRAX -----END PGP SIGNATURE-----

5 8

howto store 2D function values and their grid points
by K.-Michael Aye Dec. 6, 2011

Dec. 6, 2011

Dear all, I can't wrap my head around this. Mathematically it's not hard, I just don't know how to store and access it without many loops. I have a function f(x,y). I would like to calculate it at x = arange(20,101,20) and y = arange(2,30,2) How do I store that in a multi-dimensional array and preserve the grid points where I did the calculation so that I could plot later groups of function plots like this: for elem in y-values plot(x_values, f(x,y=elem) or for elem in x-values: plot(y_values, f(x=elem,y) I can smell that the solution can not be so hard, but I currently understand how to keep the points where I did the evaluation of the function. Maybe it is also imaginable to do something with functional tricks as 'map'? Thanks for any suggestions! Best regards, Michael

2 1

adding unsigned int and int
by Skipper Seabold Dec. 6, 2011

Dec. 6, 2011

Hi, Is this intended? [~/] [1]: np.result_type(np.uint, np.int) [1]: dtype('float64') [~/] [2]: np.version.version [2]: '2.0.0.dev-aded70c' Skipper

2 2

ignore NAN in numpy.true_divide()
by questions anon Dec. 6, 2011

Dec. 6, 2011

I am trying to calculate the mean across many netcdf files. I cannot use numpy.mean because there are too many files to concatenate and I end up with a memory error. I have enabled the below code to do what I need but I have a few nan values in some of my arrays. Is there a way to ignore these somewhere in my code. I seem to face this problem often so I would love a command that ignores blanks in my array before I continue on to the next processing step. Any feedback is greatly appreciated. netCDF_list=[] for dir in glob.glob(MainFolder + '*/01/')+ glob.glob(MainFolder + '*/02/')+ glob.glob(MainFolder + '*/12/'): for ncfile in glob.glob(dir + '*.nc'): netCDF_list.append(ncfile) slice_counter=0 print netCDF_list for filename in netCDF_list: ncfile=netCDF4.Dataset(filename) TSFC=ncfile.variables['T_SFC'][:] fillvalue=ncfile.variables['T_SFC']._FillValue TSFC=MA.masked_values(TSFC, fillvalue) for i in xrange(0,len(TSFC)-1,1): slice_counter +=1 #print slice_counter try: running_sum=N.add(running_sum, TSFC[i]) except NameError: print "Initiating the running total of my variable..." running_sum=N.array(TSFC[i]) TSFC_avg=N.true_divide(running_sum, slice_counter) N.set_printoptions(threshold='nan') print "the TSFC_avg is:", TSFC_avg

3 7

Re: [Numpy-discussion] numpy 1.7.0 release
by Tim Burgess Dec. 5, 2011

Dec. 5, 2011

> > On Mon, Dec 5, 2011 at 9:13 PM, Charles R Harris > <charlesr.harris(a)gmail.com>wrote: > >> >> >> On Mon, Dec 5, 2011 at 1:08 PM, Travis Oliphant <oliphant(a)enthought.com>wrote: >> >>> I like the idea. Is there resolution to the NA question? >>> >> >> No, people still disagree and are likely to do so for years to come with >> no end in sight. That's why the preview label. >> >> Agreed that it's not resolved, but I think we at least got to the point > where we agreed not to back out the complete missing data additions. So if > we clearly say that we keep all options for future API changes open > (=preview label), I don't think that the issue should hold up a numpy > release indefinitely. > > Ralf I think a release is a good idea. In addition to the previous points mentioned, having NA in as a preview in a 1.7.0 release will likely raise it's visibility - a lot of people will read release notes of a newer version but won't ever track discussions in a mailing list. Tim Burgess Software Engineer - Coral Reef Watch Satellite Applications and Research - NESDIS National Oceanic and Atmospheric Administration

1 0

failure to register ufunc loops for user defined types
by Geoffrey Irving Dec. 5, 2011

Dec. 5, 2011

Hello, I'm trying to add a fixed precision rational number dtype to numpy, and am running into an issue trying to register ufunc loops. The code in question looks like int npy_rational = PyArray_RegisterDataType(&rational_descr); PyObject* equal = ... // extract equal object from the imported numpy module int types[3] = {npy_rational,npy_rational,NPY_BOOL}; if (PyUFunc_RegisterLoopForType((PyUFuncObject*)ufunc,npy_rational,rational_ufunc_##name,_types,0)<0) return 0; In Python 2.6.7 with the latest numpy from git, I get >>> from rational import * >>> i = array([rational(5,3)]) >>> i array([5/3], dtype=rational) >>> equal(i,i) Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: ufunc 'equal' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'' The same thing happens with (rational,rational)->rational ufuncs like multiply. The full extension module code is here: https://github.com/girving/poker/blob/rational/rational.cpp I realize this isn't much information to go on, but let me know if anything comes to mind in terms of possible reasons or further tests to run. Unfortunately it looks like the ufunc ntypes and types properties aren't updated based on user-defined loops, so I'm not yet sure if the problem is in registry or resolution. It's also possible someone else hit this before: http://projects.scipy.org/numpy/ticket/1913. Thanks, Geoffrey

5 16

astype does not work with NA object
by Bruce Southey Dec. 5, 2011

Dec. 5, 2011

Hi, I mistakenly filed ticket 1973 "Can not display a masked array containing np.NA values even if masked" (http://projects.scipy.org/numpy/ticket/1973) against masked array because that was where I found it. But the actual error is that the astype function does not handle the NA object: $ python Python 2.7 (r27:82500, Sep 16 2010, 18:02:00) [GCC 4.5.1 20100907 (Red Hat 4.5.1-3)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy as np >>> np.__version__ '2.0.0.dev-059334c' >>> np.array([1,2,3,4]).astype(float) array([ 1., 2., 3., 4.]) >>> np.array([1,2,3,np.NA]).astype(float) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: Cannot assign NA to an array which does not support NAs >>> a=np.array([1,2,3,4], maskna=True) >>> a[3]=np.NA >>> a array([1, 2, 3, NA]) >>> a.astype(float) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: Cannot assign NA to an array which does not support NAs >>>a*1.0 array([ 1., 2., 3., NA]) Bruce

1 0