Mailman 3 September 2010 - NumPy-Discussion

compile process of numpy-1.5.0 fails in Itanium cluster with icc
by Miguel Pan Fidalgo 21 Sep '10

21 Sep '10

hi, I'm trying to install numpy-1.5.0 in our Itanium (IA-64) supercomputer [http://www.cesga.es/content/view/917/115/lang,en/ ]. This is my site.cfg (I have to install umfpack but I think this is not the problem): $ cat site.cfg [DEFAULT] library_dirs = /opt/cesga/fftw-3.2.2/lib:/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64 include_dirs = /opt/cesga/fftw-3.2.2/include:/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include #[umfpack] #umfpack_libs = umfpack [fftw] libraries = fftw3 [mkl] library_dirs = /opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64 lapack_libs = mkl_lapack mkl_libs = mkl_core The only hacks that I made: $ diff numpy/distutils/system_info.py ../numpy- scipy/numpy-1.5.0/numpy/distutils/system_info.py 818c818 < _lib_mkl = ['mkl','mkl_lapack','guide'] --- > _lib_mkl = ['mkl','vml','guide'] $ diff numpy/distutils/intelccompiler.py ../numpy- scipy/numpy-1.5.0/numpy/distutils/intelccompiler.py 11c11 < cc_exe = 'icc -O3 -openmp -lpthread -fPIC' --- > cc_exe = 'icc' And I have used this script to launch the compilation process: $ cat compilar.sh #!/bin/bash module load nose/0.10.4 fftw/3.2.2 ifort/11.1.056 icc/11.1.056 mkl/10.2.2 python setup.py config --compiler=intele --fcompiler=intele build install -- prefix=/home/cesga/mpan/numpy-1.5.0 exit 0 blas and lapack are detected properly: $ head -n 40 compilar.log Running from numpy source directory.F2PY Version 1 blas_opt_info: blas_mkl_info: FOUND: libraries = ['mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include'] FOUND: libraries = ['mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include'] lapack_opt_info: lapack_mkl_info: mkl_info: FOUND: libraries = ['mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include'] FOUND: libraries = ['mkl_lapack', 'mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include'] FOUND: libraries = ['mkl_lapack', 'mkl_core', 'pthread'] library_dirs = ['/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/lib/64'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/cesga/fftw-3.2.2/include', '/opt/cesga/intel/intel11.0/Compiler/11.1/056/mkl/include'] running config running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options but, unfortunately, the compilation process ends with an unexpected realocation problem: $ tail -n 11 compilar.log /opt/cesga/intel/intel11.0/Compiler/11.1/056/bin/ia64/icc -shared build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o -Lbuild/temp.linux-ia64-2.4 -o build/lib.linux-ia64-2.4/numpy/core/umath_tests.so ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol inner1d_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol innerwt_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol matrix_multiply_signature ld: final link failed: Nonrepresentable section on output ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol inner1d_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol innerwt_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol matrix_multiply_signature ld: final link failed: Nonrepresentable section on output error: Command "/opt/cesga/intel/intel11.0/Compiler/11.1/056/bin/ia64/icc - shared build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o -Lbuild/temp.linux-ia64-2.4 -o build/lib.linux-ia64-2.4/numpy/core/umath_tests.so" failed with exit status 1 I have tried to link this files manually but I haven't any success: $ /opt/cesga/intel/intel11.0/Compiler/11.1/056/bin/ia64/icc -shared build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o -Lbuild/temp.linux-ia64-2.4 -o build/lib.linux-ia64-2.4/numpy/core/umath_tests.so ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol inner1d_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol innerwt_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol matrix_multiply_signature ld: final link failed: Nonrepresentable section on output $ /opt/cesga/intel/intel11.0/Compiler/11.1/056/bin/ia64/icc -fPIC -shared build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o -Lbuild/temp.linux-ia64-2.4 -o build/lib.linux-ia64-2.4/numpy/core/umath_tests.so ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol inner1d_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol innerwt_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol matrix_multiply_signature ld: final link failed: Nonrepresentable section on output $ /opt/cesga/intel/intel11.0/Compiler/11.1/056/bin/ia64/icc -i-dynamic -shared build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o -Lbuild/temp.linux-ia64-2.4 -o build/lib.linux-ia64-2.4/numpy/core/umath_tests.so ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol inner1d_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol innerwt_signature ld: build/temp.linux-ia64-2.4/build/src.linux- ia64-2.4/numpy/core/src/umath/umath_tests.o: @gprel relocation against dynamic symbol matrix_multiply_signature ld: final link failed: Nonrepresentable section on output with differents combinations (MKL 9.1, no optimatazions flags, no hacks.....) I have the same problem. any idea? -- Atentamente, Miguel Pan Fidalgo (Applications Technician) mail: mpan at cesga.es web: http://www.cesga.es Avda. de Vigo s/n 15705, Santiago de Compostela Telf.: +34 981 569810 - Fax: 981 594616

1 0

sum of outer products
by Hagen Fürstenau 21 Sep '10

21 Sep '10

Hi, I don't know if I'm overlooking something obvious, but is there a compact way of computing the 3-array X_{ijk} = \sum_{l} A_{il}*B_{jl}*C_{kl} out of the 2-arrays A, B, and C? - Hagen

2 2

Question about masked arrays
by Gökhan Sever 21 Sep '10

21 Sep '10

Hello, Consider these two sets of container arrays --one defined as usual np array the others as ma arrays: all_measured = np.ma.zeros((16, 18)) all_predicted = np.ma.zeros((16, 18)) all_measured2 = np.zeros((16, 18)) all_predicted2 = np.zeros((16, 18)) I do a computation within a for loop to assign 16 set of measurements into a length of 18 arrays (thus constructing a 2D array to perform overall statistics and plotting.) For the simplicity I only show a portion of all_measured and all_measured2 as: all_measured masked_array(data = [[512.632175 527.33373 565.36541 567.53967 593.86833 570.31319 574.40965 582.72649 588.21336 618.48789 593.09007 620.33474 591.10203 611.06443 655.60614 638.13193 626.71769 625.63584] [626.6435 -- -- 1183.67671 1206.82453 1183.13248 1162.5514 1180.70062 1086.53246 1078.78711 997.1642 856.57159 645.35167 696.86947 778.40914 816.03059 862.88297 901.7237] ... all_measured2 array([[ 512.632175 , 527.33373 , 565.36541 , 567.53967 , 593.86833 , 570.31319 , 574.40965 , 582.72649 , 588.21336 , 618.48789 , 593.09007 , 620.33474 , 591.10203 , 611.06443 , 655.60614 , 638.13193 , 626.71769 , 625.63584 ], [ 626.6435 , 0. , 0. , 1183.67671 , 1206.82453 , 1183.13248 , 1162.5514 , 1180.70062 , 1086.53246 , 1078.78711 , 997.1642 , 856.57159 , 645.35167 , 696.86947 , 778.40914 , 816.03059 , 862.88297 , 901.7237 ],... The issue is why masked arrays casted to regular numpy arrays as in all_measured2 case? whereas a simple numpy function np.mean and ma equivalent np.ma.mean yields same results on all_measured? Because the former requires a priori knowledge about the type of arrays, however the latter doesn't necessitate such restriction. Hope this is clear. Thanks. -- Gökhan

5 7

Viewer for 2D Numpy arrays (GUI)
by Mayank P Jain 20 Sep '10

20 Sep '10

Currently I am exporting them to csv files, but I wonder if there is a viewer that can be used with native numpy array files to view and preferably modify the 2D arrays. Any help would be appreciated. Regards Mayank P Jain V R TechNiche Transportation Modeler/Planner Phone: +91 901 356 0583

8 9

Inverting argsort(a, axis=0) to obtain column-wise ranks
by Alexander Michael 20 Sep '10

20 Sep '10

Calculating ranks by inverting the results of an argsort is straightforward and fast for 1D arrays: indices = argsort(a1) ranks = zeros_like(indices) ranks[indices] = arange(len(indices)) I was wondering if there was an equally pithy way to do this for multiple data samples stored column-wise in a 2D array. That is, is there a trick to invert the results of argsort(a2, axis=0) without iterating (in python) over the columns? Thanks, Alex

4 7

Re: [Numpy-discussion] loadtxt stop
by Neil Hodgson 19 Sep '10

19 Sep '10

oops, I meant to save my post but I sent it instead - doh! In the end, the question was; is worth adding start= and stop= markers into loadtxt to allow grabbing sections of a file between two known headers? I imagine it's something that people come up against regularly. Thanks, Neil ________________________________ From: Neil Hodgson <hodgson.neil(a)yahoo.co.uk> To: numpy-discussion(a)scipy.org Sent: Fri, 17 September, 2010 14:17:12 Subject: loadtxt stop Hi, I been looking around and could spot anything on this. Quite often I want to read a homogeneous block of data from within a file. The skiprows option is great for missing out the section before the data starts, but if there is anything below then loadtxt will choke. I wondered if there was a possibility to put an endmarker= ? For example, if I want to load text from a large! file that looks like this header line header line 1 2.0 3.0 2 4.5 5.7 ... 500 4.3 5.4 END more headers more headers 1 2.0 3.0 3.14 1.1414 2 4.5 5.7 1.14 3.1459 ... 500 4.3 5.4 0.000 0.001 END Then I can use skiprows=2, but loadtxt will choke when it gets to 'END'. To read t

5 8

Re: [Numpy-discussion] weighted mean; weighted standard error of the mean (sem)
by Christopher Barrington-Leigh 19 Sep '10

19 Sep '10

Interesting. Thanks Erin, Josef and Keith. There is a nice article on this at http://www.stata.com/support/faqs/stat/supweight.html. In my case, the model I've in mind is to assume that the expected value (mean) is the same for each sample, and that the weights are/should be normalised, whence a consistent estimator for sem is straightforward (if second moments can be assumed to be well behaved?). I suspect that this (survey-like) case is also one of the two most standard/most common expression that people want when they ask for an s.e. of the mean for a weighted dataset. The other would be when the weights are not to be normalised, but represent standard errors on the individual measurements. Surely what one wants, in the end, is a single function (or whatever) called mean or sem which calculates different values for different specified choices of model (assumptions)? And where possible that it has a default model in mind for when none is specified? thanks, Chris On Thu, Sep 9, 2010 at 9:13 PM, Keith Goodman <kwgoodman(a)gmail.com> wrote: > >>>> ma.std() > >> 3.2548815339711115 > > > > or maybe `w` reflects an underlying sampling scheme and you should > > sample in the bootstrap according to w ? > > Yes.... > > > if weighted average is a sum of linear functions of (normal) > > distributed random variables, it still depends on whether the > > individual observations have the same or different variances, e.g. > > http://en.wikipedia.org/wiki/Weighted_mean#Statistical_properties > > ...lots of possibilities. As you have shown the problem is not yet > well defined. Not much specification needed for the weighted mean, > lots needed for the standard error of the weighted mean. > > > What I can't figure out is whether if you assume simga_i = sigma for > > all observation i, do we use the weighted or the unweighted variance > > to get an estimate of sigma. And I'm not able to replicate with simple > > calculations what statsmodels.WLS gives me. > > My guess: if all you want is sigma of the individual i and you know > sigma is the same for all i, then I suppose you don't care about the > weight. > > > > > ??? > > > > Josef

2 3

Re: [Numpy-discussion] [OT: MATLAB] Any way to globally make Matlab struct attributes Python-property-like
by David Goldsmith 19 Sep '10

19 Sep '10

> Date: Tue, 14 Sep 2010 20:55:03 -0400 > From: Ken Watford <kwatford+scipy(a)gmail.com> > Subject: Re: [Numpy-discussion] [OT: MATLAB] Any way to globally make > Matlab struct attributes Python-property-like > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > On Mon, Sep 13, 2010 at 10:24 PM, David Goldsmith > <d.l.goldsmith(a)gmail.com> wrote: >> I.e., I'd, at minimum, like to globally replace >> >> get(Handel, 'Property') >> >> with >> >> object.Property >> >> and >> >> set(Handel, 'Property', value) >> >> with >> >> object.Property = value >> >> to an arbitrary level of composition. > > Both structures and objects of both handle and value types work > exactly like you're asking for. > > You're probably talking about "handle graphics" objects (which are > just opaque numeric values representing things related to the UI), > which do still require the get/set functions. Yes, my mistake, sorry (I figured out the distinction a day or two after I posted). > They're just doubles, > after all. It should be feasible to write a class that wraps them for > ease of use, but that wouldn't be a topic for here. If that intrigues > you, try asking about it on StackOverflow. Of course, I'll go there if/when need be, thanks! (For now, I've managed to weasel my way into using Python for my next assignment--yippee!) DG -- In science it often happens that scientists say, 'You know that's a really good argument; my position is mistaken,' and then they would actually change their minds and you never hear that old view from them again. They really do it. It doesn't happen as often as it should, because scientists are human and change is sometimes painful. But it happens every day. I cannot recall the last time something like that happened in politics or religion. - Carl Sagan, 1987 CSICOP address

1 0

Financial TS models
by Virgil Stokes 19 Sep '10

19 Sep '10

I am considering the development of an all Python package (with numpy and matplotlib) for the modeling and analysis of financial time series. This is a rather ambitious and could take sometime before I can have something that is useful. Thus, any suggestions, pointers, etc. to related work would be appreciated. Thank you, --V

3 3

Numpy SVN frozen; move to Git
by Pauli Virtanen 18 Sep '10

18 Sep '10

Dear all, Numpy SVN repository is now frozen, and does not accept new commits. Future development should end up in the Git repository: http://github.com/numpy/numpy The next things on the TODO list: - Update any links that point to http://svn.scipy.org/svn/numpy or talk about SVN. E.g. numpy.org needs updating. - Put up documentation on how to contribute to Numpy via Git. Gitwash-generated stuff could be added to the Numpy docs. - Decide if the `numpy-svn` email list is still needed. Github has RSS feeds for the repositories (but it can also send email to the list, if we want to keep the list alive). - Core devs: create accounts on github.com and ask for push permissions to the numpy repository. Or, just push your changes to your personal forks, and send pull requests -- I'm sure we have enough people to handle it also this way. - Trac integration -- our bug DB will still stay at projects.scipy.org but other Trac functionality can maybe be integrated. Pauli

5 7