Mailman 3 November 2011 - NumPy-Discussion

Re: [Numpy-discussion] compiling and linking MKL with numpy/scipy
by Xavier Barthelemy 24 Nov '11

24 Nov '11

Dear everyone I have been a bit better patching the files I finally get something in .basrc source /opt/intel/composer_xe_2011_sp1.7.256/bin/compilervars.sh intel64 source /opt/intel/composer_xe_2011_sp1.7.256/mkl/bin/mklvars.sh intel64 lp64 export OMP_NUM_THREADS=4 export KMP_AFFINITY=verbose,compact env variables: export LD_LIBRARY_PATH=/opt/intel/composer_xe_2011_sp1.7.256/mkl/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/compiler/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/:$LD_LIBRARY_PATH site.cfg [mkl] library_dirs = /opt/intel/composer_xe_2011_sp1.7.256/mkl/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/compiler/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/ include_dirs = /opt/intel/composer_xe_2011_sp1.7.256/mkl/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/compiler/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64/:/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/ lapack_libs = mkl_libs = mkl_def, mkl_intel_lp64, mkl_core , mkl_intel_thread, mkl_mc distutils/intelcompiler.py cc_exe = 'icc -m64 -O2 -fomit-frame-pointer -openmp -lpthread -fPIC -xHost -fp-model strict -L/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64 -I/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64 -lm' ccxx_exe = 'icpc -m64 -O2 -fomit-frame-pointer -openmp -lpthread -fPIC -xHost -fp-model strict -L/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64 -I/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64 -lm' distutils/fcompiler/intel.py return ['-O3 -openmp -fpe0 -L/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64 -lpthread -xHost -I/opt/intel/composer_xe_2011_sp1.7.256/mkl/include/intel64/lp64 -I/opt/intel/composer_xe_2011_sp1.7.256/mkl/include -lm'] in numpy.test("full") i get test_double (test_linalg.TestCond2) ... python: symbol lookup error: /opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/libmkl_sequential.so: undefined symbol: mkl_serv_lock I am trying to look for mkl_serv_lock, but impossible to find. some ideas, someone? Xavier

2 2

dtype and pep3118
by Fabrice Silva 23 Nov '11

23 Nov '11

Hi folks, how should I specify a PEP3118 buffer format that could be understand by numpy as the following dtype: dtype = [('t11', '|f8'), ('t22', '|f8'), ('t33', '|f8'), ('t23', '|f8'), ('t13', '|f8'), ('t12', '|f8')] so that I can manipulate a structured array and its fields ? I tried strings like "T{d:t11: d:t22: d:t33: d:t23: d:t13: d:t12:}:Tensor2d:" "d:t11: d:t22: d:t33: d:t23: d:t13: d:t12:" without success -- Fabrice Silva

2 2

bug in reshape ?
by josef.pktd＠gmail.com 23 Nov '11

23 Nov '11

might be an old story >>> np.__version__ -> '1.5.1' It thought for once it's easier to use reshape to add a new axis instead of ...,None but my results got weird (normal(0,1) sample of 2.13795875e-314) >>> x = 1 >>> y = np.arange(3) >>> z = np.arange(2)[:,None] >>> np.broadcast(x,y,z) <numpy.broadcast object at 0x04C0DCA0> >>> np.broadcast_arrays(x,y,z) [array([[1, 1, 1], [1, 1, 1]]), array([[0, 1, 2], [0, 1, 2]]), array([[0, 0, 0], [1, 1, 1]])] >>> x1, y1, z1 = np.broadcast_arrays(x,y,z) >>> map(np.shape, (x1, y1, z1)) [(2, 3), (2, 3), (2, 3)] shape looks fine, let's add an extra axis with reshape >>> x1.reshape(2,3,1) array([[[ 1], [ 1], [ 1099464714]], [[-2147481592], [ 184], [ 1]]]) what's that ? >>> (0+x1).reshape(2,3,1) array([[[1], [1], [1]], [[1], [1], [1]]]) >>> (y1*1.).reshape(2,3,1) array([[[ 0.], [ 1.], [ 2.]], [[ 0.], [ 1.], [ 2.]]]) >>> (y1).reshape(2,3,1) array([[[ 0], [ 1], [ 2]], [[ 0], [ 1099447643], [-2147483648]]]) >>> x1, y1, z1 = np.broadcast_arrays(x,y,z) >>> x1[...,None] array([[[1], [1], [1]], [[1], [1], [1]]]) >>> x1.shape (2, 3) >>> x1.reshape(2,3,1) array([[[ 1], [ 1], [ 1099464730]], [[-2147479536], [ -445054780], [ 1063686842]]]) the background story: playing broadcasting tricks for http://projects.scipy.org/scipy/ticket/1544 Josef

2 1

compiling and linking MKL with numpy/scipy
by Xavier Barthelemy 23 Nov '11

23 Nov '11

dear everyone, I am hard trying to compile numpy / and scipy with mkl. unfortunately it does not work. i have tried a lot of solution, and the closest for me to work is: *I have * Linux manning 2.6.32-27-server #49-Ubuntu SMP Thu Dec 2 02:05:21 UTC 2010 x86_64 GNU/Linux python -c 'import os,sys;print os.name,sys.platform' -> posix linux python 2.6 numpy 1.6.1 scipy 0.10 Intel® Fortran Compiler XE Version 12.1 Update 1, Intel® Debugger Version 12.1 Update 1, Intel® Math Kernel Library (Intel® MKL) Version 10.3 Update 7 *I used this to help me*: http://software.intel.com/en-us/articles/intel-mkl-link-line-advisor/ *the relevant part of site.cfg* in numpy: [mkl] library_dirs = /opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/ include_dirs = /opt/intel/composer_xe_2011_sp1.7.256/mkl/include/ lapack_libs = mmkl_libs = mkl_intel_ilp64 ,mkl_core ,mkl_intel_thread,mkl_mc *i**n distutils/intecompiler.py* cc_exe = 'icc -m64 -O2 -fomit-frame-pointer -openmp -lpthread -fPIC -xHost -fp-model strict -DMKL_ILP64 ' ccxx_exe = 'icpc -m64 -O2 -fomit-frame-pointer -openmp -lpthread -fPIC -xHost -fp-model strict -DMKL_ILP64 ' * in distutils/fcompiler/intel.py* def get_flags_opt(self): return ['-O3 -openmp -i8 -fpe0 -L$(MKLROOT)/lib/intel64 -lpthread,'] *I run in .bashrc* source /opt/intel/composer_xe_2011_sp1.7.256/bin/compilervars.sh intel64 source /opt/intel/composer_xe_2011_sp1.7.256/mkl/bin/mklvars.sh intel64 ilp64 *and my complation script is:* export LD_LIBRARY_PATH=/opt/intel/composer_xe_2011_sp1.7.256/mkl/lib/intel64/:/opt/intel/composer_xe_2011_sp1.7.256/compiler/lib/intel64/:$LD_LIBRARY_PATH python setup.py config --compiler=intelem build_clib --compiler=intelem build_ext --compiler=intelem install *I had to use* "-fp-model strict" to pass the simple test of numpy numpy.test() *but when I run numpy.test('full')* test_assumed_shape.TestAssumedShapeSumExample.test_all ... Segmentation fault *and then with the same config for scipy * scipy.test('full') test_definition (test_basic.TestDoubleFFT) ... Segmentation fault *which obviously come from numpy* So what do I miss to make it work? Thanks Xavier

1 0

mask one array using another array
by questions anon 22 Nov '11

22 Nov '11

I am trying to mask one array using another array. I have created a masked array using mask=MA.masked_equal(myarray, 0), that looks something like: [1 - - 1, 1 1 - 1, 1 1 1 1, - 1 - 1] I have an array of values that I want to mask whereever my mask has a a '-'. how do I do this? I have looked at http://www.cawcr.gov.au/bmrc/climdyn/staff/lih/pubs/docs/masks.pdf but the command: d = array(a, mask=c.mask() results in this error: TypeError: 'numpy.ndarray' object is not callable I basically want to do exactly what that article does in that equation. Any feedback will be greatly appreciated.

2 4

subclassing ndarray
by Chris.Barker 22 Nov '11

22 Nov '11

Hi folks, I'm working on a "ragged array" class -- an array that can store and work with what can be considered tabular data, with the rows of different lengths: """ ragged_array A "ragged" array class -- build on numpy The idea is to be able to store data that is essentially 2-d, but each row is an arbitrary length, like: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 ... At the moment, my implementation (see enclosed) stores the data in a 1-d numpy array as an attribute, and also an index array that stores the indexes into the rows. This is working fine. However, I'd like to have it support any of the usual numpy operations that make sense for a ragged array: arr.sum() arr *= a_scalar arr * a_scalar etc, etc, etc. So I thought maybe I'd do a subclass, instead of having the data array an attribute of the class. But I can't figure out how to solve the indexing problem: I want to re-map indexing, so that: arr[i] returns the ith "row": In [2]: ra = ragged_array([(1,2), (3,4,5), (6,7)]) In [4]: print ra ragged array: [1 2] [3 4 5] [6 7] In [5]: ra[1] Out[5]: array([3, 4, 5]) I'm currently doing (error checking removed): def __getitem__(self,index): """ returns a numpy array of one row. """ row = (self._data_array[self._index_array[index]:self._index_array[index+1]] ) return row But if I subclass ndarray, then self._data_array becomes jsut plain "self", and I've overloaded indexing (and slicing), so I don't know how I could index into the "flat" array to get the subset of the array I need. any ideas? Other comments about the class would be great, too. -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 0

PyPy guys deserve some help on micronumpy from you, numpy gurus!
by Valery Khamenya 21 Nov '11

21 Nov '11

Hi (replying please Cc to me) There is a module micronumpy that appeared at PyPy source tree: https://bitbucket.org/pypy/pypy/src/dfae5033127e/pypy/module/micronumpy/ The great contributions of Justin Peel and Ilya Osadchiy to micronumpy module revive step-by-step the functionality of numpy. It would be great if some of numpy-gurus could jump in to assist, contribute some code and also, perhaps, guide a bit where the things go deeply in numpy nature. For those who don't know yet much about PyPy: PyPy is a fast implementation of Python 2.7. As a rule of thumb, PyPy is currently about 4x times faster than CPython. Certain benchmarks taken from the real life show 20x speed-up and more. The successes of PyPy performance are very remarkable http://speed.pypy.org/ best regards -- Valery A.Khamenya

3 4

Build warnings
by Mads Ipsen 20 Nov '11

20 Nov '11

Hi, I am currently building numpy 1.6.1 on Ubuntu 11.04 using gcc 4.4.5. Some compiler warnings are printed during the compilation stage. Is it of any interest to post these somewhere in order for some of the core developers to fix these? Best regards, Mads -- +-----------------------------------------------------+ | Mads Ipsen | +----------------------+------------------------------+ | Gåsebæksvej 7, 4. tv | | | DK-2500 Valby | phone: +45-29716388 | | Denmark | email: mads.ipsen(a)gmail.com | +----------------------+------------------------------+

1 0

Vectorizing!!!!
by Solimyr 19 Nov '11

19 Nov '11

Hi all!!!I'm pretty new here!! I'm just use my free time to learn something more about python....I just discovered a new word named 'Vectorizing'.... I'm going to explain my question...I have a matrix (no care about the size) and I want made some mathematical operations like mean,standard deviation,variance,ecc... I post my result so can be usefull for all the newbie to understand the vectorizing mean: Mean using a 3x3 window and 2 FOR cicle (a=matrix of interest, b same size then a but filled with zeros at the start): for i in range(1,row-1): for j in range(1,col-1): b[i][j]= numpy.mean([a[i-1][j-1],a[i][j-1],a[i+1][j-1],a[i-1][j],a[i][j],a[i+1][j],a[i-1][j+1],a[i][j+1],a[i+1][j+1]]) Very disappointing in term of time..This is with the vectorizing(a=matrix of interest, c same size then a but filled with zeros at the start): c[1:row-1,1:col-1]=(a[0:row-2,0:col-2]+a[0:row-2,1:col-1]+a[2:row,2:col]+a[1:row-1,1:col-1]+a[1:row-1,0:col-2]+a[0:row-2,2:col]+a[1:row-1,2:col]+a[2:row,0:col-2]+a[2:row,1:col-1])/9 I have seen that I get a big advantage!!!But my question is: If I want to calculate the variance in a 3x3 window, can I use the vectorizing method and 'numpy.var' or I must explain the variance formula? I don't know if the question is understandable! I have thought something like: c[1:row-1,1:col-1]=numpy.var(a[0:row-2,0:col-2]+a[0:row-2,1:col-1]+a[2:row,2:col]+a[1:row-1,1:col-1]+a[1:row-1,0:col-2]+a[0:row-2,2:col]+a[1:row-1,2:col]+a[2:row,0:col-2]+a[2:row,1:col-1]) But doesn't work because I think that numpy.var work over all the matrix and not only in the 3x3 window!!Is it correct??? Thanks in advance for any answers!!! Solimyr -- View this message in context: http://old.nabble.com/Vectorizing%21%21%21%21-tp32868162p32868162.html Sent from the Numpy-discussion mailing list archive at Nabble.com.

2 1

matrix product in parallel?
by Nicolas Bock 19 Nov '11

19 Nov '11

Hello list, is it possible to get numpy to do a matrix product in parallel? I presume that the multiply in the back-end is really done by dgemm() or equivalent, so is it possible to use a parallel dgemm() there? Thanks, nick

1 1