Mailman 3 January 2010 - NumPy-Discussion

Stupid question (at least coming from me it is)
by David Goldsmith Jan. 9, 2010

Jan. 9, 2010

So, to get the new numpy.polynomial "sub-package," one has to update to 1.4 (or is there a 1.3.x that has it)? Thanks! DG PS: my pressing need (another stupid question, at least coming from me): chebyshev.chebdomain = [0,1] or [-1,1]? Thanks again!

2 4

performance matrix multiplication vs. matlab
by Xue (Sue) Yang Jan. 8, 2010

Jan. 8, 2010

Hi David, Thank you for the reply which is useful. I also tried to Install numpy with intel mkl 9.1 I still used gfortran for numpy installation as intel mkl 9.1 supports gnu compiler. I only uncomment these lines for site.cfg in site.cfg.example [mkl] library_dirs = /usr/physics/intel/mkl/lib/32 include_dirs = /usr/physics/intel/mkl/include lapack_libs = mkl_lapack then I tested the numpy with > python >>import numpy >>a = numpy.random.randn(6000, 6000) >>numpy.dot(a, a) This time, only one cpu was used. Does it mean that our installed intel mkl 9.1 is not threaded? I don't think so. We have used it for openMP parallelization for quite a while. Thanks! Sue

5 5

1.4.0 installer fails on OSX 10.6.2
by neil weisenfeld Jan. 8, 2010

Jan. 8, 2010

Hi all, I'm having an odd problem with the package installer for numpy 1.4.0. It complains: numpy 1.4.0 can't be installed on this disk. numpy requires System Python 2.6 to install. I'm running a stock system with a stock python, so I'm not sure why the test is failing. Any ideas how to debug this? Some info: weisen@Neil-Weisenfeld-MacBook-Pro:~ [507]$ which python /usr/bin/python weisen@Neil-Weisenfeld-MacBook-Pro:~ [508]$ python Python 2.6.1 (r261:67515, Jul 7 2009, 23:51:51) [GCC 4.2.1 (Apple Inc. build 5646)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> weisen@Neil-Weisenfeld-MacBook-Pro:~ [509]$ ls -l /System/Library/Frameworks/Python.framework/Versions/Current lrwxr-xr-x 1 root wheel 3 Sep 22 17:49 /System/Library/Frameworks/Python.framework/Versions/Current@ -> 2.6 Thanks, Neil

9 24

FIY: a (new ?) practical profiling tool on linux
by David Cournapeau Jan. 8, 2010

Jan. 8, 2010

Hi, I don't know if many people are aware of it, but I have recently discovered perf, a tool available from the kernel sources. It is extremely simple to use, and very useful when looking at numpy/scipy perf issues in compiled code. For example, I can get this kind of results for looking at the numpy neighborhood iterator performance in one simple command, without special compilation flags: 44.69% python /home/david/local/stow/scipy.git/lib/python2.6/site-packages/scipy/signal/sigtools.so [.] _imp_correlate_nd_double 39.47% python /home/david/local/stow/numpy-1.4.0/lib/python2.6/site-packages/numpy/core/multiarray.so [.] get_ptr_constant 9.98% python /home/david/local/stow/numpy-1.4.0/lib/python2.6/site-packages/numpy/core/multiarray.so [.] get_ptr_simple 0.65% python /usr/bin/python2.6 [.] 0x0000000012b8a0 0.40% python /usr/bin/python2.6 [.] 0x000000000a6662 0.37% python /usr/bin/python2.6 [.] 0x0000000004c10d 0.32% python /usr/bin/python2.6 [.] PyEval_EvalFrameEx 0.15% python [kernel] [k] __d_lookup 0.14% python /lib/libc-2.10.1.so [.] _int_malloc 0.12% python /usr/bin/python2.6 [.] 0x0000000004f90e 0.10% python [kernel] [k] __link_path_walk 0.09% python /usr/bin/python2.6 [.] PyObject_Malloc 0.09% python /lib/ld-2.10.1.so [.] do_lookup_x 0.09% python /lib/libc-2.10.1.so [.] __GI_memcpy 0.08% python [kernel] [k] __ticket_spin_lock 0.07% python /usr/bin/python2.6 [.] PyParser_AddToken And even cooler, annotated sources: ------------------------------------------------ Percent | Source code & Disassembly of multiarray.so ------------------------------------------------ : : : : Disassembly of section .text: : : 000000000001d8a0 <get_ptr_constant>: : _coordinates[c] = bd; : : /* set the dataptr from its current coordinates */ : static char* : get_ptr_constant(PyArrayIterObject* _iter, npy_intp *coordinates) : { 15.69 : 1d8a0: 48 81 ec 08 01 00 00 sub $0x108,%rsp : int i; : npy_intp bd, _coordinates[NPY_MAXDIMS]; : PyArrayNeighborhoodIterObject *niter = (PyArrayNeighborhoodIterObject*)_iter; : PyArrayIterObject *p = niter->_internal_iter; : : for(i = 0; i < niter->nd; ++i) { 0.02 : 1d8a7: 48 83 bf 48 0a 00 00 cmpq $0x0,0xa48(%rdi) 0.00 : 1d8ae: 00 : get_ptr_constant(PyArrayIterObject* _iter, npy_intp *coordinates) : { : int i; : npy_intp bd, _coordinates[NPY_MAXDIMS]; : PyArrayNeighborhoodIterObject *niter = (PyArrayNeighborhoodIterObject*)_iter; : PyArrayIterObject *p = niter->_internal_iter; 0.01 : 1d8af: 48 8b 87 50 0b 00 00 mov 0xb50(%rdi),%rax : : for(i = 0; i < niter->nd; ++i) { 7.92 : 1d8b6: 7e 64 jle 1d91c <get_ptr_constant+0x7c> : _INF_SET_PTR(i) 0.01 : 1d8b8: 48 8b 0e mov (%rsi),%rcx 0.00 : 1d8bb: 48 03 48 28 add 0x28(%rax),%rcx 0.03 : 1d8bf: 48 3b 88 40 07 00 00 cmp 0x740(%rax),%rcx 7.97 : 1d8c6: 7c 68 jl 1d930 <get_ptr_constant+0x90> 0.02 : 1d8c8: 45 31 c9 xor %r9d,%r9d 0.00 : 1d8cb: 31 d2 xor %edx,%edx 0.00 : 1d8cd: 48 3b 88 48 07 00 00 cmp 0x748(%rax),%rcx 7.75 : 1d8d4: 7e 32 jle 1d908 <get_ptr_constant+0x68> 0.00 : 1d8d6: eb 58 jmp 1d930 <get_ptr_constant+0x90> 0.00 : 1d8d8: 0f 1f 84 00 00 00 00 nopl 0x0(%rax,%rax,1) 0.00 : 1d8df: 00 7.68 : 1d8e0: 4c 8d 42 74 lea 0x74(%rdx),%r8 0.00 : 1d8e4: 48 8b 0c d6 mov (%rsi,%rdx,8),%rcx 0.00 : 1d8e8: 48 03 4c d0 28 add 0x28(%rax,%rdx,8),%rcx 0.00 : 1d8ed: 49 c1 e0 04 shl $0x4,%r8 7.89 : 1d8f1: 49 3b 0c 00 cmp (%r8,%rax,1),%rcx 0.00 : 1d8f5: 7c 39 jl 1d930 <get_ptr_constant+0x90> 0.01 : 1d8f7: 49 89 d0 mov %rdx,%r8 0.11 : 1d8fa: 49 c1 e0 04 shl $0x4,%r8 7.18 : 1d8fe: 4a 3b 8c 00 48 07 00 cmp 0x748(%rax,%r8,1),%rcx 0.00 : 1d905: 00 0.09 : 1d906: 7f 28 jg 1d930 <get_ptr_constant+0x90> : int i; : npy_intp bd, _coordinates[NPY_MAXDIMS]; : PyArrayNeighborhoodIterObject *niter = (PyArrayNeighborhoodIterObject*)_iter; : PyArrayIterObject *p = niter->_internal_iter; : It works for C and Fortran, BTW, cheers, David

1 0

Numpy & MKL
by Xue (Sue) Yang Jan. 8, 2010

Jan. 8, 2010

This is what I had (when I built numpy, I chose gnu compilers instead of intel compilers), >>> numpy.show_config() lapack_opt_info: libraries = ['mkl_lapack', 'mkl', 'vml', 'guide', 'pthread'] library_dirs = ['/usr/physics/intel/mkl/lib/32'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/usr/physics/intel/mkl/include'] blas_opt_info: libraries = ['mkl', 'vml', 'guide', 'pthread'] library_dirs = ['/usr/physics/intel/mkl/lib/32'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/usr/physics/intel/mkl/include'] lapack_mkl_info: libraries = ['mkl_lapack', 'mkl', 'vml', 'guide', 'pthread'] library_dirs = ['/usr/physics/intel/mkl/lib/32'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/usr/physics/intel/mkl/include'] blas_mkl_info: libraries = ['mkl', 'vml', 'guide', 'pthread'] library_dirs = ['/usr/physics/intel/mkl/lib/32'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/usr/physics/intel/mkl/include'] mkl_info: libraries = ['mkl', 'vml', 'guide', 'pthread'] library_dirs = ['/usr/physics/intel/mkl/lib/32'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/usr/physics/intel/mkl/include'] Thanks! Sue >> Do I need any specifications when I run numpy with intel MKL (MKL9.1)? >> numpy developers would be able to answer this question? >Are you sure you've compiled against MKL properly? What is printed by >numpy.show_config()? >David

2 1

Numpy & MKL
by Xue (Sue) Yang Jan. 8, 2010

Jan. 8, 2010

I understand that intel mkl uses openMP parallel model. Therefore I set environment variable >>os.environ['OMP_NUM_THREADS'] = '4' With same test example, however, still one cpu is used. Do I need any specifications when I run numpy with intel MKL (MKL9.1)? numpy developers would be able to answer this question? I changed the name of numpy-discussion thread to "Numpy & MKL" attempting to draw attentions from wide range of readers. Thanks! Sue On Thu, Jan 7, 2010 at 11:20 AM, Xue (Sue) Yang <x.y...(a)physics.usyd.edu.au> wrote: > This time, only one cpu was used. Does it mean that our installed intel mkl > 9.1 is not threaded? You would have to consult the MKL documentation - I believe you can control how many threads are used from an environment variable. Also, the exact build commands depend on the version of the MKL, as its libraries often change between versions. David > Thank you for the reply which is useful. > > I also tried to Install numpy with intel mkl 9.1 > I still used gfortran for numpy installation as intel mkl 9.1 supports > gnu compiler. > > I only uncomment these lines for site.cfg in site.cfg.example > > [mkl] > library_dirs = /usr/physics/intel/mkl/lib/32 > include_dirs = /usr/physics/intel/mkl/include > lapack_libs = mkl_lapack > > then I tested the numpy with > > > python > >>import numpy > >>a = numpy.random.randn(6000, 6000) > >>numpy.dot(a, a) > > This time, only one cpu was used. Does it mean that our installed > intel mkl 9.1 is not threaded? > I don't think so. We have used it for openMP parallelization for quite > a while. > > Thanks! > > Sue

2 1

cPickle/unPickle across archs
by James Mazer Jan. 7, 2010

Jan. 7, 2010

Hi, I've got a some Numeric arrays that were created without an explicit byte size in the initial declaration and pickled. Something like this: >>> cPickle.write(array(ones((3,3,)), 'f'), open('foo.pic', 'w')) as opposed to: >>> cPickle.write(array(ones((3,3,)), Float32), open('foo.pic', 'w')) This works as long as the word size doesn't change between the reading and writing machines. The data were generated under a 32bit linux kernel and now I'm trying to read them under a 64bit kernel, so the word size has changed and Numeric assumes that the 'f' type is the NATIVE float and 'l' type is the NATIVE long) and dies miserable when the native types don't match the actual types (which defeats the whole point of pickling, to some extent -- I thought that cPickle.save/load were "ensured" to be invertable...) I've got terrabytes of data that need to be read by both 32bit and 64bit machines (and it's not really feasible to scan all the files into new structures with explict types on a 32bit machine). Anybody have hints for addressing this problem? I found similar questions, but no answers, so I'm not completely alone iwth this problem. Thanks, /jamie -- James Mazer Department of Neurobiology Yale School of Medicine phone: 203-737-5853 fax: 203-785-5263

2 1

Repeated dot products
by T J Jan. 7, 2010

Jan. 7, 2010

Hi, Suppose I have an array of shape: (n, k, k). In this case, I have n k-by-k matrices. My goal is to compute the product of a (potentially large) user-specified selection (with replacement) of these matrices. For example, x = [0,1,2,1,3,3,2,1,3,2,1,5,3,2,3,5,2,5,3,2,1,3,5,6] says that I want to take the 0th matrix and dot it with the 1st matrix and dot that product with the 2nd matrix and dot that product with the 1st matrix again and so on... Essentially, I am looking for efficient ways of doing this. It seems like what I *need* is for dot to be a ufunc with a reduce() operator. Then I would construct an array of the matrices, as specified by the input x. For now, I am using a python loop and this is unbearable. >>> prod = np.eye(k) >>> for i in x: ... prod = dot(prod, matrices[i]) ... Is there a better way?

5 4

Re: [Numpy-discussion] performance matrix multiplication vs. matlab
by David Cournapeau Jan. 7, 2010

Jan. 7, 2010

On Thu, Jan 7, 2010 at 11:20 AM, Xue (Sue) Yang <x.yang(a)physics.usyd.edu.au> wrote: > This time, only one cpu was used. Does it mean that our installed intel mkl > 9.1 is not threaded? You would have to consult the MKL documentation - I believe you can control how many threads are used from an environment variable. Also, the exact build commands depend on the version of the MKL, as its libraries often change between versions. David

1 0

extracting data from ODF files
by Manuel Wittchen Jan. 6, 2010

Jan. 6, 2010

Hi, is there a (simple) solution to extract data from OpenDocument files (espacially OpenOffice.org Calc files) into a Numpy Array? At the moment I copy the colums from OO.org Calc manually into a tab-separatet Plaintext file which is quite annoying. Regards, Manuel Wittchen

5 4