Mailman 3 January 2007 - NumPy-Discussion

Fast vector multiplication
by Mark Janikas Jan. 20, 2007

Jan. 20, 2007

Hello all, I am trying to figure out the most efficient way to get the sum of the product of two vectors where id != id. E.g.: X = array([1,2,3]) Y = array([1,2,3]) Z = (1*2) + (1*3) + (2*1) + (2*3) + (3*1) + (3*2) = 22 I could obviously do this with loops, but I was wondering if there is a version of multiply that could be used... or a form of vectorize. Any ideas would be greatly appreciated. Mark Janikas Product Engineer ESRI, Geoprocessing 380 New York St. Redlands, CA 92373 909-793-2853 (2563) mjanikas(a)esri.com

3 3

concatenate a tuple of lists 10x slower in numpy 1.0.1
by Martin Spacek Jan. 20, 2007

Jan. 20, 2007

Hello, I just upgraded from numpy 1.0b5 to 1.0.1, and I noticed that a part of my code that was using concatenate() was suddenly far slower. I downgraded to 1.0, and the slowdown disappeared. Here's the code and the profiler results for 1.0 and 1.0.1: >>> import numpy as np >>> np.version.version '1.0' >>> a = [] >>> for i in range(10): ... a.append([np.random.rand(100000)]) # make nested list of arrays ... >>> import profile >>> profile.run('np.concatenate(tuple(a))') # concatenate tuple of lists # gives a 10 x 100000 array 4 function calls in 0.046 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.045 0.045 0.045 0.045 :0(concatenate) 1 0.000 0.000 0.000 0.000 :0(setprofile) 1 0.001 0.001 0.046 0.046 <string>:1(?) 1 0.000 0.000 0.046 0.046 profile:0(np.concatenate(tuple(a))) 0 0.000 0.000 profile:0(profiler) >>> import numpy as np >>> np.version.version '1.0.1' >>> a = [] >>> for i in range(10): ... a.append([np.random.rand(100000)]) ... >>> import profile >>> profile.run('np.concatenate(tuple(a))') 4 function calls in 0.532 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.531 0.531 0.531 0.531 :0(concatenate) 1 0.000 0.000 0.000 0.000 :0(setprofile) 1 0.001 0.001 0.532 0.532 <string>:1(?) 1 0.000 0.000 0.532 0.532 profile:0(np.concatenate(tuple(a))) 0 0.000 0.000 profile:0(profiler) Going from numpy 1.0 to 1.0.1, there's a slowdown of over 10x. In retrospect, I'm doing this in a weird way. If I get rid of the tuple of lists, replace it with a flat list and use reshape instead, it's much faster and gives me the same 10 x 100000 resulting array: >>> b = [] >>> for i in range(10): ... b.append(np.random.rand(100000)) ... >>> profile.run('np.concatenate(b).reshape(10, 100000)') 5 function calls in 0.023 CPU seconds Ordered by: standard name ncalls tottime percall cumtime percall filename:lineno(function) 1 0.021 0.021 0.021 0.021 :0(concatenate) 1 0.000 0.000 0.000 0.000 :0(reshape) 1 0.000 0.000 0.000 0.000 :0(setprofile) 1 0.001 0.001 0.022 0.022 <string>:1(?) 1 0.000 0.000 0.023 0.023 profile:0(np.concatenate(b).reshape(10, 100000)) 0 0.000 0.000 profile:0(profiler) The reshape method is equally fast for both 1.0 and 1.0.1. Still, I thought it prudent to bring up the slowdown with the tuple of lists method. Is this issue known? I ran these tests in Python 2.4.4 in a Windows console. I use the win32 py24 binaries. Cheers, Martin

2 1

Re: [Numpy-discussion] svn numpy did not build on os x
by Srinath Vadlamani Jan. 20, 2007

Jan. 20, 2007

------------------------------ Message: 5 Date: Fri, 19 Jan 2007 17:29:53 -0700 From: "Brian Granger" <ellisonbg.net(a)gmail.com> Subject: Re: [Numpy-discussion] svn numpy did not build on os x To: "Discussion of Numerical Python" <numpy-discussion(a)scipy.org> Message-ID: <6ce0ac130701191629y26524f93vae2146c20ecd500a(a)mail.gmail.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed I just did a clean checkout of numpy a few minutes ago (Intel mac book, Python 2.4.3 universal) and a simple python setup.py build worked fine. What Python are you using (where did you get it)? Is there something wierd going on with your gcc? Brian This is what I'm using: Python 2.4.4 (#1, Oct 18 2006, 10:34:39) [GCC 4.0.1 (Apple Computer, Inc. build 5341)] on darwin I'll try a new check out and try it again. I don't know where I got python...where should I get it from? Srinath V. -- ========================================================== Srinath Vadlamani srinath(a)txcorp.com Tech-X Corp. o:303-996-2034 5621 Arapahoe Ave. Suite A f:303-448-7756 Boulder, CO 80303 ==========================================================

2 1

how to recognize a numpy-scalar type at the C level and how to convert
by Sebastian Haase Jan. 20, 2007

Jan. 20, 2007

Hi! After converting to numpy my SWIG-wrapped code runs a bunch of error of this type: TypeError: argument number 9: a 'float' is expected, 'numpy.float32(-2.10786056519)' is received This comes of course from the fact that numpy introduced all these new scalar types (addressing the shortcoming of the (few) scalar type provided by standard python). What would be an easy way to "recognize" such a scalar type in C-API numpy code and can I extract the C-scalar value from it ? Thanks for any hints, Sebastian Haase

3 2

svn numpy did not build on os x
by Srinath Vadlamani Jan. 20, 2007

Jan. 20, 2007

I just did an svn up (1/19/07) and tried building numpy with no success. It built fine yesterday (1/18/07). This is on OS X tiger. Attached is the error output. I hope that attachments work with the mailing list. -- ========================================================== Srinath Vadlamani srinath(a)txcorp.com Tech-X Corp. o:303-996-2034 5621 Arapahoe Ave. Suite A f:303-448-7756 Boulder, CO 80303 ========================================================== Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' No module named __svn_version__ F2PY Version 2_3513 blas_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-faltivec', '-I/System/Library/Frameworks/vecLib.framework/Headers'] lapack_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-faltivec'] running build running config_fc running build_src building py_modules sources building extension "numpy.core.multiarray" sources adding 'build/src.macosx-10.3-fat-2.4/numpy/core/config.h' to sources. executing numpy/core/code_generators/generate_array_api.py adding 'build/src.macosx-10.3-fat-2.4/numpy/core/__multiarray_api.h' to sources. adding 'build/src.macosx-10.3-fat-2.4/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files= ['build/src.macosx-10.3-fat-2.4/numpy/core/src/scalartypes.inc', 'build/src.macosx-10.3-fat-2.4/numpy/core/src/arraytypes.inc', 'build/src.macosx-10.3-fat-2.4/numpy/core/config.h', 'build/src.macosx-10.3-fat-2.4/numpy/core/__multiarray_api.h'] building extension "numpy.core.umath" sources adding 'build/src.macosx-10.3-fat-2.4/numpy/core/config.h' to sources. executing numpy/core/code_generators/generate_ufunc_api.py adding 'build/src.macosx-10.3-fat-2.4/numpy/core/__ufunc_api.h' to sources. adding 'build/src.macosx-10.3-fat-2.4/numpy/core/src' to include_dirs. numpy.core - nothing done with h_files= ['build/src.macosx-10.3-fat-2.4/numpy/core/src/scalartypes.inc', 'build/src.macosx-10.3-fat-2.4/numpy/core/src/arraytypes.inc', 'build/src.macosx-10.3-fat-2.4/numpy/core/config.h', 'build/src.macosx-10.3-fat-2.4/numpy/core/__ufunc_api.h'] building extension "numpy.core._sort" sources adding 'build/src.macosx-10.3-fat-2.4/numpy/core/config.h' to sources. executing numpy/core/code_generators/generate_array_api.py adding 'build/src.macosx-10.3-fat-2.4/numpy/core/__multiarray_api.h' to sources. numpy.core - nothing done with h_files= ['build/src.macosx-10.3-fat-2.4/numpy/core/config.h', 'build/src.macosx-10.3-fat-2.4/numpy/core/__multiarray_api.h'] building extension "numpy.core.scalarmath" sources adding 'build/src.macosx-10.3-fat-2.4/numpy/core/config.h' to sources. executing numpy/core/code_generators/generate_array_api.py adding 'build/src.macosx-10.3-fat-2.4/numpy/core/__multiarray_api.h' to sources. executing numpy/core/code_generators/generate_ufunc_api.py adding 'build/src.macosx-10.3-fat-2.4/numpy/core/__ufunc_api.h' to sources. numpy.core - nothing done with h_files= ['build/src.macosx-10.3-fat-2.4/numpy/core/config.h', 'build/src.macosx-10.3-fat-2.4/numpy/core/__multiarray_api.h', 'build/src.macosx-10.3-fat-2.4/numpy/core/__ufunc_api.h'] building extension "numpy.core._dotblas" sources adding 'numpy/core/blasdot/_dotblas.c' to sources. building extension "numpy.lib._compiled_base" sources building extension "numpy.numarray._capi" sources building extension "numpy.fft.fftpack_lite" sources building extension "numpy.linalg.lapack_lite" sources adding 'numpy/linalg/lapack_litemodule.c' to sources. building extension "numpy.random.mtrand" sources customize NAGFCompiler customize AbsoftFCompiler customize IbmFCompiler Could not locate executable f77 Could not locate executable f95 customize GnuFCompiler customize GnuFCompiler customize GnuFCompiler using config C compiler: gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 compile options: '-Inumpy/core/src -Inumpy/core/include -I/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4 -c' gcc: _configtest.c _configtest.c:7:2: error: #error No _WIN32 _configtest.c:7:2: error: #error No _WIN32 lipo: can't figure out the architecture type of: /var/tmp//ccS3IyWE.out _configtest.c:7:2: error: #error No _WIN32 _configtest.c:7:2: error: #error No _WIN32 lipo: can't figure out the architecture type of: /var/tmp//ccS3IyWE.out failure. removing: _configtest.c _configtest.o building data_files sources running build_py copying build/src.macosx-10.3-fat-2.4/numpy/__config__.py -> build/lib.macosx-10.3-fat-2.4/numpy copying build/src.macosx-10.3-fat-2.4/numpy/distutils/__config__.py -> build/lib.macosx-10.3-fat-2.4/numpy/distutils copying numpy/f2py/__svn_version__.py -> build/lib.macosx-10.3-fat-2.4/numpy/f2py copying numpy/core/__svn_version__.py -> build/lib.macosx-10.3-fat-2.4/numpy/core running build_ext customize UnixCCompiler customize UnixCCompiler using build_ext building 'numpy.core.multiarray' extension compiling C sources C compiler: gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 compile options: '-Ibuild/src.macosx-10.3-fat-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.macosx-10.3-fat-2.4/numpy/core -Inumpy/core/src -Inumpy/core/include -I/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4 -c' gcc: numpy/core/src/multiarraymodule.c /var/tmp//ccDQNZ7Y.s:18606:no such 386 instruction: `fisttp' /var/tmp//ccDQNZ7Y.s:18649:no such 386 instruction: `fisttp' /var/tmp//ccDQNZ7Y.s:18690:no such 386 instruction: `fisttp' /var/tmp//ccDQNZ7Y.s:20532:no such 386 instruction: `fisttp' /var/tmp//ccDQNZ7Y.s:20575:no such 386 instruction: `fisttp' /var/tmp//ccDQNZ7Y.s:20616:no such 386 instruction: `fisttp' lipo: can't open input file: /var/tmp//cco1Zdpm.out (No such file or directory) /var/tmp//ccDQNZ7Y.s:18606:no such 386 instruction: `fisttp' /var/tmp//ccDQNZ7Y.s:18649:no such 386 instruction: `fisttp' /var/tmp//ccDQNZ7Y.s:18690:no such 386 instruction: `fisttp' /var/tmp//ccDQNZ7Y.s:20532:no such 386 instruction: `fisttp' /var/tmp//ccDQNZ7Y.s:20575:no such 386 instruction: `fisttp' /var/tmp//ccDQNZ7Y.s:20616:no such 386 instruction: `fisttp' lipo: can't open input file: /var/tmp//cco1Zdpm.out (No such file or directory) error: Command "gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -fno-common -dynamic -DNDEBUG -g -O3 -Ibuild/src.macosx-10.3-fat-2.4/numpy/core/src -Inumpy/core/include -Ibuild/src.macosx-10.3-fat-2.4/numpy/core -Inumpy/core/src -Inumpy/core/include -I/Library/Frameworks/Python.framework/Versions/2.4/include/python2.4 -c numpy/core/src/multiarraymodule.c -o build/temp.macosx-10.3-fat-2.4/numpy/core/src/multiarraymodule.o" failed with exit status 1

3 3

numpy.random.permutation bug?
by Emanuele Olivetti Jan. 18, 2007

Jan. 18, 2007

Look at this: ----------bug.py------- import numpy a=numpy.array([1,2]) b=a.sum() print type(b) c=numpy.random.permutation(b) ----------------------- If I run it (Python 2.5, numpy 1.0.1 on a Linux box) I get: ----------- #> python /tmp/bug.py <type 'numpy.int32'> Traceback (most recent call last): File "/tmp/bug.py", line 5, in <module> c=numpy.random.permutation(b) File "mtrand.pyx", line 1227, in mtrand.RandomState.permutation File "mtrand.pyx", line 1211, in mtrand.RandomState.shuffle TypeError: len() of unsized object ----------- permutation() likes 'int' and dislikes 'numpy.int32' integers :( Seems a bug. HTH, Emanuele

4 9

recompiling needed for binary module after numpy 1.0
by belinda thom Jan. 17, 2007

Jan. 17, 2007

Sebastian, I had the same problem awhile back; I'm curious---how'd you fix your mismatch (i.e. where'd you go to get code, did you run python setup?, ...). I realize these are very basic questions, but I've never installed anything from source (aside from using easy_install), so it would be nice to extend my very limited knowledge in this area. Thx, --b ============================ You are right again, of course ! Sorry for the noise - I should have just checked the date of my so file (which is August 15) At least I understood the "official numpy intention of version 1.0" right then - just checking ... Thanks, Sebastian. On 1/5/07, Robert Kern <robert.kern at gmail.com> wrote: > Sebastian Haase wrote: > > Hi! > > After I upgraded from numpy 1.0 to 1.0.1 > > I get an abort in a C-module: > > > > RuntimeError: module compiled against version 1000002 of C-API but this > > version of numpy is 1000009 > > Fatal Python error: numpy.core.multiarray failed to import... exiting. > > /opt/bin/priithonN: line 37: 1409 Aborted $PY $* > > > > I thought that within numpy 1.0 there was no recompile > > for external C-modules needed !? > > Ummm, the number was bumped before the 1.0 release. > > Bump of the version number: > http://projects.scipy.org/scipy/numpy/changeset/3361 > > Tagging the 1.0 release: > http://projects.scipy.org/scipy/numpy/changeset/3396/tags/1.0 > > Are you sure that you weren't upgrading from a beta instead of the 1.0 release? >

6 22

possible bug in recordarray assigning to a '4u4' field
by Sebastian Haase Jan. 17, 2007

Jan. 17, 2007

Hi, ProStr = N.rec.array(None, formats="u4,f4,u4,u4,4u4", names=('count','clock','InitDio', 'nDigital', 'nAnalog'), aligned=True, shape=1) ProStr[0] ['nAnalog'] = 1 I get this error message: ValueError: shape-mismatch on array construction ProStr ['nAnalog'] = 1 works fine. Surprisingly ProStr['nAnalog'] = [1,2,3,4] works. Could someone explain ? Thanks, Sebastian

1 0

ownership of numpy arrays
by josh kantor Jan. 17, 2007

Jan. 17, 2007

I am working on a project where I am doing some interfacing with numpy arrays at a C level. One thing that I want to do (using pyrex) is to call numpy functions, then directly access the underlying double * array of the resulting ndarray and operate on it in C/pyrex (I don't want to copy it). I want to ensure that the underlying double * wont be deallocated until I'm ready. I was going to take the resulting ndarray and set the ownership bit of the result to 0 and then I would be responsible for deallocating the memory when I was ready. Contrary to what I expected I noticed that it seems all numpy functions, for example numpy.linalg.pinv as well as numpy.linalg.eig return matrices that don't own their data. (If m is a numpy array) n=numpy.linalg.pinv(m) n.flags shows that n does not own its data. So I can't take their underlying data * with any certaintly that it won't be deallocated before I'm ready. If n doesn't own its data, who does. Is there a safe way to do what I want? Thanks, Josh _________________________________________________________________ The MSN Entertainment Guide to Golden Globes is here. Get all the scoop. http://tv.msn.com/tv/globes2007/?icid=nctagline2

3 2

weird numpy.any() behavior; bug, or 6-legged feature?
by Matthew Koichi Grimes Jan. 16, 2007

Jan. 16, 2007

Numpy's any() function gives unintuitive results when given a generator rather than a sequence: >>> import numpy as N >>> N.any( i < 0 for i in range(3) ) True If the generator is instead given as a list (using a list comprehension), then the expected answer is given: >>> N.any( [i < 0 for i in range(3)] ) False Python 2.5's built-in any() seems to deal fine with generators: >>> any( i < 0 for i in range(3) ) True My first guess at what's happening in numpy.any() is that the generator object itself is being interpreted as a "True", in the type-weak C sense of the word. Is this a bug? It seems like having to generate a whole list before calling any() defeats the whole appeal of using any() in the first place; namely that it'll exit on the first False returned by the generator, without going through the rest of the elements. -- Matt

2 1