Mailman 3 November 2007 - NumPy-Discussion

Numpy book and PyArray_CastToType
by Matthieu Brucher 12 Nov '07

12 Nov '07

Hi, I'm trying to use PyArray_CastToType(), but according to the book, there are two arguments, a PyArrayObject* and a PyArrayDescr*, but according to the header file, there are three arguments, an additional int. Does someone know its use ? Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher

2 3

Re: [Numpy-discussion] Problem with numpy.linalg.eig?
by Lou Pecora 12 Nov '07

12 Nov '07

Works fine on my computer (Mac OS X 10.4), Python 2.4. Runs in a second or so. -- Lou Pecora ---Peter wrote: Hi all, The following code calling numpy v1.0.4 fails to terminate on my machine, which was not the case with v1.0.3.1 from numpy import arange, float64 from numpy.linalg import eig a = arange(13*13, dtype = float64) a.shape = (13,13) a = a%17 eig(a) Regards, Peter __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com

1 0

weibull distribution has only one parameter?
by D.Hendriks (Dennis) 12 Nov '07

12 Nov '07

According to (for instance) http://en.wikipedia.org/wiki/Weibull_distribution the Weibull distribution has two parameters: lambda > 0 is the scale parameter (real) and k > 0 is the shape parameter (real). However, the numpy.random.weibull function has only a single 'a' parameter (except for the size parameter which indicates the size of the array to fill with values - this is NOT a parameter of the distribution itself). My question is how this 'a' parameter translates to the Weibull distribution as it 'normally' is and how to sample the distribution when I have the lambda and k parameters?

4 6

Compilation with Visual Studio 2003
by Matthieu Brucher 11 Nov '07

11 Nov '07

Hi, I want to add some support to ACML, but currently, I can't even compile numpy (Visual Studio and g77), the current SVN fails with : running build_ext No module named msvccompiler in numpy.distutils; trying from distutils customize MSVCCompiler customize MSVCCompiler using build_ext building 'numpy.core.multiarray' extension compiling C sources D:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.exe /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Ibuild\src.win32-2.5\numpy\core\src-Inumpy\core\include -Ibuild\src.win32-2.5\numpy\core -Inumpy\core\src -Inumpy\core\include -ID:\Pyth on25\include -ID:\Python25\PC /Tcnumpy\core\src\multiarraymodule.c /Fobuild\temp .win32-2.5\Release\numpy\core\src\multiarraymodule.obj multiarraymodule.c d:\Travail\numpy\numpy\core\src\arrayobject.c(662) : warning C4244: '=' : conver sion de 'npy_longlong' en 'long', perte possible de données numpy\core\src\multiarraymodule.c(7592) : error C2065: 'NPY_ALLOW_THREADS' : ide ntificateur non déclaré error: Command "D:\Program Files\Microsoft Visual Studio .NET 2003\Vc7\bin\cl.ex e /c /nologo /Ox /MD /W3 /GX /DNDEBUG -Ibuild\src.win32-2.5\numpy\core\src-Inum py\core\include -Ibuild\src.win32-2.5\numpy\core -Inumpy\core\src -Inumpy\core\i nclude -ID:\Python25\include -ID:\Python25\PC /Tcnumpy\core\src\multiarraymodule .c /Fobuild\temp.win32-2.5\Release\numpy\core\src\multiarraymodule.obj" failed w ith exit status 2 Is there something I forgot to add in the site.cfg ? Matthieu -- French PhD student Website : http://miles.developpez.com/ Blogs : http://matt.eifelle.com and http://blog.developpez.com/?blog=92 LinkedIn : http://www.linkedin.com/in/matthieubrucher

3 4

numpy 1.04 numpy.test() hang
by Geoffrey Zhu 10 Nov '07

10 Nov '07

Good morning. I just installed the Windows binary of numpy 1.04. When I ran numpy.test() in IDLE (the Python shell that comes with Python), the program hang (or at least is running for half an hour). I am using Windows XP, duel core intel CPU. Does anyone know what is going on? Thanks, Geoffrey

6 12

Unnecessarily bad performance of elementwise operators with Fortran-arrays
by Hans Meine 10 Nov '07

10 Nov '07

Hi! I wonder why simple elementwise operations like "a * 2" or "a + 1" are not performed in order of increasing memory addresses in order to exploit CPU caches etc. - as it is now, their speed drops by a factor of around 3 simply by transpose()ing. Similarly (but even less logical), copy() and even the constructor are affected (yes, I understand that copy() creates contiguous arrays, but shouldn't it respect/retain the order nevertheless?): ### constructor ### In [89]: %timeit -r 10 -n 1000000 numpy.ndarray((3,3,3)) 1000000 loops, best of 10: 1.19 s per loop In [90]: %timeit -r 10 -n 1000000 numpy.ndarray((3,3,3), order="f") 1000000 loops, best of 10: 2.19 s per loop ### copy 3x3x3 array ### In [85]: a = numpy.ndarray((3,3,3)) In [86]: %timeit -r 10 a.copy() 1000000 loops, best of 10: 1.14 s per loop In [87]: a = numpy.ndarray((3,3,3), order="f") In [88]: %timeit -r 10 -n 1000000 a.copy() 1000000 loops, best of 10: 3.39 s per loop ### copy 256x256x256 array ### In [74]: a = numpy.ndarray((256,256,256)) In [75]: %timeit -r 10 a.copy() 10 loops, best of 10: 119 ms per loop In [76]: a = numpy.ndarray((256,256,256), order="f") In [77]: %timeit -r 10 a.copy() 10 loops, best of 10: 274 ms per loop ### fill ### In [79]: a = numpy.ndarray((256,256,256)) In [80]: %timeit -r 10 a.fill(0) 10 loops, best of 10: 60.2 ms per loop In [81]: a = numpy.ndarray((256,256,256), order="f") In [82]: %timeit -r 10 a.fill(0) 10 loops, best of 10: 60.2 ms per loop ### power ### In [151]: a = numpy.ndarray((256,256,256)) In [152]: %timeit -r 10 a ** 2 10 loops, best of 10: 124 ms per loop In [153]: a = numpy.asfortranarray(a) In [154]: %timeit -r 10 a ** 2 10 loops, best of 10: 458 ms per loop ### addition ### In [160]: a = numpy.ndarray((256,256,256)) In [161]: %timeit -r 10 a + 1 10 loops, best of 10: 139 ms per loop In [162]: a = numpy.asfortranarray(a) In [163]: %timeit -r 10 a + 1 10 loops, best of 10: 465 ms per loop ### fft ### In [146]: %timeit -r 10 numpy.fft.fft(vol, axis=0) 10 loops, best of 10: 1.16 s per loop In [148]: %timeit -r 10 numpy.fft.fft(vol0, axis=2) 10 loops, best of 10: 1.16 s per loop In [149]: vol.flags Out[149]: C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : True WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [150]: vol0.flags Out[150]: C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False In [9]: %timeit -r 10 numpy.fft.fft(vol0, axis=0) 10 loops, best of 10: 939 ms per loop ### mean ### In [173]: %timeit -r 10 vol.mean() 10 loops, best of 10: 272 ms per loop In [174]: %timeit -r 10 vol0.mean() 10 loops, best of 10: 683 ms per loop ### max ### In [175]: %timeit -r 10 vol.max() 10 loops, best of 10: 63.8 ms per loop In [176]: %timeit -r 10 vol0.max() 10 loops, best of 10: 475 ms per loop ### min ### In [177]: %timeit -r 10 vol.min() 10 loops, best of 10: 63.8 ms per loop In [178]: %timeit -r 10 vol0.min() 10 loops, best of 10: 476 ms per loop ### rot90 ### In [10]: %timeit -r 10 numpy.rot90(vol) 100000 loops, best of 10: 6.97 s per loop In [12]: %timeit -r 10 numpy.rot90(vol0) 100000 loops, best of 10: 6.92 s per loop -- Ciao, / / /--/ / / ANS

8 30

NumPy 1.04, MKL 10.0, & Intel 10.1 icc & ifort
by rex 09 Nov '07

09 Nov '07

Build was successful after a change in distutils. Core 2 Duo, Debian Etch-32, Python 2.5, icc 10.1, ifort 10.1, & mkl 10.0. MKL & the compilers were installed to their default locations: /opt/intel/mkl/, /opt/intel/cc/, /opt/intel/fc/ Installation will not interfere with earlier versions. NumPy site.cfg for MKL 10.0 ========================================== [DEFAULT] library_dirs=/opt/intel/mkl/10.0.011/lib/32 include_dirs=/opt/intel/mkl/10.0.011/include [mkl] libraries=mkl,guide [lapack_info] libraries=mkl_lapack,mkl,guide ========================================== Changes in distutils: ========================================================= In MKL 9.1 & 10.0, mkl_lapack32 & mkl_lapack64 no longer exist. They were replaced by /32/mkl_lapack and /64/mkl_lapack. As a result, numpy-1.04/numpy/distutils/system_info.py needs to be changed: #lapack_libs = self.get_libs('lapack_libs',['mkl_lapack32','mkl_lapack64']) lapack_libs = self.get_libs('lapack_libs',['mkl_lapack']) If the above change is not made numpy builds, but without using MKL. In numpy-1.04/numpy/distutils/ccompiler.py change: #cc_exe = 'icc' #works, but is suboptimal cc_exe = 'icc -msse3 -fast' #adjust to suit your cpu ========================================================= Running the command below (all one line) from the numpy-1.04 directory python setup.py config --compiler=intel build_clib --compiler=intel build_ext --compiler=intel install > inst.log Gave this inst.log: ===================================================== F2PY Version 2_4412 blas_opt_info: blas_mkl_info: FOUND: libraries = ['mkl', 'pthread', 'mkl', 'guide'] library_dirs = ['/opt/intel/mkl/10.0.011/lib/32'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/mkl/10.0.011/include'] FOUND: libraries = ['mkl', 'pthread', 'mkl', 'guide'] library_dirs = ['/opt/intel/mkl/10.0.011/lib/32'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/mkl/10.0.011/include'] lapack_opt_info: lapack_mkl_info: mkl_info: FOUND: libraries = ['mkl', 'pthread', 'mkl', 'guide'] library_dirs = ['/opt/intel/mkl/10.0.011/lib/32'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/mkl/10.0.011/include'] FOUND: libraries = ['mkl_lapack', 'mkl', 'pthread', 'mkl', 'guide', 'mkl', 'guide'] library_dirs = ['/opt/intel/mkl/10.0.011/lib/32'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/mkl/10.0.011/include'] FOUND: libraries = ['mkl_lapack', 'mkl', 'pthread', 'mkl', 'guide', 'mkl', 'guide'] library_dirs = ['/opt/intel/mkl/10.0.011/lib/32'] define_macros = [('SCIPY_MKL_H', None)] include_dirs = ['/opt/intel/mkl/10.0.011/include'] running config running build_clib running build_ext running build_src building py_modules sources creating build [...] =========================================================== For reference, here are the respective lib/32 files for MKL 9.1 and 10.0: c2d0:/opt/intel/mkl/9.1/lib/32# ls libguide.a libmkl_gfortran.a libmkl_ias.so libmkl_p3.so libmkl_p4.so libmkl_vml_def.so libmkl_vml_p4p.so libguide.so libmkl_gfortran.so libmkl_lapack.a libmkl_p4m.so libmkl.so libmkl_vml_p3.so libmkl_vml_p4.so libmkl_def.so libmkl_ia32.a libmkl_lapack.so libmkl_p4p.so libmkl_solver.a libmkl_vml_p4m.so libvml.so c2d0:/opt/intel/mkl/10.0.011/lib/32# ls libguide.a libmkl_cdft_core.a libmkl_intel.a libmkl_p4.so libmkl_vml_ia.so libguide.so libmkl_core.a libmkl_intel.so libmkl_scalapack.a libmkl_vml_p3.so libiomp5.a libmkl_core.so libmkl_intel_thread.a libmkl_scalapack_core.a libmkl_vml_p4m2.so libiomp5.so libmkl_def.so libmkl_intel_thread.so libmkl_sequential.a libmkl_vml_p4m.s o libmkl_blacs.a libmkl_gf.a libmkl_lapack.a libmkl_sequential.so libmkl_vml_p4p.so libmkl_blacs_intelmpi20.a libmkl_gf.so libmkl_lapack.so libmkl.so libmkl_vml_p4.so libmkl_blacs_intelmpi.a libmkl_gnu_thread.a libmkl_p3.so libmkl_solver.a libmkl_blacs_openmpi.a libmkl_gnu_thread.so libmkl_p4m.so libmkl_solver_sequential.a libmkl_cdft.a libmkl_ia32.a libmkl_p4p.so libmkl_vml_def.so Thanks to all who have helped me with earlier versions. -rex

1 0

Suggestions on the best way to customize compilation flags
by David Cournapeau 09 Nov '07

09 Nov '07

Hi, I would like know the opinion from other people on the best way to customize compilation flags for building numpy. I am talking about the end-user point of view (e.g. I want to use the --optimize--so-aggressively-that-my-cpu-burns-into-flames flags, but without breaking build of python extensions because I forgot -fPIC). One of the goal of using scons within distutils was to separate the logic between compilation options which are necessary to make the binary work at all (basically, what are the options to build a python extension on supported platforms, how to build a dll for ctypes, etc...) and the ones which are 'secondary' (optimization, debug, warnings). I have started working in this direction, using compiler configuration objects, which are just dictionaries: http://projects.scipy.org/scipy/numpy/browser/branches/numpy.scons/numpy/di… The idea is that: - those configurations can be easily changed during the configure step of a package (for example, use sse unit instead of x87, etc...) - those configuration are not used directly when building projects: they are first copied to the actually used flags. The idea is that if necessary, you could request build environments without some warning flags (think pyrex/swig compilation), and only add them on user request. - those configurations are totally independent from the options necessary to build python extensions (those are not user accessible; I have not yet cleaned everything, but the goal is to have those flags set outside the extension builder logic, to avoid the distutils mess, which put and modifying flags everywhere deep down in the code). thanks, David

1 0

Warning or error on conversion from complex to float.
by Michael McNeil Forbes 08 Nov '07

08 Nov '07

Hi, Is it possible or easy to add a warning and/or error when array assignments are made that lose information? I just got caught with the following type of code: def f(): return numpy.array([1j,2.0]) x = numpy.empty((2,),dtype=float) x[:] = f() I am pre-allocating arrays for speed, but made a change to f() resulting in complex results. My code now silently ignores the imaginary part. It would have been very helpful if the assignment check to make sure that the conversion was not going to lose information (presumably with an assert so that performance is not affected outside of debugging). Any simple suggestions on how to avoid this problem in the future? How hard would it be to add such a check in numpy? Thanks, Michael.

2 1

Improving numpy.interp
by LB 08 Nov '07

08 Nov '07

I often need to make a linear interpolation for a single scalar value but this is not very simple with numpy.interp : >>> import numpy as n >>> n.__version__ '1.0.5.dev4420' >>> xp = n.arange(10) >>> yp = 2.5 + xp**2 -x >>> x = 3.2 >>> n.interp(x, xp, yp) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: object of too small depth for desired array So I use the following trick (which is really ugly) : >>> n.interp([x], xp, yp)[0] 7.7000000000000011 Did I miss an obvious way to do it ? If not, I'd be tempted to patch interp to let it accept scalar values as first argument as follow : >>> def interp(x, *args, **kwds): ... if type(x) in (float, int): ... return n.interp([x], *args, **kwds).item() ... else : ... return n.interp(x, *args, **kwds) ... >>> >>> interp(x, xp, yp) 7.7000000000000011 >>> interp([x,2*x], xp, yp) array([ 7.7, 38.5]) -- LB

1 0