Mailman 3 February 2011 - NumPy-Discussion

Python ctypes and OpenMP mystery
by Eric Carlson Feb. 17, 2011

Feb. 17, 2011

Hello All, I have been toying with OpenMP through f2py and ctypes. On the whole, the results of my efforts have been very encouraging. That said, some results are a bit perplexing. I have written identical routines that I run directly as a C-derived executable, and through ctypes as a shared library. I am running the tests on a dual-Xeon Ubuntu system with 12 cores and 24 threads. The C executable is SLIGHTLY faster than the ctypes at lower thread counts, but the C eventually has a speedup ratio of 12+, while the python caps off at 7.7, as shown below: threads C-speedup Python-speedup 1 1 1 2 2.07 1.98 3 3.1 2.96 4 4.11 3.93 5 4.97 4.75 6 5.94 5.54 7 6.83 6.53 8 7.78 7.3 9 8.68 7.68 10 9.62 7.42 11 10.38 7.51 12 10.44 7.26 13 7.19 6.04 14 7.7 5.73 15 8.27 6.03 16 8.81 6.29 17 9.37 6.55 18 9.9 6.67 19 10.36 6.9 20 10.98 7.01 21 11.45 6.97 22 11.92 7.1 23 12.2 7.08 These ratios are quite consistent from 100KB double arrays to 100MB double arrays, so I do not think it reflects a Python overhead issue. There is no question the routine is memory bandwidth constrained, and I feel lucky to squeeze the eventual 12+ ratio, but I am very perplexed as to why the performance of the Python-invoked routine seems to cap off. Does anyone have an explanation for the caps? Am I seeing some effect from ctypes, or the Python engine, or what? Cheers, Eric

2 3

Get the index of a comparison of two lists
by FRENK Andreas Feb. 16, 2011

Feb. 16, 2011

Hi, Thanks for the two solutions. Indeed, there are much faster than the brute force method using in. The setmember1d is deprecate din newer python releases, therefore I already switched to in1d(). (Thanks Josef.) Andreas

1 0

unwrap enhancement?
by Neal Becker Feb. 16, 2011

Feb. 16, 2011

I want to use unwrap to process a sequence of segments, such that the sequence appears continous. That is, unwrap maintains a state which represents a multiple of 2\pi. I want to continue that state with the next call to unwrap. Or in other words, I'd like to specify an initial phase for the start of unwrap. This could, of course, be done by concatenating the initial phase to the beginning of the segment, but that is not very efficient (memory allocation, copying, etc). It would be nice to just add an optional initial phase arg to unwrap. Thoughts?

1 0

SOLVED: f2py "target file xxx not generated"
by Thomas Ingeman-Nielsen Feb. 16, 2011

Feb. 16, 2011

Found a solution myself: I tried to compile the fib1.f file directly using the gfortran compiler, and found that indentation in the (fixed format) file was wrong. Correcting this took care of the "target file xxx not generated" error, but still an error resulted: ... ... error: Unable to find vcvarsall.bat A bit of googling pointed me to the following thread: http://code.google.com/p/rdflib/issues/detail?id=104#c4 which suggests creating a "distutils.cfg" file. After creating this file as described in the above reference, everything wraps/compiles smoothly, and the fib1 subroutine is callable from Python. Best regards, Thomas 2011/2/16 Thomas Ingeman-Nielsen <thomas(a)i-n.dk> > Hi, > > I'm trying to get started with f2py on a Windows 7 environment using the > Python(x,y) v 2.6.5.6 distribution. > I'm following the introductory example of the f2py userguide and try to > wrap the file FIB1.F using the command: > > f2py.py -c fib1.f -m fib1 > > from the windows command line. I get the following output: > > running build > running config_cc > unifing config_cc, config, build_clib, build_ext, build commands --compiler > opti > ons > running config_fc > unifing config_fc, config, build_clib, build_ext, build commands > --fcompiler opt > ions > running build_src > build_src > building extension "fib" sources > f2py options: [] > f2py:> > c:\users\thomas\appdata\local\temp\tmpamyxnx\src.win32-2.6\fibmodule.c > creating c:\users\thomas\appdata\local\temp\tmpamyxnx > creating c:\users\thomas\appdata\local\temp\tmpamyxnx\src.win32-2.6 > Reading fortran codes... > Reading file 'fib1.f' (format:fix,strict) > Post-processing... > Post-processing (stage 2)... > Building modules... > error: f2py target file > 'c:\\users\\thomas\\appdata\\local\\temp\\tmpamyxnx\\src > .win32-2.6\\fibmodule.c' not generated > > > > output of f2py.py -c --help-fcompiler: > > > Gnu95FCompiler instance properties: > archiver = ['c:\\GCC451\\bin\\gfortran.exe', '-cr'] > compile_switch = '-c' > compiler_f77 = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-ffixed- > form', '-fno-second-underscore', '-mno-cygwin', '-O3', > '- > funroll-loops'] > compiler_f90 = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', > '-fno-second- > underscore', '-mno-cygwin', '-O3', '-funroll-loops'] > compiler_fix = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-ffixed- > form', '-fno-second-underscore', '-mno-cygwin', > '-Wall', ' > -fno-second-underscore', '-mno-cygwin', '-O3', > '-funroll- > loops'] > libraries = ['gfortran'] > library_dirs = ['c:\\gcc451\\lib', 'c:\\gcc451\\lib\\gcc\\i686-pc- > mingw32\\4.5.1'] > linker_exe = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-mno- > cygwin', '-Wall', '-mno-cygwin'] > linker_so = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-mno- > cygwin', '-Wall', '-mno-cygwin', '-shared'] > object_switch = '-o ' > ranlib = ['c:\\GCC451\\bin\\gfortran.exe'] > version = LooseVersion ('4.5.1') > version_cmd = ['c:\\GCC451\\bin\\gfortran.exe', '--version', '-mno- > cygwin'] > Fortran compilers found: > --fcompiler=gnu95 GNU Fortran 95 compiler (4.5.1) > Compilers available for this platform, but not found: > --fcompiler=absoft Absoft Corp Fortran Compiler > --fcompiler=compaqv DIGITAL or Compaq Visual Fortran Compiler > --fcompiler=g95 G95 Fortran Compiler > --fcompiler=gnu GNU Fortran 77 compiler > --fcompiler=intelem Intel Fortran Compiler for EM64T-based apps > --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps > --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps > --fcompiler=intelvem Intel Visual Fortran Compiler for 64-bit apps > Compilers not available on this platform: > --fcompiler=compaq Compaq Fortran Compiler > --fcompiler=hpux HP Fortran 90 Compiler > --fcompiler=ibm IBM XL Fortran Compiler > --fcompiler=intel Intel Fortran Compiler for 32-bit apps > --fcompiler=intele Intel Fortran Compiler for Itanium apps > --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler > --fcompiler=mips MIPSpro Fortran Compiler > --fcompiler=nag NAGWare Fortran 95 Compiler > --fcompiler=none Fake Fortran compiler > --fcompiler=pg Portland Group Fortran Compiler > --fcompiler=sun Sun or Forte Fortran 95 Compiler > --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler > For compiler details, run 'config_fc --verbose' setup command. > > > > Running f2py without arguments result in the following version information: > > Version: 1 > numpy Version: 1.5.1 > Requires: Python 2.3 or higher. > License: NumPy license (see LICENSE.txt in the NumPy source code) > Copyright 1999 - 2005 Pearu Peterson all rights reserved. > http://cens.ioc.ee/projects/f2py2e/ > > > > I'm puzzled by this, I expect I have the compiler correctly installed, > since f2py recognizes it (and I am able to compile Fortran code using GCC > from Eclipse). > How about the f2py version number = 1? I was under the impression that > development had reached version >2. > > > Reinstalling Numpy from SourceForge numpy-1.5.1-win32-superpack-python2.6.exe > result in exactly the same output as above. > > Trying to install f2py using easy_install and then running f2py without > arguments result in the following error: > > Traceback (most recent call last): > File "C:\Python26\Scripts\f2py.py", line 5, in <module> > pkg_resources.run_script('f2py==2.45.241-1926', 'f2py.py') > File "C:\Python26\lib\site-packages\pkg_resources.py", line 489, in > run_script > > self.require(requires)[0].run_script(script_name, ns) > File "C:\Python26\lib\site-packages\pkg_resources.py", line 1207, in > run_scrip > t > execfile(script_filename, namespace, namespace) > File > "c:\python26\lib\site-packages\f2py-2.45.241_1926-py2.6-win32.egg\EGG-INF > O\scripts\f2py.py", line 3, in <module> > import f2py2e > File > "C:\Python26\lib\site-packages\f2py-2.45.241_1926-py2.6-win32.egg\f2py2e\ > __init__.py", line 10, in <module> > import f2py2e > File > "C:\Python26\lib\site-packages\f2py-2.45.241_1926-py2.6-win32.egg\f2py2e\ > f2py2e.py", line 26, in <module> > import crackfortran > File > "C:\Python26\lib\site-packages\f2py-2.45.241_1926-py2.6-win32.egg\f2py2e\ > crackfortran.py", line 1586 > as=b['args'] > ^ > SyntaxError: invalid syntax > > > > Any comments on how to adapt my setup to allow f2py to function correctly > would be much appreciated! > > Best regards, > > Thomas > >

1 0

[ANN] New package: SpaceFuncs (2D, 3D, ND geometric modeling, optimization, solving)
by Dmitrey Feb. 16, 2011

Feb. 16, 2011

Hi all, I'm glad to inform you about new, 4th OpenOpt Suite module: SpaceFuncs - a tool for 2D, 3D, N-dimensional geometric modeling with possibilities of parametrized calculations, numerical optimization and solving systems of geometrical equations with automatic differentiation. The module is written in Python + NumPy, requires FuncDesigner (and OpenOpt, DerApproximator for some operations). It has completely free license: BSD. For details see its home page http://openopt.org/SpaceFuncs and documentation http://openopt.org/SpaceFuncsDoc Also, you can try it online via our Sage-server (sometimes hangs due to high load, through) http://sage.openopt.org/welcome Regards, Dmitrey.

1 0

Re: [Numpy-discussion] convolving (or correlating) with sliding windows (Jonathan Hilmer)
by Davide Cittaro Feb. 16, 2011

Feb. 16, 2011

On 16/feb/2011, at 00:04, numpy-discussion-request(a)scipy.org wrote: > > I'm sorry that I don't have some example code for you, but you > probably need to break down the problem if you can't fit it into > memory: http://en.wikipedia.org/wiki/Overlap-add_method > > Jonathan Thanks! You saved my day, also I found I can apply this to many other problems... I only need a good "signal processing tips and tricks" book now :-) d

1 0

f2py "target file xxx not generated"
by Thomas Ingeman-Nielsen Feb. 15, 2011

Feb. 15, 2011

Hi, I'm trying to get started with f2py on a Windows 7 environment using the Python(x,y) v 2.6.5.6 distribution. I'm following the introductory example of the f2py userguide and try to wrap the file FIB1.F using the command: f2py.py -c fib1.f -m fib1 from the windows command line. I get the following output: running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler opti ons running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler opt ions running build_src build_src building extension "fib" sources f2py options: [] f2py:> c:\users\thomas\appdata\local\temp\tmpamyxnx\src.win32-2.6\fibmodule.c creating c:\users\thomas\appdata\local\temp\tmpamyxnx creating c:\users\thomas\appdata\local\temp\tmpamyxnx\src.win32-2.6 Reading fortran codes... Reading file 'fib1.f' (format:fix,strict) Post-processing... Post-processing (stage 2)... Building modules... error: f2py target file 'c:\\users\\thomas\\appdata\\local\\temp\\tmpamyxnx\\src .win32-2.6\\fibmodule.c' not generated output of f2py.py -c --help-fcompiler: Gnu95FCompiler instance properties: archiver = ['c:\\GCC451\\bin\\gfortran.exe', '-cr'] compile_switch = '-c' compiler_f77 = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-ffixed- form', '-fno-second-underscore', '-mno-cygwin', '-O3', '- funroll-loops'] compiler_f90 = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-fno-second- underscore', '-mno-cygwin', '-O3', '-funroll-loops'] compiler_fix = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-ffixed- form', '-fno-second-underscore', '-mno-cygwin', '-Wall', ' -fno-second-underscore', '-mno-cygwin', '-O3', '-funroll- loops'] libraries = ['gfortran'] library_dirs = ['c:\\gcc451\\lib', 'c:\\gcc451\\lib\\gcc\\i686-pc- mingw32\\4.5.1'] linker_exe = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-mno- cygwin', '-Wall', '-mno-cygwin'] linker_so = ['c:\\GCC451\\bin\\gfortran.exe', '-Wall', '-mno- cygwin', '-Wall', '-mno-cygwin', '-shared'] object_switch = '-o ' ranlib = ['c:\\GCC451\\bin\\gfortran.exe'] version = LooseVersion ('4.5.1') version_cmd = ['c:\\GCC451\\bin\\gfortran.exe', '--version', '-mno- cygwin'] Fortran compilers found: --fcompiler=gnu95 GNU Fortran 95 compiler (4.5.1) Compilers available for this platform, but not found: --fcompiler=absoft Absoft Corp Fortran Compiler --fcompiler=compaqv DIGITAL or Compaq Visual Fortran Compiler --fcompiler=g95 G95 Fortran Compiler --fcompiler=gnu GNU Fortran 77 compiler --fcompiler=intelem Intel Fortran Compiler for EM64T-based apps --fcompiler=intelev Intel Visual Fortran Compiler for Itanium apps --fcompiler=intelv Intel Visual Fortran Compiler for 32-bit apps --fcompiler=intelvem Intel Visual Fortran Compiler for 64-bit apps Compilers not available on this platform: --fcompiler=compaq Compaq Fortran Compiler --fcompiler=hpux HP Fortran 90 Compiler --fcompiler=ibm IBM XL Fortran Compiler --fcompiler=intel Intel Fortran Compiler for 32-bit apps --fcompiler=intele Intel Fortran Compiler for Itanium apps --fcompiler=lahey Lahey/Fujitsu Fortran 95 Compiler --fcompiler=mips MIPSpro Fortran Compiler --fcompiler=nag NAGWare Fortran 95 Compiler --fcompiler=none Fake Fortran compiler --fcompiler=pg Portland Group Fortran Compiler --fcompiler=sun Sun or Forte Fortran 95 Compiler --fcompiler=vast Pacific-Sierra Research Fortran 90 Compiler For compiler details, run 'config_fc --verbose' setup command. Running f2py without arguments result in the following version information: Version: 1 numpy Version: 1.5.1 Requires: Python 2.3 or higher. License: NumPy license (see LICENSE.txt in the NumPy source code) Copyright 1999 - 2005 Pearu Peterson all rights reserved. http://cens.ioc.ee/projects/f2py2e/ I'm puzzled by this, I expect I have the compiler correctly installed, since f2py recognizes it (and I am able to compile Fortran code using GCC from Eclipse). How about the f2py version number = 1? I was under the impression that development had reached version >2. Reinstalling Numpy from SourceForge numpy-1.5.1-win32-superpack-python2.6.exe result in exactly the same output as above. Trying to install f2py using easy_install and then running f2py without arguments result in the following error: Traceback (most recent call last): File "C:\Python26\Scripts\f2py.py", line 5, in <module> pkg_resources.run_script('f2py==2.45.241-1926', 'f2py.py') File "C:\Python26\lib\site-packages\pkg_resources.py", line 489, in run_script self.require(requires)[0].run_script(script_name, ns) File "C:\Python26\lib\site-packages\pkg_resources.py", line 1207, in run_scrip t execfile(script_filename, namespace, namespace) File "c:\python26\lib\site-packages\f2py-2.45.241_1926-py2.6-win32.egg\EGG-INF O\scripts\f2py.py", line 3, in <module> import f2py2e File "C:\Python26\lib\site-packages\f2py-2.45.241_1926-py2.6-win32.egg\f2py2e\ __init__.py", line 10, in <module> import f2py2e File "C:\Python26\lib\site-packages\f2py-2.45.241_1926-py2.6-win32.egg\f2py2e\ f2py2e.py", line 26, in <module> import crackfortran File "C:\Python26\lib\site-packages\f2py-2.45.241_1926-py2.6-win32.egg\f2py2e\ crackfortran.py", line 1586 as=b['args'] ^ SyntaxError: invalid syntax Any comments on how to adapt my setup to allow f2py to function correctly would be much appreciated! Best regards, Thomas

1 0

convolving (or correlating) with sliding windows
by Davide Cittaro Feb. 15, 2011

Feb. 15, 2011

Hi all, I have to work with huge numpy.array (i.e. up to 250 M long) and I have to perform either np.correlate or np.convolve between those. The process can only work on big memory machines but it takes ages. I'm writing to get some hint on how to speed up things (at cost of precision, maybe...), possibly using a moving window... is it correct to perform this: - given a window W and a step size S - given data1 and data2 - pad with zeros data1 and data2 by adding W/2 0-arrays - get the np.correlation like y = np.array([np.correlate(data1[x:x+W], data2[x:x+W], mode='valid') for x in np.arange(0, len(data1) - W, S)]).ravel() instead of np.correlate(data1, data2, mode='same') - interpolate the correlation using scipy.interpolate ? Thanks d

3 3

conditional array indexing
by Bryan Woods Feb. 14, 2011

Feb. 14, 2011

I have spent over an hour now trying to figure out what I thought would be a really basic conditional array assignment. I am very new to python and am in need of help. I have a 2D integer array of land cover classes and a 1D array with a roughness value for each class. I am trying to create a new 2D array containing a map of the roughness values. I am trying the following code: roughness = np.array(landcover.shape,dtype='f') for i in range(np.min(landcover),np.max(landcover)): roughness[landcover == i] = z0_legend[i] Alas it tells me that I can only have a 1D boolean array index. What is the easiest way around this without having to loop through each row of the 2D array. I need to keep this vectorized as much as possible as these arrays are very large. Thanks, Bryan

4 5

NumPy Featured on RCE-Cast
by Brock Palen Feb. 14, 2011

Feb. 14, 2011

Thank you to Travis for taking time to speak about NumPy on rce-cast.com You can find the interview and links to other shows at: http://www.rce-cast.com/Podcast/rce-48-numpy.html Thank you! Brock Palen www.umich.edu/~brockp Center for Advanced Computing brockp(a)umich.edu (734)936-1985

2 1