Mailman 3 May 2011 - NumPy-Discussion

finding longest contiguous array slices
by Ben Elliston 22 May '11

22 May '11

Hi all. I have a 3-d array of booleans (5 x 5 x 1000, say) and would like to return a 5 x 5 of array slices containing the longest contiguous sequence of True values along the z axis. Any tricks that I am missing that will make this easy? Thanks, Ben

2 1

How can Import DATA from a Fortran file
by Aradenatorix Veckhom Vacelaevus 21 May '11

21 May '11

Hi everybody: I have a file in simple text with information obtained in Fortran 77 and I need to use the data inside for visualize with Mayavi. I was fighting for a while with the VTK simple legacy format. Finally I could run an small example, but now I need to select specific information from that file. Let me explain how is it. I have a header of two lines where tell me the number of domains I have, and later the values of the position and pressure Later for each domain I have two arrays with 125 rows, the first one has another header with two lines where we can find the name of each variable (column) and the dimension of the array for each coordinate (x, y, z). Later we have the array (125 x 8) where the first column is the number or ID for each point (row) the next three contains the coordinates for the point pnx, pny, pnz the following three the displacement Dnx, Dny, Dnz finally the last column contains the value of the pressure field in each point. The second array has only one line as header where specifies the number of points (rows) and the names of each variable (column). This array has the dimension 125 x 11, as in the first array the first column has the number of row, the following columns contain the values of the alpha elements used for the Finite Element Analysis from where I have to find the velocity (vx = 0.5 (alpha1 + alpha2), vy = 0.5 (alpha3 + alpha4), vz = 0.5 (alpha5 + alpha6)) finally the last column is a string that says pos-velocity so we can forget it. In an schematic form: Header: 5 NUMBER OF SUBDOMAINS 4852108.558056722000000 4.858791656580212E+008 POSITION-PRESSURE First array 10.00 10.00 10.00 LX, LY, LZ, ... #header 5 5 5 LS,MS,NS, - NEE,X,Y,Z,DX,DY,DZ,LAMBDA (NEE) #header 1 .000E+00 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 485879165.658 #125 rows with 8 columns 2 .200E+01 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 362994604.232 3 .400E+01 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 287889668.714 4 .600E+01 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 249984468.929 5 .800E+01 .000E+00 .000E+00 .200E+01 .200E+01 .200E+01 224851296.708 . . . 125 .800E+01 .800E+01 .800E+01 .200E+01 .200E+01 .200E+01 192572200.800 Second array: 125 L, X, Y, Z, ALPHA(I1), ALPHA(I2), ALPHA(J1), ALPHA(J2), ALPHA(K1), ALPHA(K2), ... #header 1 1.000 1.000 1.000 .000E+00 .845E-04 .000E+00 .826E-04 .000E+00 .828E-04 POS-VELOCITY 2 3.000 1.000 1.000 .845E-04 .308E-04 .000E+00 .267E-04 .000E+00 .269E-04 POS-VELOCITY 3 5.000 1.000 1.000 .308E-04 .177E-04 .000E+00 .633E-05 .000E+00 .666E-05 POS-VELOCITY 4 7.000 1.000 1.000 .177E-04 .122E-04 .000E+00 .246E-05 .000E+00 .297E-05 POS-VELOCITY 5 9.000 1.000 1.000 .122E-04 .908E-05 .000E+00 .114E-05 .000E+00 .183E-05 POS-VELOCITY . . . 125 9.000 9.000 9.000 .102E-04 .160E-04 .133E-05 .000E+00 .457E-05 .000E+00 POS-VELOCITY And both arrays repeat other 4 times, it means I have 5 pairs of arrays, a pair for each domain. I want to divide the file in five pieces one for each domain, but what I really need is can manipulate the arrays by columns. I know that numpy is able to import files in many formats, and I want to believe that inside numpy I can easily manipulate an array by columns, but I don't how, so all this explanation is for ask your helping and find a way to can get from this arrays the information I need to write a new file with the info for export as vtk file and can visualize in Mayavi2 (do you have a better idea or way for visualize this?). Thanks for your helping and your time. Aradnix!

4 8

Conversion from structured dtype to object changed in Numpy 1.6.0.
by Mark Dickinson 21 May '11

21 May '11

With numpy 1.5.1: Python 2.7.1 |EPD 7.0-2 (32-bit)| (r271:86832, Dec 3 2010, 15:41:32) [GCC 4.0.1 (Apple Inc. build 5488)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from numpy import array >>> x = array((1.2, 2.3), dtype=[('field1', float), ('field2', float)]) >>> x.astype(object) array((1.2, 2.3), dtype=object) >>> So the conversion from the structured dtype item to type 'object' produces a tuple, which makes sense. With numpy 1.6.0, the same computation gives a different result, apparently truncating the tuple: Python 2.7.1 |EPD 7.0-2 (32-bit)| (r271:86832, Dec 3 2010, 15:41:32) [GCC 4.0.1 (Apple Inc. build 5488)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> from numpy import array >>> x = array((1.2, 2.3), dtype=[('field1', float), ('field2', float)]) >>> x.astype(object) array(1.2, dtype=object) >>> Was this change intentional, or should I file a bug? I couldn't find any reference to it in the changelog or release notes. Thanks, Mark

3 3

Broken numpy install
by Milan Mitrovic 21 May '11

21 May '11

Hello everyone, I have a problem with my numpy installation. I installed numpy, scipy and matplotlib from source, about one year ago (the macports install wouldn't work for me). This worked fine untill I had some problems two days ago (as it later turned out, completely unrelated to numpy) but my thesis advisor suggested that I update numpy and scipy. Since then I've been trying to make it work (and haven't worked on my thesis at all). So, I updated the git/svn code to the latest versions, re-ran build and install and things don't work any more. I have a script that I am trying to use and here is the error message that I get: RuntimeError: module compiled against ABI version 1000009 but this version of numpy is 2000000 Traceback (most recent call last): File "../ppm/libppm/utils/ppm_dbg/ppmdbg.py", line 2, in <module> from mpl_toolkits.mplot3d import Axes3D File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mpl_toolkits/mplot3d/__init__.py", line 1, in <module> from axes3d import Axes3D File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/mpl_toolkits/mplot3d/axes3d.py", line 13, in <module> from matplotlib.axes import Axes, rcParams File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/axes.py", line 12, in <module> import matplotlib.artist as martist File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/artist.py", line 6, in <module> from transforms import Bbox, IdentityTransform, TransformedBbox, TransformedPath File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/matplotlib/transforms.py", line 34, in <module> from matplotlib._path import affine_transform ImportError: numpy.core.multiarray failed to import I have tried googling and followed every piece of advice I could find. Nothing worked. Please let me know what info I need to provide so you can help me. Or tell me a way to completely clean my system and start all over (cause it seems that it is being built for the wrong python). Btw, some people on #scipy on freenode suggested that I try running the script with all interpreters I get from which -a python... they all failed with the same error message. Thanks in advance, Milan Mitrovic

2 6

f2py on Windows, compiler options
by Mathew Yeates 19 May '11

19 May '11

Hi I am trying to run f2py and link to some libraries. I get a link error LINK : fatal error LNK1104: cannot open file 'LIBC.lib' because (I think) the libraries are compiled with /MT (multithreaded). I tried adding /NODFEAU:TLIB:libc.lib but then I have unresolved dependencies. I want to try compiling the code with /MT. How can tell f2py to use this option when compiling (not linking, this option is passed to cl). -Mathew

1 2

more python 3.2 fun
by josef.pktd＠gmail.com 19 May '11

19 May '11

(just illustrating some porting fun) I was struggling with another python 3.2 bug in scikits.statsmodels (with grunfeld data) with numpy 1.5.1, I'm reading the data with data = recfromtxt(open(filepath + '/grunfeld.csv', 'rb'), delimiter=",", names=True, dtype="f8,f8,f8,a17,f8") >>> grun_exog['firm'][:10] array([b'General Motors', b'General Motors', b'General Motors', b'General Motors', b'General Motors', b'General Motors', b'General Motors', b'General Motors', b'General Motors', b'General Motors'], dtype='|S17') using predefined firms defined as string >>> firms = ['General Motors', 'Chrysler', 'General Electric', 'Westinghouse', 'US Steel'] >>> i 'General Motors' comparing bytes and string produces False not an element wise comparison >>> grun_exog['firm'] == i False >>> grun_exog['firm'][0] == i False >>> grun_exog['firm'][0] , i (b'General Motors', 'General Motors') numpy has a convenient conversion function >>> from numpy.compat import asbytes >>> ( grun_exog['firm'] == asbytes(i) )[:10] array([ True, True, True, True, True, True, True, True, True, True], dtype=bool) Josef

1 0

Memory leak/fragmentation when using np.memmap
by G Jones 19 May '11

19 May '11

Hello, I need to process several large (~40 GB) files. np.memmap seems ideal for this, but I have run into a problem that looks like a memory leak or memory fragmentation. The following code illustrates the problem import numpy as np x = np.memmap('mybigfile.bin',mode='r',dtype='uint8') print x.shape # prints (42940071360,) in my case ndat = x.shape[0] for k in range(1000): y = x[k*ndat/1000:(k+1)*ndat/1000].astype('float32') #The astype ensures that the data is read in from disk del y One would expect such a program would have a roughly constant memory footprint, but in fact 'top' shows that the RES memory continually increases. I can see that the memory usage is actually occurring because the OS eventually starts to swap to disk. The memory usage does not seem to correspond with the total size of the file. Has anyone see this behavior? Is there a solution? I found this article: http://pushingtheweb.com/2010/06/python-and-tcmalloc/ which sounds similar, but it seems that the ~40 MB chunks I am loading would be using mmap anyway so could be returned to the OS. I am using nearly the latest version of numpy from the git repository (np.__version__ returns 2.0.0.dev-Unknown), Python 2.7.1, and RHEL 5 on x86_64. I appreciate any suggestions. Thanks, Glenn

3 4

Windows Registry Keys
by Mathew Yeates 19 May '11

19 May '11

Hi I have installed a new version of Python27 in a new directory. I want to get this info into the registry so, when I install Numpy, it will use my new Python TIA -Mathew

3 7

Windows dev-build Numpy 1.6.1
by Wieland Brendel 19 May '11

19 May '11

I succeeded now in installing the latest Numpy version. There was some problem in mingw32ccompiler.py. I had to change the lines elif self.gcc_version < "4.": self.set_executables(compiler='gcc -mno-cygwin -O2 -Wall', compiler_so='gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes', linker_exe='g++ -mno-cygwin', linker_so='g++ -mno-cygwin -shared') else: # gcc-4 series releases do not support -mno-cygwin option self.set_executables(compiler='gcc -O2 -Wall', compiler_so='gcc -O2 -Wall -Wstrict-prototypes', linker_exe='g++ ', linker_so='g++ -shared') into else: self.set_executables(compiler='gcc -mno-cygwin -O2 -Wall', compiler_so='gcc -mno-cygwin -O2 -Wall -Wstrict-prototypes', linker_exe='g++ -mno-cygwin', linker_so='g++ -mno-cygwin -shared') Thanks very much for your help again! Wieland

3 4

array order in numpy.dot
by RadimRehurek 19 May '11

19 May '11

Hello, what code paths does `numpy.dot(matrix, matrix)` take with regard to the order of the input matrices (c/fortran contiguous?). In particular, under what conditions does it make a copy of the input matrices, and under which conditions does it set the gemm transposition flags internally? Does this change with different numpy versions, is there a contract re this behaviour? My concern is 1) memory (matrices are large, cannot afford extra copies) and 2) performance. Sorry if I missed this in the documentation. Cheers! Radim

1 0