Mailman 3 February 2010 - NumPy-Discussion

odd ascii format and genfromtxt
by Ralf Gommers 26 Feb '10

26 Feb '10

Hi all, I'm trying to read in data from text files with genfromtxt, and have some trouble figuring out the right combination of keywords. The format is: ['0\t\t4.000000000000000e+007,0.000000000000000e+000\n', '\t9.860280631554179e-001,-1.902586503306264e-002\n', '\t9.860280631554179e-001,-1.902586503306264e-002'] Note that there are two delimiters, tab and comma. Also, the first line has an extra integer plus tab (this is a repeating pattern). The files are large, there's a lot of them, and they're generated by a binary I can't modify. Here are some things I've tried: In [216]: np.genfromtxt('ascii2test.raw', invalid_raise=False) Out[216]: array([ 0., NaN]) In [217]: np.genfromtxt('ascii2test.raw', invalid_raise=False, delimiter=['\t', ',']) TypeError: cannot perform accumulate with flexible type In [228]: np.genfromtxt('ascii2test.raw', delimiter=['\t', ','], dtype=[('intvar', '<i8'), ('fltvar', '<f8'), ('fltvar2', '<f8')]) TypeError: cannot perform accumulate with flexible type Any suggestions? Thanks, Ralf

2 2

problem w 32bit binomial?
by James Bergstra 25 Feb '10

25 Feb '10

In case this hasn't been solved in more recent numpy... I've tried the following lines on two installations of numpy 1.3 with python 2.6 numpy.random.binomial(n=numpy.asarray([2,3,4], dtype='int64'), p=numpy.asarray([.1, .2, .3], dtype='float64')) A 64bit computer gives an output of array length 3. A 32bit computer gives an error: TypeError: array cannot be safely cast to required type If I change the int64 cast to an int32 cast then it works on both machines. Thanks, James -- http://www-etud.iro.umontreal.ca/~bergstrj -- http://www-etud.iro.umontreal.ca/~bergstrj

2 2

read ascii file with quote delimited strings
by Sam Tygier 25 Feb '10

25 Feb '10

Hi I am trying to read an ascii file which mixes ints, floats and stings. eg. 1 2.3 'a' 'abc ' 2 3.2 'b' ' ' 3 3.4 ' ' 'hello' Within a column that data is always the same. the strings are sometimes contain with spaces. I have tried giving loadtxt a dtype that specifies the length of the strings: [('a', int), ('b', float), ("c", "a1"), ("d", "a5")] or including the quotes: [('a', int), ('b', float), ("c", "a3"), ("d", "a7")] but it seems that loadtxt uses split() before looking at the dtype, so for example line 3 becomes ["3", "3.4", "'", "'", "'hello'"] and my 2 string elements contain only quotes. Would it be possible a dtype of "a3" to force reading 3 chars? Or would it make more sense for loadtxt to have a quote_char, that i could set to "'". This would make it ignore whitespace between quote_chars. Sam

3 2

Numpy array performance issue
by Bruno Santos 25 Feb '10

25 Feb '10

Hello everyone, I am using numpy arrays whenever I demand performance from my algorithms. Nevertheless, I am having a performance issue at the moment mainly because I am iterating several times over numpy arrays. Fot that reason I decided to use timeit to see the performance of different versions of the same procedure. What surprised me was that in fact Python lists are performing almost ten times faster than numpy. Why is this happening. My test code is this: list1 = [random.randint(0,20) for i in xrange(100)] list2 = numpy.zeros(100,dtype='Int64') for i in xrange(100):list2[i]=random.randint(0,20) def test1(listx): return len([elem for elem in list if elem >=10]) t = timeit.Timer("test1(list1)","from __main__ import *") >>> t.timeit() 6.4516620635986328 t = timeit.Timer("test1(list2)","from __main__ import *") >>> t.timeit() 76.807533979415894 Thanks in advance, Bruno

3 18

Want cumsum-like function
by Peter Shinners 25 Feb '10

25 Feb '10

I want a function that works like cumsum, but starts at zero, instead of starting with the first actual value. For example; I have an array with [4,3,3,1]. Cumsum will give me an array with [4,7,10,11]. I want an array that is like [0,4,7,8]. It looks like I could indirectly do this: tallies = np.cumsum(initial_array) np.subtract(tallies, tallies[0], tallies) But is there a more efficient way to get this specific result?

3 5

For-less code
by Gökhan Sever 25 Feb '10

25 Feb '10

Hello, I am working on a code shown at http://code.google.com/p/ccnworks/source/browse/trunk/thesis/part1/logn-fit… I use the code to analyse a couple dataset also placed in the same directory. In the first part I use for-loops all over, but later decided to write them without using for loops. The script runs correctly for the two section, however I have a few question regarding to the for-less structures: This part is taken after line 371: gm10 = np.exp((1/Nt10)*(h10*np.log(Dp)).sum(axis=1)) gsd10 = np.exp(((1/Nt10)*(h10*np.log(Dp/gm10[:,np.newaxis])**2).sum(axis=-1))**0.5) dN_dDp10 = (Nt10[:,np.newaxis]/((2*np.pi)**0.5*np.log(gsd10[:,np.newaxis])*d))*np.exp(-(np.log(d)-\ np.log(gm10[:,np.newaxis]))**2/(2*np.log(gsd10[:,np.newaxis])**2)) a10 = (dN_dDp10[0:300,d >= dc10u]*0.001).sum(axis=1) Shape informations for the arrays as follow: I[306]: gm10.shape; gsd10.shape, Dp.shape, d.shape, dN_dDp10.shape, a10.shape O[306]: (300,), (300,), (11,), (991,), (300, 991), (300,) 1-) In gsd10 line what does axis=-1 really means, and why negative axis value is allowed? 2-) [:,np.newaxis] decreases the readability of the code. Is there any alternative for it? 3-) In the last line (dN_dDp10[0:300,d >= dc10u]) could I achieve the same result with different syntax? I have found it somewhat not in accordance with the previous lines. Thanks for your time and comments. -- Gökhan

2 1

RHEL 5.3+ build?
by David Carmean 24 Feb '10

24 Feb '10

Does anyone use/build this stuff on RHEL 5.3+ (x64)? :) Seems not so much. I'd like to use numpy (and PyTables) for a few tasks where it would be much more efficient to have much of the processing performed on the servers generating the data (about 400 systems) than backhauling the huge amount of input data across our WAN around the continent. However, the vast majority of these systems are 64-bit RedHat EL 5.3 and 5.4, and I'm having trouble building numpy 1.3.0 with gcc. I found an RPM for 1.2.0 so that will get me through most of the R&D, and I'd rather wait for the next stable release before spending any more time trying to build. But I'm wondering if there's anybody on the team or in the active contributors/users world who is regularly building numpy on various flavors of CentOS/RHEL5.x. Thanks.

3 4

distutils problem with NumPy-1.4 & Py-2.7a3 (Snow Leopard)
by Tom Loredo 24 Feb '10

24 Feb '10

Hi- I've been testing Python-2.7a3 on Mac OS 10.6.2. NumPy-1.4.0 will not install; it appears something has changed within distutils that breaks it: $ export MACOSX_DEPLOYMENT_TARGET=10.6 $ export CFLAGS="-arch x86_64" $ export FFLAGS="-m64" $ export LDFLAGS="-Wall -undefined dynamic_lookup -bundle -arch x86_64" $ time python setup.py build --fcompiler=gnu95 Running from numpy source directory. Traceback (most recent call last): File "setup.py", line 187, in <module> setup_package() File "setup.py", line 155, in setup_package from numpy.distutils.core import setup File "/Volumes/System/Users/loredo/Downloads/numpy-1.4.0-OSX/numpy/distutils/__init__.py", line 6, in <module> import ccompiler File "/Volumes/System/Users/loredo/Downloads/numpy-1.4.0-OSX/numpy/distutils/ccompiler.py", line 17, in <module> _old_init_posix = distutils.sysconfig._init_posix AttributeError: 'module' object has no attribute '_init_posix' I realize NumPy makes no claim to be compatible with 2.7(alpha); I'm reporting this as a heads-up. -Tom PS: For testing purposes: To get nose to install for 2.7a3, you need to use the current hg branch. The last release (including the out-of-date dev branch on PyPI) is not compatible with 2.7 changes to unittest internals. ------------------------------------------------- This mail sent through IMP: http://horde.org/imp/

3 6

Building Windows binaries on OS X
by Ralf Gommers 24 Feb '10

24 Feb '10

Hi David and all, I have a few questions on setting up the build environment on OS X for Windows binaries. I have Wine installed with Python 2.5 and 2.6, MakeNsis and MinGW. The first question is what is meant in the Paver script by "cpuid plugin". Wine seems to know what to do with a cpuid instruction, but I can not find a plugin. Searching for "cpuid plugin" turns up nothing except the NumPy pavement.py file. What is this? Second question is about Fortran. It's needed for SciPy at least, so I may as well get it right now. MinGW only comes with g77, and this page: http://www.scipy.org/Installing_SciPy/Windows says that this is the default compiler. So Fortran 77 on Windows and Fortran 95 on OS X as defaults, is that right? No need for g95/gfortran at all? Final question is about Atlas and friends. Is 3.8.3 the best version to install? Does it compile out of the box under Wine? Is this page http://www.scipy.org/Installing_SciPy/Windows still up-to-date with regard to the Lapack/Atlas info and does it apply for Wine? And do I have to compile it three times, with the only difference the '-arch' flag set to "SSE2", "SSE3" and "<what's NoSSE?>"? Thanks, Ralf

5 16

long(a) vs a.__long__() for scalar arrays
by David Cournapeau 23 Feb '10

23 Feb '10

Hi, I am a bit puzzled by the protocol for long(a) where a is a scalar array. For example, for a = np.float128(1), I was expecting long(a) to call a.__long__, but it does not look like it is the case. int(a) does not call a.__int__ either. Where does the long conversion happen in numpy for scalar arrays ? cheers, David

4 4