Mailman 3 August 2007 - NumPy-Discussion

How to implement a 'pivot table?'
by Geoffrey Zhu 06 Aug '07

06 Aug '07

Hi Everyone, I am wondering what is the best (and fast) way to build a pivot table aside from the 'brute force way?' I want to transform an numpy array into a pivot table. For example, if I have a numpy array like below: Region Date # of Units ---------- ---------- -------------- East 1/1 10 East 1/1 20 East 1/2 30 West 1/1 40 West 1/2 50 West 1/2 60 I want to transform this into the following table, where f() is a given aggregate function: Date Region 1/1 1/2 ---------- East f(10,20) f(30) West f(40) f(50,60) I can regroup them into 'sets' and do it the brute force way, but that is kind of slow to execute. Does anyone know a better way? Thanks, Geoffrey

5 10

Re: [Numpy-discussion] fourier with single precision
by Lars Friedrich 06 Aug '07

06 Aug '07

Hello, thanks for your comments. If I got you right, I should look for a FFT-code that uses SSE (what does this actually stand for?), which means that it vectorizes 32bit-single-operations into larger chunks that make efficient use of recent CPUs. You mentioned FFTW and MKL. Is this www.fftw.org and the 'intel math kernel library'? If I would like to use one of them, is numpy the right place to put it in? Does anyone know, if it is possible to switch on SSE support (at compile time) in the fftpack.c that numpy uses? Thanks Lars

4 4

multinomial error?
by adam.powell＠ucl.ac.uk 05 Aug '07

05 Aug '07

Hi, I appear to be having a problem with the random.multinomial function. For some reason if i attempt to loop over a large number of single-trial multinomial picks then the function begins to ignore some non-zero entries in my 1-D array of multinomial probabilities... Is seems that there is no upper limit on the size of the probability array for a one off multinomial pick, but if looping over the process multiple times the function can't handle the whole array and seems to truncate it arbitrarily before performing the trial with only the remaining probabilities. There is a reason why i need to loop over a large number of single-trial events, rather than just replacing the loop with a large number of trials in one single multinomial pick (annoying, as that's so much quicker!). Thanks for any help, Adam

2 1

Re: [Numpy-discussion] fourier with single precision
by Lars Friedrich 03 Aug '07

03 Aug '07

Hello, David Cournapeau wrote: > As far as I can read from the fft code in numpy, only double is > supported at the moment, unfortunately. Note that you can get some speed > by using scipy.fftpack methods instead, if scipy is an option for you. What I understood is that numpy uses FFTPACK's algorithms. From www.netlib.org/fftpack (is this the right address?) I took that there is a single-precision and double-precision-version of the algorithms. How hard would it be (for example for me...) to add the single-precision versions to numpy? I am not a decent C-hacker, but if someone tells me, that this task is not *too* hard, I would start looking more closely at the code... Would it make sense, that if one passes an array of dtype = numpy.float32 to the fft function, a complex64 is returned, and if one passes an array of dtype = numpy.float64, a complex128 is returned? Lars

3 4

reference leaks in array() and arange()
by Lisandro Dalcin 03 Aug '07

03 Aug '07

As PyArray_DescrConverter return new references, I think there could be many places were PyArray_Descr* objects get its reference count incremented. Here, I send a patch correcting this for array() and arange(), but not sure if this is the more general solution. BTW, please see my previous comments in previous mail on using convertor functions (returning new refs) and the (very common) idiom "if(!PyArg_PaseXXX(,,,) return NULL", as this seems to be used in almost all places in numpy C sources and is a potential source of ref leaks. Regards, -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

1 0

reference leacks in numpy.asarray
by Lisandro Dalcin 02 Aug '07

02 Aug '07

using numpy-1.0.3, I believe there are a reference leak somewhere. Using a debug build of Python 2.5.1 (--with-pydebug), I get the following import sys, gc import numpy def testleaks(func, args=(), kargs={}, repeats=5): for i in xrange(repeats): r1 = sys.gettotalrefcount() func(*args,**kargs) r2 = sys.gettotalrefcount() rd = r2-r1 print 'before: %d, after: %d, diff: [%d]' % (r1, r2, rd) def npy_asarray_1(): a = numpy.zeros(5, dtype=int) b = numpy.asarray(a, dtype=float) del a, b def npy_asarray_2(): a = numpy.zeros(5, dtype=float) b = numpy.asarray(a, dtype=float) del a, b if __name__ == '__main__': testleaks(npy_asarray_1) testleaks(npy_asarray_2) $ python npyleaktest.py before: 84531, after: 84532, diff: [1] before: 84534, after: 84534, diff: [0] before: 84534, after: 84534, diff: [0] before: 84534, after: 84534, diff: [0] before: 84534, after: 84534, diff: [0] before: 84531, after: 84533, diff: [2] before: 84535, after: 84536, diff: [1] before: 84536, after: 84537, diff: [1] before: 84537, after: 84538, diff: [1] before: 84538, after: 84539, diff: [1] It seems npy_asarray_2() is leaking a reference. I am missing something here?. The same problem is found in C, using PyArray_FROM_OTF (no time to go inside to see what's going on, sorry) If this is know and solved in SVN, please forget me. Regards, -- Lisandro Dalcín --------------- Centro Internacional de Métodos Computacionales en Ingeniería (CIMEC) Instituto de Desarrollo Tecnológico para la Industria Química (INTEC) Consejo Nacional de Investigaciones Científicas y Técnicas (CONICET) PTLC - Güemes 3450, (3000) Santa Fe, Argentina Tel/Fax: +54-(0)342-451.1594

2 4

16bit Integer Array/Scalar Inconsistency
by Ryan May 02 Aug '07

02 Aug '07

Hi, I ran into this while debugging a script today: In [1]: import numpy as N In [2]: N.__version__ Out[2]: '1.0.3' In [3]: d = N.array([32767], dtype=N.int16) In [4]: d + 32767 Out[4]: array([-2], dtype=int16) In [5]: d[0] + 32767 Out[5]: 65534 In [6]: type(d[0] + 32767) Out[6]: <type 'numpy.int64'> In [7]: type(d[0]) Out[7]: <type 'numpy.int16'> It seems that numpy will automatically promote the scalar to avoid overflow, but not in the array case. Is this inconsistency a bug, just a (known) gotcha? I myself don't have any problems with the array not being promoted automatically, but the inconsistency with scalar operation made debugging my problem more difficult. Ryan -- Ryan May Graduate Research Assistant School of Meteorology University of Oklahoma

2 1

Memory efficient equality test for arrays
by Tom Goddard 02 Aug '07

02 Aug '07

Is there a numpy call to test if two large arrays (say 1 Gbyte each) are equal (same shape and elements) without creating another large array of booleans as happens with "a == b", numpy.equal(a,b), or numpy.array_equal(a,b)? I want a memory efficient and fast comparison. Tom

1 0

fourier with single precision
by Lars Friedrich 02 Aug '07

02 Aug '07

Hello, is there a way to tell numpy.fft.fft2 to use complex64 instead of complex128 as output dtype to speed the up transformation? Thanks Lars

2 1

f2py self documenting not working
by Randy Direen 01 Aug '07

01 Aug '07

Im using f2py under numpy. I've written several simple examples and f2py has not generated any documentations for the routines I have made. Any help would be great, I am very new to f2py and I would like to use the tool to wrap a rather large program written in Fortran90. Thanks! Randy Direen NIST

1 0