Mailman 3 July 2010 - NumPy-Discussion

Re: [Numpy-discussion] Array concatenation performance
by jf＠heliotis.ch July 15, 2010

July 15, 2010

I will be out of office till 8. August For urgent matters, please contact Patrick.Lambelet(a)heliotis.ch

1 0

Crosstabulation
by Ionut Sandric July 14, 2010

July 14, 2010

Sorry, the first email was sent before to finish it... Hi: I have two raster data and I would like to do a crosstabulation between them and export the results to a table in a text file. Is it possible to do it with NumPy? Does someone have an example? Thank you, Ionut

2 1

cannot set dtype on record array with 04 datetime records
by John Hunter July 14, 2010

July 14, 2010

I use record arrays extensively with python datetimes, which works if you pass in a list of lists of data with the names. numpy can accurately infer the dtypes and create a usable record array. Eg, import datetime import numpy as np rows = [ [datetime.date(2001,1,1), 12, 23.], [datetime.date(2002,1,1), 10, 13.], [datetime.date(2003,1,1), -2, 1.], ] r1 = np.rec.fromrecords(rows, names='a,b,c') print r1.dtype prints out: [('a', '|O4'), ('b', '<i4'), ('c', '<f8')] but if I want to speed things up by providing the dtype, numpy raises ValueError: dtype = [('a', '|O4'), ('b', '<i4'), ('c', '<f8')] r2 = np.rec.fromrecords(rows, dtype=dtype) /home/titan/johnh/test.py 12 dtype = [('a', '|O4'), ('b', '<i4'), ('c', '<f8')] 13 ---> 14 r2 = np.rec.fromrecords(rows, dtype=dtype) 15 16 /home/titan/johnh/dev/lib/python2.4/site-packages/numpy/core/records.pyc in fromrecords(recList, dtype, shape, formats, names, titles, aligned, byteorder) 610 611 try: --> 612 retval = sb.array(recList, dtype=descr) 613 except TypeError: # list of lists instead of list of tuples 614 if (shape is None or shape == 0): ValueError: Setting void-array with object members using buffer. WARNING: Failure executing file: <test.py> Running from svn HEAD: In [2]: numpy.__version__ Out[2]: '2.0.0.dev8480' Here is a complete script:: import datetime import numpy as np rows = [ [datetime.date(2001,1,1), 12, 23.], [datetime.date(2002,1,1), 10, 13.],I use record arrays extensively with python datetimes, which works if you pass in a list of lists of data with the names. numpy can accurately infer the dtypes and create a usable record array. Eg, import datetime import numpy as np rows = [ [datetime.date(2001,1,1), 12, 23.], [datetime.date(2002,1,1), 10, 13.], [datetime.date(2003,1,1), -2, 1.], ] r1 = np.rec.fromrecords(rows, names='a,b,c') print r1.dtype prints out: [('a', '|O4'), ('b', '<i4'), ('c', '<f8')] but if I want to speed things up by providing the dtype, numpy raises ValueError: dtype = [('a', '|O4'), ('b', '<i4'), ('c', '<f8')] r2 = np.rec.fromrecords(rows, dtype=dtype) /home/titan/johnh/test.py 12 dtype = [('a', '|O4'), ('b', '<i4'), ('c', '<f8')] 13 ---> 14 r2 = np.rec.fromrecords(rows, dtype=dtype) 15 16 /home/titan/johnh/dev/lib/python2.4/site-packages/numpy/core/records.pyc in fromrecords(recList, dtype, shape, formats, names, titles, aligned, byteorder) 610 611 try: --> 612 retval = sb.array(recList, dtype=descr) 613 except TypeError: # list of lists instead of list of tuples 614 if (shape is None or shape == 0): ValueError: Setting void-array with object members using buffer. WARNING: Failure executing file: <test.py> Running from svn HEAD: In [2]: numpy.__version__ Out[2]: '2.0.0.dev8480' Here is a complete script:: import datetime import numpy as np rows = [ [datetime.date(2001,1,1), 12, 23.], [datetime.date(2002,1,1), 10, 13.], [datetime.date(2003,1,1), -2, 1.], ] r1 = np.rec.fromrecords(rows, names='a,b,c') print r1.dtype dtype = [('a', '|O4'), ('b', '<i4'), ('c', '<f8')] r2 = np.rec.fromrecords(rows, dtype=dtype) [datetime.date(2003,1,1), -2, 1.], ] r1 = np.rec.fromrecords(rows, names='a,b,c') print r1.dtype dtype = [('a', '|O4'), ('b', '<i4'), ('c', '<f8')] r2 = np.rec.fromrecords(rows, dtype=dtype) I filed a ticket on the tracker: http://projects.scipy.org/numpy/ticket/1544 Thanks! JDH

1 0

HOWTO build NumPy without external math libs
by Stefan Seefeld July 14, 2010

July 14, 2010

Hello, the NumPy docs (at http://docs.scipy.org/doc/numpy/user/install.html) suggest that NumPy doesn't require external linear algebra libraries (such as ATLAS). How can I compiler NumPy without using them, even if they are detected ? Are there any undocumented options I can pass to setup.py ? (`setup.py build --help` didn't report anything.) Thanks, Stefan -- ...ich hab' noch einen Koffer in Berlin...

6 7

numpy installation question (IronPython)
by William Johnston July 13, 2010

July 13, 2010

Hello, I simply installed numpy in my Python26 installation, and then copied the numpy directory to my site-packages folder of my IronPython installation. Did I miss any installation steps in doing so? The multiarray module could not be found using IronPython. Thanks.

4 3

[ANN] Autumn School "Advanced Scientific Programming in Python" in Trento, Italy
by Tiziano Zito July 13, 2010

July 13, 2010

Advanced Scientific Programming in Python ========================================= an Autumn School by the G-Node, the Center for Mind/Brain Sciences and the Fondazione Bruno Kessler Scientists spend more and more time writing, maintaining, and debugging software. While techniques for doing this efficiently have evolved, only few scientists actually use them. As a result, instead of doing their research, they spend far too much time writing deficient code and reinventing the wheel. In this course we will present a selection of advanced programming techniques with theoretical lectures and practical exercises tailored to the needs of a programming scientist. New skills will be tested in a real programming project: we will team up to develop an entertaining scientific computer game. We'll use the Python programming language for the entire course. Python works as a simple programming language for beginners, but more importantly, it also works great in scientific simulations and data analysis. Clean language design and easy extensibility are driving Python to become a standard tool for scientific computing. Some of the most useful open source libraries for scientific computing and visualization will be presented. This school is targeted at Post-docs and PhD students from all areas of science. Competence in Python or in another language such as Java, C/C++, MATLAB, or Mathematica is absolutely required. A basic knowledge of the Python language is assumed. Participants without prior experience with Python should work through the proposed introductory materials. Date and Location ================= October 4th—8th, 2010. Trento, Italy. Preliminary Program =================== Day 0 (Mon Oct 4) — Software Carpentry & Advanced Python • Documenting code and using version control • Object-oriented programming, design patterns, and agile programming • Exception handling, lambdas, decorators, context managers, metaclasses Day 1 (Tue Oct 5) — Software Carpentry • Test-driven development, unit testing & Quality Assurance • Debugging, profiling and benchmarking techniques • Data serialization: from pickle to databases Day 2 (Wed Oct 6) — Scientific Tools for Python • Advanced NumPy • The Quest for Speed (intro): Interfacing to C • Programming project Day 3 (Thu Oct 7) — The Quest for Speed • Writing parallel applications in Python • When parallelization does not help: the starving CPUs problem • Programming project Day 4 (Fri Oct 8) — Practical Software Development • Efficient programming in teams • Programming project • The Pac-Man Tournament Every evening we will have the tutors' consultation hour: Tutors will answer your questions and give suggestions for your own projects Applications ============ You can apply on-line at http://www.g-node.org/python-autumnschool Applications must be submitted before August 31th, 2010. Notifications of acceptance will be sent by September 4th, 2010. No fee is charged but participants should take care of travel, living, and accommodation expenses. Candidates will be selected on the basis of their profile. Places are limited: acceptance rate in past editions was around 30%. Prerequisites ============= You are supposed to know the basics of Python to participate in the lectures! Look on the website for a list of introductory material. Faculty ======= • Francesc Alted, author of PyTables, Castelló de la Plana, Spain • Pietro Berkes, Volen Center for Complex Systems, Brandeis University, USA • Valentin Haenel, Berlin Institute of Technology and Bernstein Center for Computational Neuroscience Berlin, Germany • Zbigniew Jędrzejewski-Szmek, Faculty of Physics, University of Warsaw, Poland • Eilif Muller, The Blue Brain Project, Ecole Polytechnique Fédérale de Lausanne, Switzerland • Emanuele Olivetti, NeuroInformatics Laboratory, Fondazione Bruno Kessler and University of Trento, Italy • Rike-Benjamin Schuppner, Bernstein Center for Computational Neuroscience Berlin, Germany • Bartosz Teleńczuk, Institute for Theoretical Biology, Humboldt-Universität zu Berlin, Germany • Bastian Venthur, Berlin Institute of Technology and Bernstein Focus: Neurotechnology, Germany • Stéfan van der Walt, Applied Mathematics, University of Stellenbosch, South Africa • Tiziano Zito, Berlin Institute of Technology and Bernstein Center for Computational Neuroscience Berlin, Germany Organized by Paolo Avesani for the Center for Mind/Brain Sciences and the Fondazione Bruno Kessler , and by Zbigniew JędrzejewscySzmek and Tiziano Zito for the German Neuroinformatics Node of the INCF. Website: http://www.g-node.org/python-autumnschool Contact: python-info(a)g-node.org

5 5

Memory leak found in ndarray (I think)?
by Wes McKinney July 13, 2010

July 13, 2010

This one was quite a bear to track down, starting from the of course very high level observation of "why is my application leaking memory". I've reproduced it on Windows XP using NumPy 1.3.0 on Python 2.5 and 1.4.1 on Python 2.6 (EPD). Basically it seems that calling .astype(bool) on an ndarray slice with object dtype is leaving a hanging reference count, should be pretty obvious to see: from datetime import datetime import numpy as np import sys def foo(verbose=True): arr = np.array([datetime.today() for _ in xrange(1000)]) arr = arr.reshape((500, 2)) sl = arr[:, 0] if verbose: print 'Rec ct of index 0: %d' % sys.getrefcount(sl[0]) for _ in xrange(10): foo = sl.astype(bool) if verbose: print 'Rec ct of index 0: %d' % sys.getrefcount(sl[0]) if __name__ == '__main__': foo() for i in xrange(10000): if not i % 1000: print i foo(verbose=False) On my machine this bleeds about 100 MB of memory that you don't get back-- let me know if I've misinterpreted the results. I'll happily create a ticket on the Trac page. Thanks, Wes

2 5

BOF notes: Fernando's proposal: NumPy ndarray with named axes
by Jonathan March July 13, 2010

July 13, 2010

Fernando Perez proposed a NumPy enhancement, an ndarray with named axes, prototyped as DataArray by him, Mike Trumpis, Jonathan Taylor, Matthew Brett, Kilian Koepsell and Stefan van der Walt. At SciPy 2010 on July 1, Fernando convened a BOF (Birds of a Feather) discussion of this proposal. The notes from this BOF can be found at: http://projects.scipy.org/numpy/wiki/NdarrayWithNamedAxes (linked from the Plans section of http://projects.scipy.org/numpy ) HELP NEEDED: Fernando does not have the resources to drive the project beyond this prototype, which already does what he needs. If this is to go anywhere, it needs people to do the work. Please step forward.

15 89

Re FFTs in SciPy (and NumPy)
by Sturla Molden July 13, 2010

July 13, 2010

There has been some discussion on FFTPACK lately. Problems with FFTPACK seems to be: - Written in old Fortran 77. - Unprecise for single precision. - Can sometimes be very slow, depending on input size. - Can only handle a few small prime factors {2,3,4,5} efficiently. - How to control integer size portably for 64-bit and f2py? - Only 1D transforms. There is another FFT library in cwplib from Colorado Mining School. It has some very nice properties: - Written in C, not Fortran, and the FFT-part of the library is just one single C file. - License is BSD it seems. - Quite fast (see benchmarks in http://www.fftw.org/fftw-paper.pdf and http://www.fftw.org/speed/) - Can handle larger set of prime factors than FFTPACK {2,3,4,5,7,8,9,11,13,16}. - 1D and 2D inplace transforms. The problem with cwplib is that it does not allow arbitrary sized FFTs. But til will be fast whenever FFTPACK is fast (it seems FFTPACK is only competitive for N=2**k case). It will also be fast for other factorizations where FFTPACK will fallback to O(N**2). How to handle arbitrary sized arrays with cwplib? One possibility is using Bluestein's FFT algorithm, which just requires a few lines og Python. This was even suggested to avoid O(N**2) fallback in FFTPACK here a while ago. If we are going to do that for FFTPACK, the cwplib's FFT will actually be more flexible with respect to array size. All in all I think it is worth looking at. The C code is in the "Seismic U*nx" download from CWP, which is huge, but contains a lot of nice C library functions for scientific computing apart from FFTs as well (all of which are BSD licensed it seems). http://www.cwp.mines.edu/cwpcodes/index.html Regards, Sturla

1 2

Here's what I've done to numpy.fft
by David Goldsmith July 13, 2010

July 13, 2010

In light of my various questions and the responses thereto, here's what I've done (but not yet committed) to numpy.fft. There are many ways to define the DFT, varying in the sign of the exponent, normalization, etc. In this implementation, the DFT is defined as .. math:: A_k = \sum_{m=0}^{n-1} a_m \exp\left\{-2\pi i{mk \over n}\right\} \qquad k = 0,\ldots,n-1 where `n` is the number of input points. In general, the DFT is defined for complex inputs and outputs, and a single-frequency component at linear frequency :math:`f` is represented by a complex exponential :math:`a_m = \exp\{2\pi i\,f m\Delta t\}`, where :math:`\Delta t` is the *sampling interval*. Note that, due to the periodicity of the exponential function, formally :math:`A_{n-1} = A_{-1}, A_{n-2} = A_{-2}`, etc. That said, the values in the result are in the so-called "standard" order: if ``A = fft(a,n)``, then ``A[0]`` contains the zero-frequency term (the sum of the data), which is always purely real for real inputs. Then ``A[1:n/2]`` contains the positive-frequency terms, and ``A[n/2+1:]`` contains the negative-frequency (in the sense described above) terms, from least (most negative) to largest (closest to zero). In particular, for `n` even, ``A[n/2]`` represents both the positive and the negative Nyquist frequencies, and is also purely real for real input. For `n` odd, ``A[(n-1)/2]`` contains the largest positive frequency, while ``A[(n+1)/2]`` contains the largest (in absolute value) negative frequency. In both cases, i.e., `n` even or odd, ``A[n-1]`` contains the negative frequency closest to zero. Feedback welcome. DG

3 4