NumPy-Discussion June 2013

numpy-discussion@python.org

69 participants
60 discussions

Reading data from structured text
by klo uo 15 Jun '13

15 Jun '13

Hi, I have data reports in text files, where first 5 lines describe the data following, which is actually continuous time series of 2048 values wrapped in 205 rows and 10 columns, and each file has 12 such sets. If I crop to first dataset and leave the headers (first 5 lines), genfromtxt(skip_header=5) raises ValueError because last row contains 8 records instead 10 like the rest. If I just try to read file as is, with same argument, genfromtxt(skip_header=5), I get many errors as genfromtxt() doesn't seem to be able to detect header present on every dataset. So can I read such file with numpy and learn for future use, or should I just write my reader?

1 0

Generating special polynomials (Chebyshev, Hermite etc.)
by Kumar Appaiah 15 Jun '13

15 Jun '13

Dear Numpy Users, I am trying to find out a way by which I can easily generate the n-th order "special" polynomial, where "special" could refer to Hermite, Chebyshev etc. Numpy 1.7 introduces several methods for such polynomials, but I couldn't find a convenience function that gives me a polynomial directly based on degree. For instance, I'd like: hermite(3) to result in array([ 0., -12., 0., 8.]) hermite(6) to result in array([-120., 0., 720., 0., -480., 0., 64.]) and so on. The quickest way I could come up with for this is: def hermite(n): if n <= 0: return numpy.array([1.0]) coeff_polynomial = [0.0] * n coeff_polynomial.extend([1]) return numpy.polynomial.hermite.herm2poly(coeff_polynomial) Now, if I am missing something, please let me know. If you think this is a useful feature, I volunteer to patch all the polynomial modules to generate such polynomials, if you could tell me appropriate function names for such convenience functions. Thanks! Kumar -- Kumar Appaiah

2 4

Will transpose ever need to copy data?
by Jaime Fernández del Río 14 Jun '13

14 Jun '13

I noticed today that the documentation for np.transpose states, for the return value, that "A view is returned whenever possible." Is there really any situation where swapping axes around could trigger the need to copy data, or will a view always be returned no matter what? I can't think of any such situation, and was planning on relying on that for some code: basically, I have an output array, which I would like to be contiguous. So I preallocate it with the right shape, then take a view of it moving a certain axis to the end to make computations easier, run all my computations on the modified view, then return the original array, not the view. If I started with an array with the axis at the end, and then transposed it, I would need to make a copy to make it contiguous, which is what I am trying to avoid. Is this a bad practice? Is that precaution in the documentation real? Should I check that my view's base is the original array and trigger a copy, or is it an unnecessary check? Thanks in advance, Jaime -- (\__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

4 5

Dropping support for Accelerate/veclib?
by Pauli Virtanen 13 Jun '13

13 Jun '13

Hi, numpy.distutils provides support for Accelerate+veclib on OSX, but does not provide Fortran compiler flags that ensure that the Fortran ABI used is compatible. As a result ASUM, SDOT, (apparently also DDOT), and other common BLAS functions return garbage when called with the wrong ABI. This breaks parts of Scipy. (There's some code to hack around this --- it worked on earlier versions of OSX, but apparently is unreliable on 10.8, demonstrating that this approach is just too brittle.) Moreover, third-party module developers who use Fortran will not be aware of this issue, so it may also break 3rd party code. As far as I see, the options are: 1. Add -ff2c (or whatever is the correct flag for Accelerate) to the Fortran flags on OSX. 2. Drop support for Veclib/Accelerate. I think Accelerate was also incompatible with multiprocessing, which would weigh for option 2. -- Pauli Virtanen

5 9

strided copy unroll, benchmarks needed
by Julian Taylor 13 Jun '13

13 Jun '13

hi, I posted a pull with a minor change instructing the GCC compiler to unroll the strided copy loops (gcc will almost never do that on its own, not even on O3). https://github.com/numpy/numpy/pull/3429 It improves performance of these copies by 20%-50% depending on the size of the data copied (if it goes out of all cpu caches you don't gain anything anymore) on a couple machines (amd phenom x4, intel core2duo, xeon 7xxx/5xxx) As overriding the compiler decision is always dodgy, I would like some numbers on a couple of cpu types to decide if its really a good idea. So if you have the time please try the pull and the benchmark in the first comment and report the difference in performance between the pull and the unchanged numpy git head in the PR. please include your cpu, gcc version and architecture (32 bit or 64 bit). The benchmark can be run with ipython: irunner --ipython bench.py Cheers, Julian

1 0

Memory leak
by Pietro Bonfa' 13 Jun '13

13 Jun '13

Dear Numpy users, I have a memory leak in my code. A simple way to reproduce my problem is: import numpy class test(): def __init__(self): pass def t(self): temp = numpy.zeros([200,100,100]) A = numpy.zeros([200], dtype = numpy.float) for i in range(200): A[i] = numpy.sum( temp[i].diagonal() ) return A a = test() c = [a.t() for i in range(100)] Running this script will require 1.5 Gb of memory since the 16 mb of temp arrays are never deallocated. How can I solve this problem? Thanks in advances, Pietro Bonfa' P.S: I asked the same question also on stack overflow (http://stackoverflow.com/questions/17085197/is-this-a-memory-leak-python-nu… ) -- Pietro Bonfa' - PhD student Dipartimento di Fisica e Scienze della Terra "Macedonio Melloni" Viale delle Scienze 7A 43124 Parma - Italy

4 5

svd + multiprocessing hangs
by Uwe Schmitt 13 Jun '13

13 Jun '13

Dear all, the following code hangs on my Ubuntu machine. I use self compiled numpy 1.7.1 and Python 2.7.3 ----- import numpy import numpy.linalg import multiprocessing def classify(): X = numpy.random.random((10,3)) print "before svd" numpy.linalg.svd(X) print "this message is not printed" if __name__ == "__main__": p = multiprocessing.Process(target=classify, args=()) p.start() p.join() ----- Regards, Uwe. -- Dr. rer. nat. Uwe Schmitt Leitung F/E Mathematik mineway GmbH Gebäude 4 Im Helmerswald 2 66121 Saarbrücken Telefon: +49 (0)681 8390 5334 Telefax: +49 (0)681 830 4376 uschmitt(a)mineway.de www.mineway.de Geschäftsführung: Dr.-Ing. Mathias Bauer Amtsgericht Saarbrücken HRB 12339

4 4

supporting quad precision
by Anne Archibald 13 Jun '13

13 Jun '13

Hi folks, I recently came across an application I needed quad precision for (high-accuracy solution of a differential equation). I found a C++ library (odeint) that worked for the integration itself, but unfortunately it appears numpy is not able to work with quad precision arrays. For my application the quad precision is only really needed for integrator state, so I can manually convert my data to long doubles as I go to and from numpy, but it's a pain. So quad precision support would be nice. There's a thread discussing quad support: http://mail.scipy.org/pipermail/numpy-discussion/2012-February/061080.html Essentially, there isn't any, but since gcc >= 4.6 supports them on Intel hardware (in software), it should be possible. (Then the thread got bogged down in bike-shedding about what to call them.) What would it take to support quads in numpy? I looked into the numpy base dtype definitions, and it's a complex arrangement involving detection of platform support and templatized C code; in the end I couldn't really see where to start. But it seems to me all the basics are available: native C syntax for basic arithmetic, "qabs"-style versions of all the basic functions, NaNs and Infs. So how would one go about adding quad support? Thanks, Anne

5 10

Avoiding messy code in Python
by Bakhtiyor Zokhidov 13 Jun '13

13 Jun '13

Hi, I have some code more or less 500 lines, but very messy code. All codes containing several functions are in one module, besides, without documentation and testing. Could anyone give me some advice to organize my messy code in an accurate style including test function as well? Any answers will be appreciated, -- Bakhtiyor Zokhidov

3 2

Re: [Numpy-discussion] Weird problem with subtracting ndarrays
by Moroney, Catherine M (398D) 12 Jun '13

12 Jun '13

Thanks for the tip. I thought it must be something simple like that. When I convert the arrays to numpy.int32 things behave normally. Another question though: a numpy.all() on the signed-int arrays shows that they are equal. So why would the subtraction of the unsigned arrays wrap around like that? Catherine On Jun 12, 2013, at 12:25 PM, <numpy-discussion-request(a)scipy.org> wrote: > Message: 9 > Date: Wed, 12 Jun 2013 15:30:22 -0400 > From: Warren Weckesser <warren.weckesser(a)gmail.com> > Subject: Re: [Numpy-discussion] weird problem with subtracting > ndarrays > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: > <CAGzF1udhTAmxPWwHsd9xQ386CBz2iXymmYC4kJJ3AOTRPZQzyg(a)mail.gmail.com> > Content-Type: text/plain; charset="iso-8859-1" > > On Wed, Jun 12, 2013 at 3:25 PM, Moroney, Catherine M (398D) < > Catherine.M.Moroney(a)jpl.nasa.gov> wrote: > >> Hello, >> >> I've got two arrays of the same shape that I read in from a file, and I'm >> trying to >> difference them. Very simple stuff, but I'm getting weird answers. >> >> Here is the code: >> >>>>> counts1 = hfile1.read_grid_field("CFbA", >> "TerrainReferencedRCCMFraction_Num") >>>>> counts2 = hfile2.read_grid_field("CFbA", >> "TerrainReferencedRCCMFraction_Num") >>>>> counts1.max(), counts2.max() >> (13, 13) >>>>> counts1.min(), counts2.min() >> (0, 0) >>>>> numpy.all(counts1 == counts2) >> False >>>>> diff = counts1 - counts2 >>>>> diff.max() >> 4294967295 !! WHAT IS HAPPENING HERE ?? >>>>> sum = counts1 + counts2 >>>>> sum.max() >> 26 >> >> As you can see, the range of values in both arrays is 0 to 13, and the sum >> behaves normally, but the difference gives this weird number. >> >> When I create dummy arrays, the subtraction works fine. So there must be >> some funny value >> lurking in either the counts1 or counts2 array, but the numpy.isnan() test >> returns False. >> >> Any ideas for how I debug this? >> >> Catherine >> >> > Check the dtype of the arrays. They are probably unsigned integers, and > the subtraction leads to wrap-around in some cases. > > For example: > > In [1]: x = np.array([0, 1, 2], dtype=np.uint32) > > In [2]: y = np.array([1, 1, 1], dtype=np.uint32) > > In [3]: x - y > Out[3]: array([4294967295, 0, 1], dtype=uint32) > > > Warren > >

2 1

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

2005

2004

2003

2002

2001

2000

NumPy-Discussion June 2013