Mailman 3 February 2010 - NumPy-Discussion

Why does np.nan{min, max} clobber my array mask?
by David Carmean 05 Mar '10

05 Mar '10

I'm just starting to work with masked arrays and I've found some behavior that definitely does not follow the Principle of Least Surprise: I've generated a 2-d array from a list of lists, where the elements are floats with a good number of NaNs. Inspections shows the expected numbers for ma.count() and ma.count_masked(). However, as soon as I run np.nanmin() or np.nanmax() over it, all of the mask elements are reset to False. (Pdb) flat = flatten(uut) # my own utility function (Pdb) len ( [ x for x in flat if x+0 == x ] ) # only way I could figure to detect 4086 (Pdb) len ( [ x for x in flat if x+0 != x ] ) # 1458 NaNs in the set. 1458 (Pdb) msk = ma.masked_invalid(uut) (Pdb) msk.shape (99, 56) (Pdb) ma.count(msk) 4086 (Pdb) ma.count_masked(msk) 1458 (Pdb) msk.hardmask False (Pdb) msk.harden_mask() # harden the mask first, for demo masked_array(data =.... (Pdb) msk.hardmask True (Pdb) rslt_hm = np.nanmin(msk, axis=1) (Pdb) rslt_hm.shape (99,) (Pdb) ma.count_masked(rslt_hm) 0 (Pdb) ma.count(rslt_hm) 99 # Is my original still OK? msk masked_array(data = ... ... [False False False ..., True True True]], fill_value = 1e+20) (Pdb) msk.soften_mask() # now re-soften the mask: masked_array(data = .... (Pdb) rslt_softmask = np.nanmin(msk, axis=1) (Pdb) rslt_softmask.shape (99,) (Pdb) msk.mask.any() False # BAM! note: 'control' is a hardmasked control copy: (Pdb) control.mask.any() True As the above shows, I discovered that I can work around this by setting the hardmask property, but ... there is no mention of such a side-effect in the docs (including the brand-new reference book). Have I found a bug? This is 1.4.0 running under 64-bit Windows 7 ( Python(x,y) distribution).

3 5

Snow Leopard
by Charles سمير Doutriaux 02 Mar '10

02 Mar '10

Hi there, I'm having a lot of trouble on Snow Leopard. I'm trying to build a 32bit only version of our system.... Turns out I had to use python 2.7a3 in order to get Python --universalsdk to work.... But now numpy fails to build as well... First because of the: >> _old_init_posix = distutils.sysconfig._init_posix error I got it to use sysconfig instead But now when running python setup.py build install I get: creating build/src.macosx-10.3-fat-2.7 creating build/src.macosx-10.3-fat-2.7/numpy creating build/src.macosx-10.3-fat-2.7/numpy/distutils building library "npymath" sources customize NAGFCompiler Could not locate executable f95 customize AbsoftFCompiler Could not locate executable f90 Could not locate executable f77 customize IBMFCompiler Could not locate executable xlf90 Could not locate executable xlf customize IntelFCompiler Could not locate executable ifort Could not locate executable ifc customize GnuFCompiler Could not locate executable g77 customize Gnu95FCompiler Could not locate executable gfortran customize G95FCompiler Could not locate executable g95 don't know how to compile Fortran code on platform 'posix' C compiler: gcc -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -arch i386 -m32 compile options: '-Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/lgm/cdat/trunk/include/python2.7 -c' gcc: _configtest.c gcc _configtest.o -o _configtest ld: warning: in _configtest.o, file is not of required architecture Undefined symbols: "_main", referenced from: __start in crt1.o ld: symbol(s) not found collect2: ld returned 1 exit status ld: warning: in _configtest.o, file is not of required architecture Undefined symbols: "_main", referenced from: __start in crt1.o ld: symbol(s) not found collect2: ld returned 1 exit status failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 210, in <module> setup_package() File "setup.py", line 203, in setup_package configuration=configuration ) File "/svn/cdat/trunk/build/numpy/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/lgm/cdat/trunk/lib/python2.7/distutils/core.py", line 152, in setup dist.run_commands() File "/lgm/cdat/trunk/lib/python2.7/distutils/dist.py", line 953, in run_commands self.run_command(cmd) File "/lgm/cdat/trunk/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/svn/cdat/trunk/build/numpy/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/lgm/cdat/trunk/lib/python2.7/distutils/command/build.py", line 127, in run self.run_command(cmd_name) File "/lgm/cdat/trunk/lib/python2.7/distutils/cmd.py", line 326, in run_command self.distribution.run_command(command) File "/lgm/cdat/trunk/lib/python2.7/distutils/dist.py", line 972, in run_command cmd_obj.run() File "/svn/cdat/trunk/build/numpy/numpy/distutils/command/build_src.py", line 152, in run self.build_sources() File "/svn/cdat/trunk/build/numpy/numpy/distutils/command/build_src.py", line 163, in build_sources self.build_library_sources(*libname_info) File "/svn/cdat/trunk/build/numpy/numpy/distutils/command/build_src.py", line 298, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "/svn/cdat/trunk/build/numpy/numpy/distutils/command/build_src.py", line 385, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 670, in get_mathlib_info raise RuntimeError("Broken toolchain: cannot link a simple C program") RuntimeError: Broken toolchain: cannot link a simple C program Note: it's saying soemtihng about mac 10.3.... Any idea on how to build a pure 32bit numpy on snow leopard? Thx, C.

3 3

Building Numpy Windows Superpack
by Patrick Marsh 01 Mar '10

01 Mar '10

Greetings, I have been trying to build the numpy superpack on windows using the binaries posted by David. Unfortunately, I haven't even been able to correctly write the site.cfg file to locate all three sets of binaries needed for the superpack. When I manually specify to use only the sse3 binaries, I can get numpy to build from trunk, but it fails miserably when running the test suite. In fact, in python26, the tests freeze python and causes it to exit. I figured I'd try to get this set up correctly before even trying to compile the cpucaps nsis plugin. If someone has successfully used David's binaries would they be willing to share their site.cfg? Thanks in advance. Patrick -- Patrick Marsh Ph.D. Student / NSSL Liaison to the HWT School of Meteorology / University of Oklahoma Cooperative Institute for Mesoscale Meteorological Studies National Severe Storms Laboratory http://www.patricktmarsh.com

4 4

[ANN]: Taylorpoly, an implementation of vectorized Taylor polynomial operations and request for opinions
by Sebastian Walter 01 Mar '10

01 Mar '10

Announcement: ----------------------- I have started to implement vectorized univariate truncated Taylor polynomial operations (add,sub,mul,div,sin,exp,...) in ANSI-C. The interface to python is implemented by using numpy.ndarray's ctypes functionality. Unit tests are implement using nose. It is BSD licencsed and hosted on github: http://github.com/b45ch1/taylorpoly Rationale: ------------------------ A truncated Taylor polynomial is of the form [x]_d = \sum_{d=0}^{D-1} x_d t^d where x_d a real number and t an external parameter. Truncated Taylor polynomials should not be confused with polynomials for interpolation, as it is implemented in numpy.polynomial. The difference is that Taylor polynomials, as implemented in `taylorpoly`, are polynomials in an "external parameter". I.e. they are *never* evaluated for some t. One should think of this algebraic class as an extension of the real numbers ( e.g. similarly to the complex numbers). At the moment, I am not aware of any such algorithms available in Python. I believe algorithms for this algebraic structure are a very important building block for more sophisticated algorithms, e.g. for Algorithmic Differentiation (AD) tools and Taylor polynomial integrators. More Detailed Description: --------------------------------------- The operations add,mul,etc. are extended to compute on truncated Taylor polynomials, e.g. for the multiplication [z]_{D+1} &=_{D+1}& [x]_{D+1} y_{D+1} \\ \sum_{d=0}^D z_d t^d & =_{D+1} & \left( \sum_{d=0}^D x_d t^d \right) \left( \sum_{c=0}^D y_c t^c \right) where D+1 is the degree of the polynomial. The coefficients z_d are only evaluated up to degree D+1, i.e. higher orders are truncated. Request for Opinions: ------------------------------- Before I continue implementing more algorithms, it would be nice to have some feedback. There are things I'm still not sure about: 1) data structure for vectorized operations: For the non-vectorized algorithms, it is quite clear that the coefficients of a Taylor polynomial \sum_{d=0}^{D-1} x_d t^d are stored in an 1D array [x_0,x_1,...,x_{D-1}]: In the vectorized version, P different Taylor polynomials are computed at once, but the base coefficient x_0 is the same for all of them. At the moment I have implemented the data structure: [x]_{D,P} := [x_0, x_{1,1},...,x_{1,D-1},x_{2,1},...,x_{P,D-1}]. Another possibility would be: [x] = [x_0, x_{1,1}, ..., x_{1,P}, x_{2, 1}, ..., x_{D-1, P}] Any thoughts about which to choose? The first version is much easier to implement. The second is possibly easier to vectorize by a compiler. 2) implementation of binary operators in Python: I have defined a class UTPS (univariate Taylor polynomial over Scalar) that basically only stores the above array [x]_{D,P} in the attribute `UTPS.data` and the algorithms add,sub,mul,div take instances of the class UTPS. I.e. the functions are not implemented as member functions. I plan to add member functions later for convenience that call those functions. Is this is a good idea? I think it would be good to be consistent with numpy. 3) coding style of the algorithms in C I'm making heavy use of pointer arithmetic, rendering the algorithms a little hard to write and to understand. The alternative would be array indexing with address computations. Does anyone know how good the compilers are nowadays at optimizing away the address computations? I'd be very happy to get some feedback. regards, Sebastian

2 12

Speedup a code using apply_along_axis
by Xavier Gnata 01 Mar '10

01 Mar '10

Hi, I'm sure I reinventing the wheel with the following code: from numpy import * from scipy import polyfit,stats def f(x,y,z): return x+y+z M=fromfunction(f,(2000,2000,10)) def foo(M): ramp=where(M<1000)[0] l=len(ramp) t=arange(l) if(l>1): return polyfit(t,ramp,1)[0] else: return 0 print apply_along_axis(foo,2,M) In real life M is not the result of one fromfunction call but it does not matter. The basic idea is to compute the slope (and only the slope) along one axis of 3D array. Only the values below a given threshold should be taken into account. The current code is ugly and slow. How to remove the len and the if statement? How to rewrite the code in a numpy oriented way? Xavier

2 2

SciPy Mail List and Contacting Dave Kuhlman
by Wayne Watson 01 Mar '10

01 Mar '10

3 2

Reading and Comparing Two Files
by Robert Love 01 Mar '10

01 Mar '10

What is the efficient numpy way to compare data from two different files? For the nth line in each file I want to operate on the numbers. I've been using loadtxt() data_5x5 = N.loadtxt("file5") data_8x8 = N.loadtxt("file8") for line in data_5x5: pos5 = N.array([line[0], line[1], line[2]]) This works fine for one file but how to I get the same line's worth of data from the other file?

3 3

Apply a function to all indices
by Ole Streicher 01 Mar '10

01 Mar '10

Hi, I want to apply a function to all indices of an array that fullfill a certain condition. What I tried: ---------------------8<-------------------------------- import numpy def myfunc(x): print 'myfunc of', x a = numpy.random.random((2,3,4)) numpy.apply_along_axis(myfunc, 0, numpy.where(a > 0.8)) ---------------------8<-------------------------------- But this prints just the first index vector and then shows a TypeError: object of type 'NoneType' has no len() What is wrong with my code and how can I do it right? Best regards Ole

4 7

Re: [Numpy-discussion] Python 3 porting
by Pauli Virtanen 28 Feb '10

28 Feb '10

Hi, The test suite passes now on Pythons 2.4 - 3.1. Further testing is very welcome -- also on Python 2.x. Please check that your favourite software still builds and works with SVN trunk Numpy. Currently, Scipy has some known failures because of (i) removed new= keyword in numpy.histogram (ii) Cython supports only native size/alignment PEP 3118 buffers, and Numpy arrays are most naturally expressed in the standardized sizes. Supporting the full struct module alignment stuff appears to be a slight PITA. I'll try to take a look at how to address this. But everything else seems to work on Python 2.6. *** Python version 2.4.6 (#2, Jan 21 2010, 23:27:36) [GCC 4.4.1] Ran 2509 tests in 18.892s OK (KNOWNFAIL=4, SKIP=2) Python version 2.5.4 (r254:67916, Jan 20 2010, 21:44:03) [GCC 4.4.1] Ran 2512 tests in 18.531s OK (KNOWNFAIL=4) Python version 2.6.4 (r264:75706, Dec 7 2009, 18:45:15) [GCC 4.4.1] Ran 2519 tests in 19.367s OK (KNOWNFAIL=4) Python version 3.1.1+ (r311:74480, Nov 2 2009, 14:49:22) [GCC 4.4.1] Ran 2518 tests in 23.239s OK (KNOWNFAIL=5) Cheers, Pauli

4 4

Sorting objects with ndarrays
by Gael Varoquaux 28 Feb '10

28 Feb '10

Hi, I need to have list of objects that contain ndarrays to be sorted. The reason that I want them sorted is that these list are populated in an arbitrary order, but there order really doesn't matter, and I am trying to make it reproducible for debugging and hashing. The problem is that ndarrays cannot be compared. So I have tried to override the 'cmp' in the 'sorted' function, however I am comparing fairly complex objects, and I am having a hard time predicting wich member of the object will contain the array. So I am building a more and more complex 'cmp' replacement. Does anybody has a good idea what a better strategy would be? Cheers, Gaël

3 4