Mailman 3 September 2009 - NumPy-Discussion

masked arrays of structured arrays
by Ernest Adrogué Sept. 2, 2009

Sept. 2, 2009

Hi there, Here is a structured array with 3 fields each of which has 3 fields in turn: In [3]: desc = [('a',int), ('b',int), ('c',int)] In [4]: desc = [('x',desc), ('y',desc), ('z',desc)] With a regular ndarray it works just fine: In [11]: x = np.zeros(2, dtype=desc) In [12]: x['x']['b'] = 2 In [13]: x['x']['b'] Out[13]: array([2, 2]) However if try the same with a masked array, it fails: In [14]: x = np.ma.masked_all(2, dtype=desc) In [15]: x['x']['b'] = 2 --------------------------------------------------------------------------- ValueError Traceback (most recent call last) /home/ernest/<ipython console> in <module>() /usr/lib/python2.5/site-packages/numpy/ma/core.pyc in __setitem__(self, indx, value) 1574 if self._mask is nomask: 1575 self._mask = make_mask_none(self.shape, self.dtype) -> 1576 ndarray.__setitem__(self._mask, indx, getmask(value)) 1577 return 1578 #........................................ ValueError: field named b not found. Any idea of what the problem is? Ernest

2 4

numpy on Snow Leopard
by Celil Rufat Sept. 2, 2009

Sept. 2, 2009

I am unable to build numpy on Snow Leopard. The error that I am getting is shown below. It is a linking issue related to the change in the the default behavior of gcc under Snow Leopard. Before it used to compile for the 32 bit i386 architecture, now the default is the 64 bit x86_64 architecture. Has anybody successfully compiled numpy for MACOSX 10.6. If so I would appreciate if you can tell me how you fixed this issue. Regards, Celil ... C compiler: gcc -arch ppc -arch i386 -isysroot /Developer/SDKs/MacOSX10.4u.sdk -fno-strict-aliasing -fno-common -dynamic -DNDEBUG -g -O3 ... gcc: _configtest.c _configtest.c:1: warning: conflicting types for built-in function ‘exp’ _configtest.c:1: warning: conflicting types for built-in function ‘exp’ gcc _configtest.o -o _configtest ld: warning: in _configtest.o, missing required architecture x86_64 in file Undefined symbols: "_main", referenced from: start in crt1.10.6.o ld: symbol(s) not found collect2: ld returned 1 exit status ld: warning: in _configtest.o, missing required architecture x86_64 in file Undefined symbols: "_main", referenced from: start in crt1.10.6.o ld: symbol(s) not found collect2: ld returned 1 exit status failure.

1 0

np.bitwise_and.identity
by Citi, Luca Sept. 2, 2009

Sept. 2, 2009

Hello, I know I am splitting the hair, but should not np.bitwise_and.identity be -1 instead of 1? I mean, something with all the bits set? I am checking whether all elements of a vector 'v' have a certain bit 'b' set: if np.bitwise_and.reduce(v) & (1 << b): # do something If v is empty, the expression is true for b==0 and false otherwise. In fact np.bitwise_and.identity is 1. I like being able to use np.bitwise_and.reduce because it many times faster than (v & (1 << b)).all() (it does not create the temporary vector). Of course there are workarounds but I was wondering if there is a reason for this behaviour. Best, Luca

2 2

How to concatenate two arrays without duplicating memory?
by "V. Armando Solé" Sept. 2, 2009

Sept. 2, 2009

Hello, Let's say we have two arrays A and B of shapes (10000, 2000) and (10000, 4000). If I do C=numpy.concatenate((A, B), axis=1), I get a new array of dimension (10000, 6000) with duplication of memory. I am looking for a way to have a non contiguous array C in which the "left" (10000, 2000) elements point to A and the "right" (10000, 4000) elements point to B. Any hint will be appreciated. Thanks, Armando

5 8

Question about np.savez
by jorgesmbox-ml＠yahoo.es Sept. 2, 2009

Sept. 2, 2009

Hi, I know the documentation states that np.savez saves numpy arrays, so my question relates to misusing it. Before reading the doc in detail, and after reading about pickle and other options to make data persistent, I passed np.savez a list of ndarrays. It didn't complain, but when I loaded the data back, the list had been turned into an ndarray. Is this behaviour expected? It did surprise me. Below there is an example: In [18]: l Out[18]: [array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]), array([ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18]), array([ 0, 3, 6, 9, 12, 15, 18, 21, 24, 27])] In [19]: np.savez('test.npz', l=l) In [20]: data = np.load('test.npz') In [21]: l1 = data['l'] In [22]: l1 Out[22]: array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9], [ 0, 2, 4, 6, 8, 10, 12, 14, 16, 18], [ 0, 3, 6, 9, 12, 15, 18, 21, 24, 27]]) jorge

6 8

numpy rev7353 w/ OS X 10.6
by Christopher Hanley Sept. 2, 2009

Sept. 2, 2009

Hi, Upgraded to Snow Leopard, left setup.py and all environment variables the same, tried latest numpy from source. This is the build error I receive: Running from numpy source directory. non-existing path in 'numpy/distutils': 'site.cfg' F2PY Version 2_7353 numpy/core/setup_common.py:81: MismatchCAPIWarning: API mismatch detected, the C API version numbers have to be updated. Current C api version is 3, with checksum 4526bc5a07e51d13a2db642715fedca5, but recorded checksum for C API version 3 in codegen_dir/cversions.txt is bf22c0d05b31625d2a7015988d61ce5a. If functions were added in the C API, you have to update C_API_VERSION in numpy/core/setup_common.pyc. MismatchCAPIWarning) blas_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-msse3', '-I/System/Library/Frameworks/vecLib.framework/Headers'] lapack_opt_info: FOUND: extra_link_args = ['-Wl,-framework', '-Wl,Accelerate'] define_macros = [('NO_ATLAS_INFO', 3)] extra_compile_args = ['-msse3'] running install running build running config_cc unifing config_cc, config, build_clib, build_ext, build commands --compiler options running config_fc unifing config_fc, config, build_clib, build_ext, build commands --fcompiler options running build_src build_srca building py_modules sources creating build creating build/src.macosx-10.3-i386-2.5 creating build/src.macosx-10.3-i386-2.5/numpy creating build/src.macosx-10.3-i386-2.5/numpy/distutils building library "npymath" sources customize NAGFCompiler Could not locate executable f95 Found executable /usr/local/bin/gfortran nag: no Fortran 90 compiler found Could not locate executable ranlib nag: no Fortran 90 compiler found nag: no Fortran 90 compiler found customize AbsoftFCompiler Could not locate executable f90 absoft: no Fortran 90 compiler found absoft: no Fortran 90 compiler found absoft: no Fortran 90 compiler found absoft: no Fortran 90 compiler found absoft: no Fortran 90 compiler found absoft: no Fortran 90 compiler found customize IBMFCompiler Could not locate executable xlf90 ibm: no Fortran 90 compiler found Could not locate executable xlf95 ibm: no Fortran 90 compiler found customize IntelFCompiler Could not locate executable ifort Could not locate executable ifc intel: no Fortran 90 compiler found intel: no Fortran 90 compiler found customize GnuFCompiler gnu: no Fortran 90 compiler found Found executable /usr/local/bin/g77 gnu: no Fortran 90 compiler found /Users/chanley/dev/numpy/numpy/distutils/fcompiler/gnu.py:131: UserWarning: Env. variable MACOSX_DEPLOYMENT_TARGET set to 10.3 warnings.warn(s) customize Gnu95FCompiler Found executable /usr/local/bin/gfortran customize Gnu95FCompiler customize Gnu95FCompiler using config C compiler: cc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/usr/stsci/pyssgdev/Python-2.5.1/include/python2.5 -c' cc: _configtest.c sh: cc: command not found sh: cc: command not found failure. removing: _configtest.c _configtest.o C compiler: cc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/usr/stsci/pyssgdev/Python-2.5.1/include/python2.5 -c' cc: _configtest.c sh: cc: command not found sh: cc: command not found failure. removing: _configtest.c _configtest.o C compiler: cc -fno-strict-aliasing -Wno-long-double -no-cpp-precomp -mno-fused-madd -DNDEBUG -g -O3 -Wall -Wstrict-prototypes compile options: '-Inumpy/core/src -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/include -I/usr/stsci/pyssgdev/Python-2.5.1/include/python2.5 -c' cc: _configtest.c sh: cc: command not found sh: cc: command not found failure. removing: _configtest.c _configtest.o Traceback (most recent call last): File "setup.py", line 186, in <module> setup_package() File "setup.py", line 179, in setup_package configuration=configuration ) File "/Users/chanley/dev/numpy/numpy/distutils/core.py", line 186, in setup return old_setup(**new_attr) File "/usr/stsci/pyssgdev/Python-2.5.1/lib/python2.5/distutils/core.py", line 151, in setup dist.run_commands() File "/usr/stsci/pyssgdev/Python-2.5.1/lib/python2.5/distutils/dist.py", line 974, in run_commands self.run_command(cmd) File "/usr/stsci/pyssgdev/Python-2.5.1/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/Users/chanley/dev/numpy/numpy/distutils/command/install.py", line 52, in run r = old_install.run(self) File "/usr/stsci/pyssgdev/Python-2.5.1/lib/python2.5/distutils/command/install.py", line 506, in run self.run_command('build') File "/usr/stsci/pyssgdev/Python-2.5.1/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/stsci/pyssgdev/Python-2.5.1/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/Users/chanley/dev/numpy/numpy/distutils/command/build.py", line 37, in run old_build.run(self) File "/usr/stsci/pyssgdev/Python-2.5.1/lib/python2.5/distutils/command/build.py", line 112, in run self.run_command(cmd_name) File "/usr/stsci/pyssgdev/Python-2.5.1/lib/python2.5/distutils/cmd.py", line 333, in run_command self.distribution.run_command(command) File "/usr/stsci/pyssgdev/Python-2.5.1/lib/python2.5/distutils/dist.py", line 994, in run_command cmd_obj.run() File "/Users/chanley/dev/numpy/numpy/distutils/command/build_src.py", line 151, in run self.build_sources() File "/Users/chanley/dev/numpy/numpy/distutils/command/build_src.py", line 162, in build_sources self.build_library_sources(*libname_info) File "/Users/chanley/dev/numpy/numpy/distutils/command/build_src.py", line 297, in build_library_sources sources = self.generate_sources(sources, (lib_name, build_info)) File "/Users/chanley/dev/numpy/numpy/distutils/command/build_src.py", line 384, in generate_sources source = func(extension, build_dir) File "numpy/core/setup.py", line 590, in get_mathlib_info mlibs = check_mathlib(config_cmd) File "numpy/core/setup.py", line 283, in check_mathlib raise EnvironmentError("math library missing; rerun " EnvironmentError: math library missing; rerun setup.py after setting the MATHLIB env variable Does Python need to be rebuild? I was using 2.5 I downloaded from python.org. Thanks, Chris -- Christopher Hanley Senior Systems Software Engineer Space Telescope Science Institute 3700 San Martin Drive Baltimore MD, 21218 (410) 338-4338

2 2

ticket for O(n) median function?
by Sturla Molden Sept. 1, 2009

Sept. 1, 2009

I could not find any, so I'll ask if it's ok to create one. I have a patch for /numpy/lib/function_base.py that uses any 'select' function to obtain the median. I'll also submit the Cython code for quickselect. Attachment (median.py.gz) contains the suggested implementation of median. I disabled overwrite_input because the median function calls numpy.apply_along_axis. Regards, Sturla Molden

3 2

np.hist with masked values
by Tim Michelsen Sept. 1, 2009

Sept. 1, 2009

Hello, should creating a histogram with masked data be different that one cretated with unmasked data? Is np.hist tuned for work with historams? I R_project I would do: # Note: values is my dataset ### masking zeros values_mask=ifelse(values==0, NA, (values)) # http://stat.ethz.ch/R-manual/R-patched/library/stats/html/ecdf.html plot(ecdf(values_mask), main="Emprical cumulative distribution function", do.points=F, xlab='Data Values') There masking the zeros has an effect. With pure numpy not. Thanks in advance for any comment. Timmie

2 1

genfromtext advice
by Tim Michelsen Sept. 1, 2009

Sept. 1, 2009

Hello, I tried to load a ASCII table into a string array. Unfortunately, this table has some empty chells Here it is: http://www.ncdc.noaa.gov/oa/climate/rcsg/cdrom/ismcs/alphanum.html After having converted this into a text file I tried this: $ np.genfromtxt('alphanum_to-text.txt', dtype=np.str_, delimiter='|', skiprows=1, missing='') But I got an error: 1043 dtype = np.dtype(ttype) 1044 # -> 1045 output = np.array(data, dtype) 1046 if usemask: 1047 if dtype.names: ValueError: setting an array element with a sequence I sometimes experience this error message with text read in. Could this message be made more helpful like telling in which line of the input file this occurs? This could really reduce the trial and error. What would be the correct way to read this data in? Thanks, Timmie

4 7

Facing the Multicore-Challenge, Heidelberg 2010
by Chris Kees Sept. 1, 2009

Sept. 1, 2009

This conference may be of interest given the many discussions at SciPy on python support for parallel programming: http://www.multicore-challenge.org

1 0