Mailman 3 October 2011 - NumPy-Discussion

[ANN] Multifactor analysis tool for experiment planning
by Dmitrey 24 Oct '11

24 Oct '11

Hi all, > > new OpenOpt feature is available: Multifactor analysis tool for experiment planning (in physics, chemistry, biology etc). It is based on numerical optimization solver BOBYQA, released in 2009 by Michael J.D. Powell, and has easy and convenient GUI frontend, written in Python + tkinter. Maybe other (alternative) engines will be available in future. > > See its webpage for details. > > Regards, Dmitrey. >

1 0

ANN: MDP 3.2 released!
by Tiziano Zito 24 Oct '11

24 Oct '11

We are glad to announce release 3.2 of the Modular toolkit for Data Processing (MDP). MDP is a Python library of widely used data processing algorithms that can be combined according to a pipeline analogy to build more complex data processing software. The base of available algorithms includes signal processing methods (Principal Component Analysis, Independent Component Analysis, Slow Feature Analysis), manifold learning methods ([Hessian] Locally Linear Embedding), several classifiers, probabilistic methods (Factor Analysis, RBM), data pre-processing methods, and many others. What's new in version 3.2? -------------------------- - improved sklearn wrappers - update sklearn, shogun, and pp wrappers to new versions - do not leave temporary files around after testing - refactoring and cleaning up of HTML exporting features - improve export of signature and doc-string to public methods - fixed and updated FastICANode to closely resemble the original Matlab version (thanks to Ben Willmore) - support for new numpy version - new NeuralGasNode (thanks to Michael Schmuker) - several bug fixes and improvements We recommend all users to upgrade. Resources --------- Download: http://sourceforge.net/projects/mdp-toolkit/files Homepage: http://mdp-toolkit.sourceforge.net Mailing list: http://lists.sourceforge.net/mailman/listinfo/mdp-toolkit-users Acknowledgments --------------- We thank the contributors to this release: Michael Schmuker, Ben Willmore. The MDP developers, Pietro Berkes Zbigniew Jędrzejewski-Szmek Rike-Benjamin Schuppner Niko Wilbert Tiziano Zito

1 0

array iterators and cython prange
by Nadav Horesh 23 Oct '11

23 Oct '11

I coded a bilateral filter class in cython based on numpy's neighborhood iterator (thanks to T. J's code example). I tried to parallel the code by replacing the standard loop (commented line 150) by a prange loop (line 151). The result are series of compilation errors mainly due the the use of iterators. Is there an *easy* way to work with nmpy iterators while the GIL is released? Platfoem: numpy 1.6.1 on python 2.7.2 and cython 0.15.1 system: gcc on linux Nadav

1 0

Re: [Numpy-discussion] [SciPy-Dev] numpy.test hangs
by akshar bhosale 23 Oct '11

23 Oct '11

Hi, i changed site.cfg : ####site.cfg#### [MKL} mkl_libs = mkl_def, mkl_intel_lp64, mkl_intel_thread, mkl_core, mkl_mc lapack_libs = mkl_lapack95_lp64, mkl_lapack, mkl_scalapack_ilp64, mkl_scalapack_lp64 #lapack_libs =mkl_lapack,mkl_scalapack_ilp64,mkl_scalapack_lp64,mkl_lapack95_lp64 library_dirs = /opt/intel/Compiler/11.0/069/mkl/lib/em64t:/opt/intel/Compiler/11.0/069/lib/intel64/ include_dirs = /opt/intel/Compiler/11.0/069/mkl/include:/opt/intel/Compiler/11.0/069/include/ and now numpy.test hangs at : python Python 2.6 (r26:66714, May 29 2011, 15:10:47) [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import numpy >>> numpy.test(verbose=3) Running unit tests for numpy NumPy version 1.6.0 NumPy is installed in /home/aksharb/Python-2.6/lib/python2.6/site-packages/numpy Python version 2.6 (r26:66714, May 29 2011, 15:10:47) [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] nose version 1.0.0 nose.config: INFO: Excluding tests matching ['f2py_ext', 'f2py_f90_ext', 'gen_ext', 'pyrex_ext', 'swig_ext'] nose.selector: INFO: /home/external/unipune/gadre/Python-2.6/lib/python2.6/site-packages/numpy/core/multiarray.so is executable; skipped nose.selector: INFO: /home/external/unipune/gadre/Python-2.6/lib/python2.6/site-packages/numpy/core/scalarmath.so is executable; skipped nose.selector: INFO: /home/external/unipune/gadre/Python-2.6/lib/python2.6/site-packages/numpy/core/umath.so is executable; skipped nose.selector: INFO: /home/external/unipune/gadre/Python-2.6/lib/python2.6/site-packages/numpy/core/multiarray_tests.so is executable; skipped nose.selector: INFO: /home/external/unipune/gadre/Python-2.6/lib/python2.6/site-packages/numpy/core/umath_tests.so is executable; skipped nose.selector: INFO: /home/external/unipune/gadre/Python-2.6/lib/python2.6/site-packages/numpy/fft/fftpack_lite.so is executable; skipped nose.selector: INFO: /home/external/unipune/gadre/Python-2.6/lib/python2.6/site-packages/numpy/linalg/lapack_lite.so is executable; skipped nose.selector: INFO: /home/external/unipune/gadre/Python-2.6/lib/python2.6/site-packages/numpy/random/mtrand.so is executable; skipped test_api.test_fastCopyAndTranspose ... ok test_arrayprint.TestArrayRepr.test_nan_inf ... ok test_str (test_arrayprint.TestComplexArray) ... ok Ticket 844. ... ok test_blasdot.test_blasdot_used ... ok test_blasdot.test_dot_2args ... ok test_blasdot.test_dot_3args ... ok test_blasdot.test_dot_3args_errors ... ok . . . Test if an appropriate exception is raised when passing bad values to ... ok Test whether equivalent subarray dtypes hash the same. ... ok Test whether different subarray dtypes hash differently. ... ok Test some data types that are equal ... ok Test some more complicated cases that shouldn't be equal ... ok Test some simple cases that shouldn't be equal ... ok test_single_subarray (test_dtype.TestSubarray) ... ok test_einsum_errors (test_einsum.TestEinSum) ... ok test_einsum_sums_cfloat128 (test_einsum.TestEinSum) ... ####################### On Sat, Oct 22, 2011 at 1:46 PM, <numpy-discussion-owner(a)scipy.org> wrote: > This is a members-only list. Your message has been automatically > rejected, since it came from a non-member's email address. Please > make sure to use the email account that you used to join this list. > > > > ---------- Forwarded message ---------- > From: akshar bhosale <akshar.bhosale(a)gmail.com> > To: SciPy Developers List <scipy-dev(a)scipy.org>, Discussion of Numerical > Python <numpy-discussion(a)scipy.org> > Date: Sat, 22 Oct 2011 13:46:49 +0530 > Subject: Re: [SciPy-Dev] numpy.test hangs > <charlesr.harris(a)gmail.com>Hi, > > python > Python 2.6 (r26:66714, May 29 2011, 15:10:47) > [GCC 4.1.2 20071124 (Red Hat 4.1.2-42)] on linux2 > Type "help", "copyright", "credits" or "license" for more information. > >>> import numpy > >>> numpy.show_config() > lapack_opt_info: > libraries = ['mkl_lapack95_lp64', 'mkl_def', 'mkl_intel_lp64', > 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'pthread'] > library_dirs = ['/opt/intel/Compiler/11.0/069/mkl/lib/em64t'] > define_macros = [('SCIPY_MKL_H', None)] > include_dirs = ['/opt/intel/Compiler/11.0/069/mkl/include', > '/opt/intel/Compiler/11.0/069/include/'] > blas_opt_info: > libraries = ['mkl_def', 'mkl_intel_lp64', 'mkl_intel_thread', > 'mkl_core', 'mkl_mc', 'pthread'] > library_dirs = ['/opt/intel/Compiler/11.0/069/mkl/lib/em64t'] > define_macros = [('SCIPY_MKL_H', None)] > include_dirs = ['/opt/intel/Compiler/11.0/069/mkl/include', > '/opt/intel/Compiler/11.0/069/include/'] > lapack_mkl_info: > libraries = ['mkl_lapack95_lp64', 'mkl_def', 'mkl_intel_lp64', > 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'pthread'] > library_dirs = ['/opt/intel/Compiler/11.0/069/mkl/lib/em64t'] > define_macros = [('SCIPY_MKL_H', None)] > include_dirs = ['/opt/intel/Compiler/11.0/069/mkl/include', > '/opt/intel/Compiler/11.0/069/include/'] > blas_mkl_info: > libraries = ['mkl_def', 'mkl_intel_lp64', 'mkl_intel_thread', > 'mkl_core', 'mkl_mc', 'pthread'] > library_dirs = ['/opt/intel/Compiler/11.0/069/mkl/lib/em64t'] > define_macros = [('SCIPY_MKL_H', None)] > include_dirs = ['/opt/intel/Compiler/11.0/069/mkl/include', > '/opt/intel/Compiler/11.0/069/include/'] > mkl_info: > libraries = ['mkl_def', 'mkl_intel_lp64', 'mkl_intel_thread', > 'mkl_core', 'mkl_mc', 'pthread'] > library_dirs = ['/opt/intel/Compiler/11.0/069/mkl/lib/em64t'] > define_macros = [('SCIPY_MKL_H', None)] > include_dirs = ['/opt/intel/Compiler/11.0/069/mkl/include', > '/opt/intel/Compiler/11.0/069/include/'] > > Akshar > > > On Sat, Oct 22, 2011 at 11:54 AM, akshar bhosale <akshar.bhosale(a)gmail.com > > wrote: > >> yes sure.. >> i have intel cluster toolkit installed on my system. (11/069 version and >> mkl 10.3). i have machine having intel xeon processor and rhel 5.2 x86_64 >> platform. i am trying with intel compilers. >> if i do >> >> python -c 'import numpy;numpy.matrix([[1, 5, 10], [1.0, 3j, 4]], >> numpy.complex128).T.I.H' >> python: symbol lookup error: >> /opt/intel/Compiler/11.0/069/mkl/lib/em64/libmkl_lapack.so: undefined >> symbol: mkl_lapack_zgeqrf >> >> my site.cfg is : >> #################### >> [mkl] >> >> mkl_libs = mkl_def, mkl_intel_lp64, mkl_intel_thread, mkl_core, mkl_mc >> lapack_libs = mkl_lapack95_lp64 >> >> library_dirs = >> /opt/intel/Compiler/11.0/069/mkl/lib/em64t:/opt/intel/Compiler/11.0/069/lib/intel64/ >> include_dirs = >> /opt/intel/Compiler/11.0/069/mkl/include:/opt/intel/Compiler/11.0/069/include/ >> #################### >> and intelcompiler.py is : >> ############################ >> from distutils.unixccompiler import UnixCCompiler >> from numpy.distutils.exec_command import find_executable >> import sys >> >> class IntelCCompiler(UnixCCompiler): >> """ A modified Intel compiler compatible with an gcc built Python.""" >> compiler_type = 'intel' >> cc_exe = 'icc' >> cc_args = 'fPIC' >> >> def __init__ (self, verbose=0, dry_run=0, force=0): >> sys.exit(0) >> UnixCCompiler.__init__ (self, verbose,dry_run, force) >> self.cc_exe = 'icc -fPIC ' >> compiler = self.cc_exe >> self.set_executables(compiler=compiler, >> compiler_so=compiler, >> compiler_cxx=compiler, >> linker_exe=compiler, >> linker_so=compiler + ' -shared -lstdc++') >> >> class IntelItaniumCCompiler(IntelCCompiler): >> compiler_type = 'intele' >> >> # On Itanium, the Intel Compiler used to be called ecc, let's search >> for >> # it (now it's also icc, so ecc is last in the search). >> for cc_exe in map(find_executable,['icc','ecc']): >> if cc_exe: >> break >> >> class IntelEM64TCCompiler(UnixCCompiler): >> """ A modified Intel x86_64 compiler compatible with a 64bit gcc built >> Python. >> """ >> compiler_type = 'intelem' >> cc_exe = 'icc -m64 -fPIC' >> cc_args = "-fPIC -openmp" >> def __init__ (self, verbose=0, dry_run=0, force=0): >> UnixCCompiler.__init__ (self, verbose,dry_run, force) >> self.cc_exe = 'icc -m64 -fPIC -openmp ' >> compiler = self.cc_exe >> self.set_executables(compiler=compiler, >> compiler_so=compiler, >> compiler_cxx=compiler, >> linker_exe=compiler, >> linker_so=compiler + ' -shared -lstdc++') >> ########################## >> LD_LIBRARY_PATH is : >> ######################### >> >> /opt/intel/Compiler/11.0/069/mkl/lib/em64t:/opt/scalasca-1.3.3/lib:/opt/PBS/lib:/opt/intel/mpi/lib64:/opt/maui/lib:/opt/jdk1.6.0_23/lib:/opt/intel/Compiler/11.0/069/ipp/em64t/sharedlib:/opt/intel/Compiler/11.0/069/mkl/lib/em64t:/opt/intel/Compiler/11.0/069/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib:/opt/intel/Compiler/11.0/069/lib/intel64:/opt/intel/Compiler/11.0/069/ipp/em64t/sharedlib:/opt/intel/Compiler/11.0/069/mkl/lib/em64t:/opt/intel/Compiler/11.0/069/tbb/em64t/cc4.1.0_libc2.4_kernel2.6.16.21/lib:/opt/intel/Compiler/11.0/069/lib/intel64:/usr/local/lib >> ######################### >> >> -AKSHAR >> >> >> >> On Sat, Oct 22, 2011 at 11:32 AM, Charles R Harris < >> charlesr.harris(a)gmail.com> wrote: >> >>> >>> >>> On Fri, Oct 21, 2011 at 11:49 PM, akshar bhosale < >>> akshar.bhosale(a)gmail.com> wrote: >>> >>>> Hi, >>>> >>>> unfortunately 1.6.1 also hangs on the same place. Can i move ahead with >>>> installing scipy? >>>> >>>> >>> Hmm. Well, give scipy a try, but it would be nice to know what the >>> problem is with einsum. I'm thinking compiler, GCC 4.1.2 might be a bit >>> old, but it could easily be something else. Can you give us more information >>> about your system? >>> >>> Chuck >>> >>>> >>>> On Sat, Oct 22, 2011 at 12:19 AM, Charles R Harris < >>>> charlesr.harris(a)gmail.com> wrote: >>>> >>>>> >>>>> >>>>> On Fri, Oct 21, 2011 at 5:25 AM, akshar bhosale < >>>>> akshar.bhosale(a)gmail.com> wrote: >>>>> >>>>>> Hi, >>>>>> does this mean that numpy is not configured properly or i can ignore >>>>>> this and go ahead with scipy installation? >>>>>> >>>>> >>>>> Scipy will probably work, but you should really install numpy 1.6.1 >>>>> instead of 1.6.0. >>>>> >>>>> <snip> >>>>> >>>>> Chuck >>>>> >>>>> _______________________________________________ >>>>> SciPy-Dev mailing list >>>>> SciPy-Dev(a)scipy.org >>>>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>>>> >>>>> >>>> >>>> _______________________________________________ >>>> SciPy-Dev mailing list >>>> SciPy-Dev(a)scipy.org >>>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>>> >>>> >>> >>> _______________________________________________ >>> SciPy-Dev mailing list >>> SciPy-Dev(a)scipy.org >>> http://mail.scipy.org/mailman/listinfo/scipy-dev >>> >>> >> > >

1 0

numpy.distutils quirk
by Robert Cimrman 19 Oct '11

19 Oct '11

Hi, I have now spent several hours hunting down a major slowdown of my code caused (apparently) by using config.add_library() for a reusable part of C source files instead of just config.add_extension(). The reason of the slowdown was different, but hard to discern, naming of options and silent ignoring of non-existing ones: add_library() : extra_compiler_args add_extension() : extra_compile_args Other build keys used for the same purpose also differ. A bug to be reported, or is this going to be solved by going bento? r.

2 3

Evaluate bivariate polynomials
by Nils Wagner 19 Oct '11

19 Oct '11

Hi all, how do I evaluate a bivariate polynomial p(x,y)=c_0 + c_1 x + c_2 y +c_3 x**2 + c_4 x*y+ c_5 y**2 + c_6 x**3 + c_7 x**2*y + c_8 x*y**2+c_9*y**3 + \dots in numpy ? In case of univariate polynomials I can use np.polyval. Any pointer would be appreciated. Nils

2 1

Boolean indexing of 2d-Array not intutive
by Till Stensitzki 19 Oct '11

19 Oct '11

Hi, for a description of the problem see here: http://stackoverflow.com/questions/7820809/understanding-weird-boolean-2d-a… I really think, that the current way of handling two boolean indices is missleading, is there any reason behind that? greetings Till

1 0

np.ma.mean is not working?
by Chao YUE 18 Oct '11

18 Oct '11

Dear all, previoulsy I think np.ma.mean() will automatically filter the masked (missing) value but it's not? In [489]: a=np.arange(20.).reshape(2,10) In [490]: a=np.ma.masked_array(a,(a==2)|(a==5)|(a==11)|(a==18),fill_value=np.nan) In [491]: a Out[491]: masked_array(data = [[0.0 1.0 -- 3.0 4.0 -- 6.0 7.0 8.0 9.0] [10.0 -- 12.0 13.0 14.0 15.0 16.0 17.0 -- 19.0]], mask = [[False False True False False True False False False False] [False True False False False False False False True False]], fill_value = nan) In [492]: a.mean(0) Out[492]: masked_array(data = [5.0 1.0 12.0 8.0 9.0 15.0 11.0 12.0 8.0 14.0], mask = [False False False False False False False False False False], fill_value = 1e+20) In [494]: np.ma.mean(a,0) Out[494]: masked_array(data = [5.0 1.0 12.0 8.0 9.0 15.0 11.0 12.0 8.0 14.0], mask = [False False False False False False False False False False], fill_value = 1e+20) In [495]: np.ma.mean(a,0)==a.mean(0) Out[495]: masked_array(data = [ True True True True True True True True True True], mask = False, fill_value = True) only use a.filled().mean(0) can I get the result I want: In [496]: a.filled().mean(0) Out[496]: array([ 5., NaN, NaN, 8., 9., NaN, 11., 12., NaN, 14.]) I am doing this because I tried to have a small fuction from the web to do moving average for data: import numpy as np def rolling_window(a, window): if window < 1: raise ValueError, "`window` must be at least 1." if window > a.shape[-1]: raise ValueError, "`window` is too long." shape = a.shape[:-1] + (a.shape[-1] - window + 1, window) strides = a.strides + (a.strides[-1],) return np.lib.stride_tricks.as_strided(a, shape=shape, strides=strides) def move_ave(a,window): temp=rolling_window(a,window) pre=int(window)/2 post=int(window)-pre-1 return np.concatenate((a[...,0:pre],np.mean(temp,-1),a[...,-post:]),axis=-1) In [489]: a=np.arange(20.).reshape(2,10) In [499]: move_ave(a,4) Out[499]: masked_array(data = [[ 0. 1. 1.5 2.5 3.5 4.5 5.5 6.5 7.5 9. ] [ 10. 11. 11.5 12.5 13.5 14.5 15.5 16.5 17.5 19. ]], mask = False, fill_value = 1e+20) thanks, Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

3 5

abs for max negative integers - desired behavior?
by Matthew Brett 18 Oct '11

18 Oct '11

Hi, I recently ran into this: In [68]: arr = np.array(-128, np.int8) In [69]: arr Out[69]: array(-128, dtype=int8) In [70]: np.abs(arr) Out[70]: -128 Of course, I can see why this happens, but it is still surprising, and it seems to me that it would be a confusing source of bugs, because of course it only happens for the maximum negative integer. One particular confusing result was: In [71]: np.allclose(arr, arr) Out[71]: False I wanted to ask whether this is the desired behavior, and whether it might be worth planning a change in the long term? Best, Matthew

8 15

anyway to check a ndarray is a mased array or not?
by Chao YUE 18 Oct '11

18 Oct '11

Just one more question, how can I check a ndarray is a masked array or not? Chao -- *********************************************************************************** Chao YUE Laboratoire des Sciences du Climat et de l'Environnement (LSCE-IPSL) UMR 1572 CEA-CNRS-UVSQ Batiment 712 - Pe 119 91191 GIF Sur YVETTE Cedex Tel: (33) 01 69 08 29 02; Fax:01.69.08.77.16 ************************************************************************************

3 5