Mailman 3 September 2011 - NumPy-Discussion

1.7.x release of NumPy
by Travis Oliphant 15 Sep '11

15 Sep '11

Hi all, Has there been a discussion of a 1.7.x release of NumPy? There are a few new features that should go into the 1.x release of NumPy, that don't require the ABI changes of 2.0. I thought I had heard Mark talk in support of such a thing. What are the plans for the release of NumPy 2.0? Between my experiences over the last 4 years of working on real-world projects, the data array summit, SciPy, and conversations with Mark Wiebe, Peter Wang, Wes McKinney and others over the past months, there is a collection of pretty concrete ideas emerging about where the next version of NumPy should go. Some of these can be folded into NumPy 2.0 (and even 1.7), but my mind is racing around NumPy 3.0 at this point. -Travis

5 4

[ANN} OpenOpt, FuncDesigner, DerApproximator, SpaceFuncs release 0.36
by Dmitrey 15 Sep '11

15 Sep '11

Hi all, new release of our free soft (OpenOpt, FuncDesigner, DerApproximator, SpaceFuncs) v. 0.36 is out: OpenOpt: > * Now solver interalg can handle all types of constraints and integration problems > * Some minor improvements and code cleanup > FuncDesigner: > * Interval analysis now can involve min, max and 1-d monotone splines R -> R of 1st and 3rd order > * Some bugfixes and improvements > SpaceFuncs: > * Some minor changes > DerApproximator: > * Some improvements for obtaining derivatives in points from R^n where left or right derivative for a variable is absent, especially for stencil > 1 > See http://openopt.org for more details. Regards, D.

1 0

Re: [Numpy-discussion] Functions for finding the relative extrema of numeric data
by Jacob Silterra 15 Sep '11

15 Sep '11

>What is your application? The most common case is looking at Fourier transforms and identifying spectral peaks. I've also analyzed images looking at 1D slices (usually very regular data) and looked for peaks there. That stackoverflow page had a nice link to a comparison of different algorithms here: http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2631518/. That paper is focused on mass-spectrometry data, but the approach would generalize to any 1D data set. Unless somebody feels otherwise, I'll close this pull request and start working on an implementation of peak finding via continuous wavelet transform (the best and most computationally intensive approach of those described above). -Jacob ------------------------------ > > Message: 4 > Date: Tue, 13 Sep 2011 22:34:01 +0200 > From: Ralf Gommers <ralf.gommers(a)googlemail.com> > Subject: Re: [Numpy-discussion] Functions for finding the relative > extrema of numeric data > To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> > Message-ID: > <CABL7CQhxCX0LKFENMW6-4ZSbdieGxz04zbsrnY4bXYVxVL78Dw(a)mail.gmail.com > > > Content-Type: text/plain; charset="iso-8859-1" > > Hi Jacob, > > On Fri, Sep 9, 2011 at 11:57 PM, Jacob Silterra <jsilter(a)gmail.com> wrote: > > > Hello all, > > > > I'd like to see functions for calculating the relative extrema in a set > of > > data included in numpy. I use that functionality frequently, and always > seem > > to be writing my own version. It seems like this functionality would be > > useful to the community at large, as it's a fairly common operation. > > > > What is your application? > > > > > For numeric data (which is presumably noisy), the definition of a > relative > > extrema isn't completely obvious. The implementation I am proposing finds > a > > point in an ndarray along an axis which is larger (or smaller) than it's > > `order` nearest neighbors (`order` being an optional parameter, default > 1). > > This is likely to find more points than may be desired, which I believe > is > > preferable to the alternative. The output is formatted the same as > > numpy.where. > > > > Code available here: https://github.com/numpy/numpy/pull/154 > > > > I'm not sure whether this belongs in numpy or scipy, that question is > > somewhat debatable. More sophisticated peak-finding functions (in N > > dimensions, as opposed to 1) may also be useful to the community, and > those > > would definitely belong in scipy. > > > > I have the feeling this belongs in scipy. Although if it's just these two > functions I'm not sure where exactly to put them. Looking at the > functionality, this is quite a simple approach. For any data of the type > I'm > usually working with it will not be able to find the right local extrema. > The same is true for your alternative definition below. > > A more powerful peak detection function would be a very good addition to > scipy imho (perhaps in scipy.interpolate?). See also > > http://stackoverflow.com/questions/1713335/peak-finding-algorithm-for-pytho… > > Cheers, > Ralf > > > > An alternative implementation would be to require that function be > > continuously descending (or ascending) for `order` points, which would > > enforce a minimum width on a peak. > > > > -Jacob Silterra > > > > _______________________________________________ > > NumPy-Discussion mailing list > > NumPy-Discussion(a)scipy.org > > http://mail.scipy.org/mailman/listinfo/numpy-discussion > > > > >

4 3

load from text files Pull Request Review
by Christopher Jordan-Squire 15 Sep '11

15 Sep '11

Hi-- I've submitted a pull request for a new method for loading data from text files into a record array/masked record array. https://github.com/numpy/numpy/pull/143 Click on the link for more info, but the general idea is to create a regular expression for what entries should look like and loop over the file, updating the regular expression if it's wrong. Once the types are determined the file is loaded line by line into a pre-allocated numpy array. Compared to genfromtxt this function has several advantages/potential advantages. *More modular (genfromtxt is a rather large, nearly 500 line, monolithic function. In my pull request no individual method is longer than around 80 lines, and they're fairly self-contained.) *delimiters can be specified via regex's *missing data can be specified via regex's *it's bit simpler and has sensible defaults *it actually works on some (unfortunately proprietary) data that genfromtxt doesn't seem robust enough for *it supports datetimes *fairly extensible for the power user *makes two passes through the file, the first to determine types/sizes for strings and the second to read in the data, and pre-allocates the array for the second pass. So no giant memory bloating for reading large text files *fairly fast, though I think there is plenty of room for optimizations All that said, it's entirely possible that the innards which determine the type should be ripped out and submitted as a function on their own. I'd love suggestions for improvements, as well as suggestions for a better name. (Currently it's called loadtable, which I don't really like. It was just a working name.) -Chris Jordan-Squire

7 24

numpy.test() failure
by Nils Wagner 14 Sep '11

14 Sep '11

ERROR: test_polyfit (test_polynomial.TestDocs) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/lib/tests/test_polynomial.py", line 106, in test_polyfit weights = arange(8,1,-1)**2/7.0 NameError: global name 'arange' is not defined ====================================================================== FAIL: Tests polyfit ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/tests/test_extras.py", line 622, in test_polyfit assert_almost_equal(a, a_) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", line 155, in assert_almost_equal err_msg=err_msg, verbose=verbose) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", line 221, in assert_array_almost_equal header='Arrays are not almost equal') File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/ma/testutils.py", line 186, in assert_array_compare verbose=verbose, header=header) File "/home/nwagner/local/lib64/python2.6/site-packages/numpy/testing/utils.py", line 677, in assert_array_compare raise AssertionError(msg) AssertionError: Arrays are not almost equal (mismatch 100.0%) x: array([ 4.25134878, 1.14131297, 0.20519666, 0.01701 ]) y: array([ 1.9345248 , 0.49711011, 0.10202554, 0.00928034])

4 3

Re: [Numpy-discussion] behaviour of np.loadtxt
by Derek Homeier 14 Sep '11

14 Sep '11

On Wed, 14 Sep 2011, Davide wrote: > Dear list, > > I'm encountering a problem with np.loadtxt. > > Suppose i have a file containing three columns of data (and 10 rows), like: > > 0.001 0.003 0.005 > 0.001 0.003 0.006 > 0.002 0.004 0.002 > 0.004 0.002 0.007 > 0.001 0.003 0.006 > 0.002 0.004 0.002 > 0.004 0.002 0.007 > 0.001 0.003 0.006 > 0.002 0.004 0.002 > 0.004 0.002 0.007 > > If i give: > > len ( np.loadtxt( filename, unpack=True ) ) > > i get 3, since i have a three rows array. That's fine. > > If instead the file has a single column, i.e., > > 0.004 > 0.002 > 0.007 > 0.004 > 0.002 > 0.007 > 0.004 > 0.002 > 0.007 > 0.004 > > the command > > len ( np.loadtxt( filename, unpack=True ) ) > > returns 10, where i would expect it to return 1, to be consistent with > the behaviour when there are multiple columns. > > Is there a reason for why it is not like that? Yes, the default for loadtxt is to automatically "squeeze" the output to the minimum no. of dimensions required, thus it returns a 1D-array in the latter case. If you are using numpy 1.6 or later, you can ensure to get a consistent shape by passing the "ndmin=2" option. Cheers, Derek -- ---------------------------------------------------------------- Derek Homeier Centre de Recherche Astrophysique de Lyon ENS Lyon 46, Allée d'Italie 69364 Lyon Cedex 07, France +33 1133 47272-8894 ----------------------------------------------------------------

1 0

behaviour of np.loadtxt
by Davide 14 Sep '11

14 Sep '11

Dear list, I'm encountering a problem with np.loadtxt. Suppose i have a file containing three columns of data (and 10 rows), like: 0.001 0.003 0.005 0.001 0.003 0.006 0.002 0.004 0.002 0.004 0.002 0.007 0.001 0.003 0.006 0.002 0.004 0.002 0.004 0.002 0.007 0.001 0.003 0.006 0.002 0.004 0.002 0.004 0.002 0.007 If i give: len ( np.loadtxt( filename, unpack=True ) ) i get 3, since i have a three rows array. That's fine. If instead the file has a single column, i.e., 0.004 0.002 0.007 0.004 0.002 0.007 0.004 0.002 0.007 0.004 the command len ( np.loadtxt( filename, unpack=True ) ) returns 10, where i would expect it to return 1, to be consistent with the behaviour when there are multiple columns. Is there a reason for why it is not like that? Cheers Davide Lasagna

1 0

Re: [Numpy-discussion] Numpy - MKL - build error
by Igor Ying 14 Sep '11

14 Sep '11

Yes, they all are present in that directory. Also, I tried with root as login. -r-xr-xr-x 1 root root 26342559 Aug 9 22:19 libmkl_avx.so -r--r--r-- 1 root root 1190224 Aug 9 22:26 libmkl_blacs_ilp64.a -r--r--r-- 1 root root 1191496 Aug 9 22:25 libmkl_blacs_intelmpi_ilp64.a -r-xr-xr-x 1 root root 497597 Aug 9 22:25 libmkl_blacs_intelmpi_ilp64.so -r--r--r-- 1 root root 676206 Aug 9 22:21 libmkl_blacs_intelmpi_lp64.a -r-xr-xr-x 1 root root 267010 Aug 9 22:21 libmkl_blacs_intelmpi_lp64.so -r--r--r-- 1 root root 674926 Aug 9 22:22 libmkl_blacs_lp64.a -r--r--r-- 1 root root 1218290 Aug 9 22:28 libmkl_blacs_openmpi_ilp64.a -r--r--r-- 1 root root 703042 Aug 9 22:23 libmkl_blacs_openmpi_lp64.a -r--r--r-- 1 root root 1191152 Aug 9 22:29 libmkl_blacs_sgimpt_ilp64.a -r--r--r-- 1 root root 675854 Aug 9 22:23 libmkl_blacs_sgimpt_lp64.a -r--r--r-- 1 root root 425802 Aug 9 20:44 libmkl_blas95_ilp64.a -r--r--r-- 1 root root 421410 Aug 9 20:44 libmkl_blas95_lp64.a -r--r--r-- 1 root root 144354 Aug 9 22:29 libmkl_cdft_core.a -r-xr-xr-x 1 root root 115588 Aug 9 22:29 libmkl_cdft_core.so -r--r--r-- 1 root root 231886824 Aug 9 22:07 libmkl_core.a -r-xr-xr-x 1 root root 16730033 Aug 9 22:18 libmkl_core.so -r-xr-xr-x 1 root root 21474555 Aug 9 22:18 libmkl_def.so -r--r--r-- 1 root root 14974574 Aug 9 22:06 libmkl_gf_ilp64.a -r-xr-xr-x 1 root root 7008828 Aug 9 22:48 libmkl_gf_ilp64.so -r--r--r-- 1 root root 15140998 Aug 9 22:06 libmkl_gf_lp64.a -r-xr-xr-x 1 root root 7055304 Aug 9 22:48 libmkl_gf_lp64.so -r--r--r-- 1 root root 16435120 Aug 9 22:07 libmkl_gnu_thread.a -r-xr-xr-x 1 root root 9816940 Aug 9 22:49 libmkl_gnu_thread.so -r--r--r-- 1 root root 14968130 Aug 9 22:06 libmkl_intel_ilp64.a -r-xr-xr-x 1 root root 7008368 Aug 9 22:48 libmkl_intel_ilp64.so -r--r--r-- 1 root root 15134406 Aug 9 22:06 libmkl_intel_lp64.a -r-xr-xr-x 1 root root 7053588 Aug 9 22:48 libmkl_intel_lp64.so -r--r--r-- 1 root root 2472940 Aug 9 22:07 libmkl_intel_sp2dp.a -r-xr-xr-x 1 root root 1191479 Aug 9 22:20 libmkl_intel_sp2dp.so -r--r--r-- 1 root root 27642508 Aug 9 22:07 libmkl_intel_thread.a -r-xr-xr-x 1 root root 17516608 Aug 9 22:49 libmkl_intel_thread.so -r--r--r-- 1 root root 5350948 Aug 9 20:44 libmkl_lapack95_ilp64.a -r--r--r-- 1 root root 5413476 Aug 9 20:44 libmkl_lapack95_lp64.a -r-xr-xr-x 1 root root 29543829 Aug 9 22:19 libmkl_mc3.so -r-xr-xr-x 1 root root 25428037 Aug 9 22:19 libmkl_mc.so -r-xr-xr-x 1 root root 22888659 Aug 9 22:18 libmkl_p4n.so -r--r--r-- 1 root root 19232716 Aug 9 22:07 libmkl_pgi_thread.a -r-xr-xr-x 1 root root 12243062 Aug 9 22:49 libmkl_pgi_thread.so -r-xr-xr-x 1 root root 4984870 Aug 9 22:49 libmkl_rt.so -r--r--r-- 1 root root 10367758 Aug 9 22:49 libmkl_scalapack_ilp64.a -r-xr-xr-x 1 root root 6574928 Aug 9 22:50 libmkl_scalapack_ilp64.so -r--r--r-- 1 root root 10292432 Aug 9 22:49 libmkl_scalapack_lp64.a -r-xr-xr-x 1 root root 6452627 Aug 9 22:50 libmkl_scalapack_lp64.so -r--r--r-- 1 root root 9958444 Aug 9 22:07 libmkl_sequential.a -r-xr-xr-x 1 root root 5926347 Aug 9 22:48 libmkl_sequential.so -r--r--r-- 1 root root 1048 Aug 9 16:50 libmkl_solver_ilp64.a -r--r--r-- 1 root root 1048 Aug 9 16:50 libmkl_solver_ilp64_sequential.a -r--r--r-- 1 root root 1048 Aug 9 16:50 libmkl_solver_lp64.a -r--r--r-- 1 root root 1048 Aug 9 16:50 libmkl_solver_lp64_sequential.a -r-xr-xr-x 1 root root 6711968 Aug 9 22:48 libmkl_vml_avx.so -r-xr-xr-x 1 root root 2795928 Aug 9 22:47 libmkl_vml_def.so -r-xr-xr-x 1 root root 5476786 Aug 9 22:48 libmkl_vml_mc2.so -r-xr-xr-x 1 root root 5778052 Aug 9 22:48 libmkl_vml_mc3.so -r-xr-xr-x 1 root root 5382511 Aug 9 22:48 libmkl_vml_mc.so -r-xr-xr-x 1 root root 4235841 Aug 9 22:48 libmkl_vml_p4n.so drwxr-xr-x 3 root root 4096 Aug 18 11:43 locale Y you can reach the person managing the list at numpy-discussion-owner(a)scipy.org Message: 1 Date: Tue, 13 Sep 2011 09:58:27 -0400 From: Olivier Delalleau <shish(a)keba.be> Subject: Re: [Numpy-discussion] Numpy - MKL - build error To: Discussion of Numerical Python <numpy-discussion(a)scipy.org> Message-ID: <CAFXk4bpDN7qcwmZe2G565gsuOntau7LMH4fGSTH_zY_R+NLbMg(a)mail.gmail.com> Content-Type: text/plain; charset="iso-8859-1" Sorry if it sounds like a stupid question, but are the files listed in the error message present in that directory? If yes, maybe try running the command with sudo, just in case it would be some weird permission issue. -=- Olivier 2011/9/13 Igor Ying <igor.ying(a)yahoo.com> > Hi, > > I am very new to Numpy and trying to build Numpy 1.6.1 with Intel MKL > 10.3.6 and getting the following errors. > > $python setup.py config --compiler=intel build_clib --compiler=intel > build_ext --compiler=intel install > > Running from numpy source directory.F2PY Version 2 > blas_opt_info: > blas_mkl_info: > libraries mkl_def,mkl_intel_lp64,mkl_intel_thread,mkl_core,iomp5md not > found in /opt/intel/composer_xe_2011_sp1/mkl/lib/intel64 > NOT AVAILABLE > > I have modified the site.cfg as below. > > [mkl] > library_dirs = /opt/intel/composer_xe_2011_sp1/mkl/lib/intel64 > include_dirs = /opt/intel/composer_xe_2011_sp1/mkl/include > mkl_libs = mkl_def,mkl_intel_lp64,mkl_intel_thread,mkl_core,iomp5md > lapack_libs = mkl_lapack_lp64 > > > Environment variables are also set to point to mkl libraries. > > source /opt/intel/composer_xe_2011_sp1/bin/compilervars.sh intel64 lp64 > > export OMP_NUM_THREADS=4 > LD_LIBRARY_PATH="/opt/intel/composer_xe_2011_sp1/mkl/lib/intel64:${LD_LIBRARY_PATH}"; > > export LD_LIBRARY_PATH > export > LD_RUN_PATH=/opt/lib:/opt1/composer_xe_2011_sp1/lib/intel64:/opt/composer_xe_2011_sp1/mkl/lib/intel64 > > Can anyone help me? > > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)scipy.org > http://mail.scipy.org/mailman/listinfo/numpy-discussion > >

2 2

upgrading to 1.6.1, lapack not used?
by Thomas K Gamble 14 Sep '11

14 Sep '11

I am trying to upgrade my numpy installation from 1.5.1 to 1.6.1 on Ubuntu 10.04. When I build, it doesn't seem to use the installed lapack libraries, but uses lapack_lite. After installation (which seems to complete without error), importing numpy fails with an error that it can't find libquadmath.so.0. Any ideas what might be wrong? Am I missing a dependent package somewhere? How can I force the build to use the installed lapack stuff? (installed via synaptic from the ubuntu debs) Numpy 1.5.1 builds and installs with no problems. -- Thomas K. Gamble Research Technologist, System/Network Administrator Chemical Diagnostics and Engineering (C-CDE) Los Alamos National Laboratory MS-E543,p:505-665-4323 f:505-665-4267 "There cannot be a crisis next week. My schedule is already full." Henry Kissinger

2 1

Functions for finding the relative extrema of numeric data
by Jacob Silterra 13 Sep '11

13 Sep '11

Hello all, I'd like to see functions for calculating the relative extrema in a set of data included in numpy. I use that functionality frequently, and always seem to be writing my own version. It seems like this functionality would be useful to the community at large, as it's a fairly common operation. For numeric data (which is presumably noisy), the definition of a relative extrema isn't completely obvious. The implementation I am proposing finds a point in an ndarray along an axis which is larger (or smaller) than it's `order` nearest neighbors (`order` being an optional parameter, default 1). This is likely to find more points than may be desired, which I believe is preferable to the alternative. The output is formatted the same as numpy.where. Code available here: https://github.com/numpy/numpy/pull/154 I'm not sure whether this belongs in numpy or scipy, that question is somewhat debatable. More sophisticated peak-finding functions (in N dimensions, as opposed to 1) may also be useful to the community, and those would definitely belong in scipy. An alternative implementation would be to require that function be continuously descending (or ascending) for `order` points, which would enforce a minimum width on a peak. -Jacob Silterra

3 2