Mailman 3 June 2015 - NumPy-Discussion

How to limit cross correlation window width in Numpy?
by Honi Sanders 21 Jun '15

21 Jun '15

I am learning numpy/scipy, coming from a MATLAB background. The xcorr function in Matlab has an optional argument "maxlag" that limits the lag range from –maxlag to maxlag. This is very useful if you are looking at the cross-correlation between two very long time series but are only interested in the correlation within a certain time range. The performance increases are enormous considering that cross-correlation is incredibly expensive to compute. What is troubling me is that numpy.correlate does not have a maxlag feature. This means that even if I only want to see correlations between two time series with lags between -100 and +100 ms, for example, it will still calculate the correlation for every lag between -20000 and +20000 ms (which is the length of the time series). This (theoretically) gives a 200x performance hit! Is it possible that I could contribute this feature? I have introduced this question as a scipy issue https://github.com/scipy/scipy/issues/4940 and on the spicy-dev list (http://mail.scipy.org/pipermail/scipy-dev/2015-June/020757.html). It seems the best place to start is with numpy.correlate, so that is what I am requesting. I have done a simple implementation (https://gist.github.com/bringingheavendown/b4ce18aa007118e4e084) which gives 50x speedup under my conditions (https://github.com/scipy/scipy/issues/4940#issuecomment-110187847). This is my first experience with contributing to open-source software, so any pointers are appreciated.

3 7

Clarification sought on Scipy Numpy version requirements.
by Charles R Harris 21 Jun '15

21 Jun '15

Hi All, I'm looking to change some numpy deprecations into errors as well as remove some deprecated functions. The problem I see is that SciPy claims to support Numpy >= 1.5 and Numpy 1.5 is really, really, old. So the question is, does "support" mean compiles with earlier versions of Numpy ? If that is the case there is very little that can be done about deprecation. OTOH, if it means Scipy can be compiled with more recent numpy versions but used with earlier Numpy versions (which is a good feat), I'd like to know. I'd also like to know what the interface requirements are, as I'd like to remove old_defines.h Chuck

4 14

I can't tell if Numpy is configured properly with show_config()
by Elliot Hallmark 20 Jun '15

20 Jun '15

Debian Sid, 64-bit. I was trying to fix the problem of np.dot running very slow. I ended up uninstalling numpy, installing libatlas3-base through apt-get and re-installing numpy. The performance of dot is greatly improved! But I can't tell from any other method whether numpy is set up correctly. Consider comparing the faster one to another in a virtual env that is still slow: ### fast one ### In [1]: import time, numpy In [2]: n=1000 In [3]: A = numpy.random.rand(n,n) In [4]: B = numpy.random.rand(n,n) In [5]: then = time.time(); C=numpy.dot(A,B); print time.time()-then 0.306427001953 In [6]: numpy.show_config() blas_info: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 lapack_info: libraries = ['lapack'] library_dirs = ['/usr/lib'] language = f77 atlas_threads_info: NOT AVAILABLE blas_opt_info: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 1)] atlas_blas_threads_info: NOT AVAILABLE openblas_info: NOT AVAILABLE lapack_opt_info: libraries = ['lapack', 'blas'] library_dirs = ['/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 1)] atlas_info: NOT AVAILABLE lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE atlas_blas_info: NOT AVAILABLE mkl_info: NOT AVAILABLE ### slow one ### In [1]: import time, numpy In [2]: n=1000 In [3]: A = numpy.random.rand(n,n) In [4]: B = numpy.random.rand(n,n) In [5]: then = time.time(); C=numpy.dot(A,B); print time.time()-then 7.88430500031 In [6]: numpy.show_config() blas_info: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 lapack_info: libraries = ['lapack'] library_dirs = ['/usr/lib'] language = f77 atlas_threads_info: NOT AVAILABLE blas_opt_info: libraries = ['blas'] library_dirs = ['/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 1)] atlas_blas_threads_info: NOT AVAILABLE openblas_info: NOT AVAILABLE lapack_opt_info: libraries = ['lapack', 'blas'] library_dirs = ['/usr/lib'] language = f77 define_macros = [('NO_ATLAS_INFO', 1)] atlas_info: NOT AVAILABLE lapack_mkl_info: NOT AVAILABLE blas_mkl_info: NOT AVAILABLE atlas_blas_info: NOT AVAILABLE mkl_info: NOT AVAILABLE ##### Further, in the following comparison between Cpython and converting to numpy array for one operation, I get Cpython being faster by the same amount in both environments. But another user got numpy being faster. In [1]: import numpy as np In [2]: pts = range(100,1000) In [3]: pts[100] = 0 In [4]: %timeit pts_arr = np.array(pts); mini = np.argmin(pts_arr) 10000 loops, best of 3: 129 µs per loop In [5]: %timeit mini = sorted(enumerate(pts))[0][1] 10000 loops, best of 3: 89.2 µs per loop The other user got In [29]: %timeit pts_arr = np.array(pts); mini = np.argmin(pts_arr) 10000 loops, best of 3: 37.7 µs per loop In [30]: %timeit mini = sorted(enumerate(pts))[0][1] 10000 loops, best of 3: 69.2 µs per loop And I can't help but wonder if there is further configuration I need to make numpy faster, or if this is just a difference between out machines In the future, should I ignore show_config() and just do this dot product test? Any guidance would be appreciated. Thanks, Elliot

4 6

Flag for np.tile to use as_strided to reduce memory
by Freddy Rietdijk 19 Jun '15

19 Jun '15

Hi, Having read that it is possible to basically 'copy' elements along an axis without actually copying the values by making use of the strides, I wonder whether it is possible to add this as an option to np.tile. It would be easier than having to use as_strided or broadcast_arrays to repeat data without actually replicating it. http://stackoverflow.com/questions/23695851/python-repeating-numpy-array-wi… https://scipy-lectures.github.io/advanced/advanced_numpy/#example-fake-dime… Frederik

3 2

1.10 release
by Charles R Harris 17 Jun '15

17 Jun '15

Hi All, I plan to branch the 1.10 release next Monday. I expect this to be a difficult release, much time has passed since 1.9 and there have been significant enhancements/changes to masked arrays, structured array assigment, and recored arrays. In addition, most ufuncs no longer return `NotImplemented`, so there have been changes associated with the handling of ndarray operators. There still remains some tasks to do, including some PEP8 cleanup before the branch, but I would much appreciate it if folks could start testing now, especially if you make use of the parts that have undergone signigicant change. This is also the time to complain if you think some needed PR has been left out. I would also appreciate help with the release notes, as I am sure there are changes that should be noted but are currently omitted. Chuck

1 0

Aternative to PyArray_SetBaseObject in NumPy 1.6?
by Sturla Molden 16 Jun '15

16 Jun '15

What would be the best alternative to PyArray_SetBaseObject in NumPy 1.6? Purpose: Keep alive an object owning data passed to PyArray_SimpleNewFromData. Sturla

2 2

Homu
by Nathaniel Smith 15 Jun '15

15 Jun '15

Hi all, As an experiment, I just enabled @homu on the main numpy repository. Basically what this means is that there's a bot named @homu, and if someone with appropriate permissions posts a comment on a pull request that says: @homu r+ then homu will (a) doublecheck that the pull request still passes tests when merged into current master, and (b) if it does, then go ahead and hit the green merge button for you. ("r+" is mozilla-ese for "I approve this patch"; @homu comes out of the rust/mozilla community.) So you can still hit the big green button if you want, no change there, but this provides a second option with a few advantages: - Normally, a green light from Travis just means that the PR passed the tests when it was submitted. If master has changed since then, things might have become broken, but you'll never know until after you merge it and master turns red. More minor advantages: - You can approve a PR before Travis has even finished running, and it will automatically be merged iff the tests pass. - In theory, it should be possible to put someone on the @homu permissions list without adding them to github proper, which would mean that they have the ability to push to the repository via PRs-that-pass-tests-and-trigger-notifications, but can't do a direct commit directly into master that doesn't create any notifications. Not sure if this is really useful, but hey. - You don't have to merge-and-then-comment-saying thanks, you can just post a single comment, saving two entire mouse clicks. Efficiency! Anyway, seemed worth taking for a spin and seeing whether we liked it; we can always turn it off again if not. I think that everyone who has commit access to numpy/numpy is also listed on @homu's access list -- if I missed anyone just let me know. Links: http://homu.io/ https://www.reddit.com/r/rust/comments/39sogp/homu_a_gatekeeper_for_your_co… http://graydon.livejournal.com/186550.html http://homu.io/q/numpy/numpy -n -- Nathaniel J. Smith -- http://vorpus.org

2 2

Re: [Numpy-discussion] f2py problem with multiple fortran source files
by 石头 13 Jun '15

13 Jun '15

Dear Pearu Peterson, Thank you for your reply! I did as you said, and I got the module mulib.so successfully, but I got another problem when I try to import this module in Python. Here are the messeage I got in Python: Enthought Canopy Python 2.7.6 | 64-bit | (default, Sep 15 2014, 17:36:10) [GCC 4.1.2 20080704 (Red Hat 4.1.2-54)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> import mylib Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: /home/ssj/Enthought/Canopy_64bit/User/bin/../lib/libgfortran.so.3: version `GFORTRAN_1.4' not found (required by ./mylib.so) >>> The version of gfortran I use is 4.9.2, and I'm using Debian 8 "Jessie". How can I fix this problem? Thanks again for your reply! Shishijie

1 0

ANN: pandas v0.16.2 released
by Jeff Reback 13 Jun '15

13 Jun '15

Hello, We are proud to announce v0.16.2 of pandas, a minor release from 0.16.1. This release includes a small number of API changes, several new features, enhancements, and performance improvements along with a large number of bug fixes. This was a release of 4 weeks with 105 commits by 32 authors encompassing 48 issues and 71 pull-requests. We recommend that all users upgrade to this version. *What is it:* *pandas* is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data both easy and intuitive. It aims to be the fundamental high-level building block for doing practical, real world data analysis in Python. Additionally, it has the broader goal of becoming the most powerful and flexible open source data analysis / manipulation tool available in any language. Highlights of this release include: - - A new *pipe* method, see here <http://pandas.pydata.org/pandas-docs/stable/whatsnew.html#whatsnew-0162-enh…> - Documentation on how to use numba <http://numba.pydata.org> with *pandas*, see here <http://pandas.pydata.org/pandas-docs/stable/enhancingperf.html#enhancingper…> See the Whatsnew in v0.16.2 <http://pandas.pydata.org/pandas-docs/stable/whatsnew.html> Documentation: http://pandas.pydata.org/pandas-docs/stable/ Source tarballs, windows binaries are available on PyPI: https://pypi.python.org/pypi/pandas windows binaries are courtesy of Christoph Gohlke and are built on Numpy 1.9 macosx wheels are courtesy of Matthew Brett Please report any issues here: https://github.com/pydata/pandas/issues Thanks The Pandas Development Team Contributors to the 0.16.2 release - Andrew Rosenfeld - Artemy Kolchinsky - Bernard Willers - Christer van der Meeren - Christian Hudon - Constantine Glen Evans - Daniel Julius Lasiman - Evan Wright - Francesco Brundu - Gaëtan de Menten - Jake VanderPlas - James Hiebert - Jeff Reback - Joris Van den Bossche - Justin Lecher - Ka Wo Chen - Kevin Sheppard - Mortada Mehyar - Morton Fox - Robin Wilson - Thomas Grainger - Tom Ajamian - Tom Augspurger - Yoshiki Vázquez Baeza - Younggun Kim - austinc - behzad nouri - jreback - lexual - rekcahpassyla - scls19fr - sinhrks

1 0

f2py problem with multiple fortran source files
by 石头 13 Jun '15

13 Jun '15

Hi,everybody, I'm new to f2py, and I got some trouble when wrapped some fortran files to Python. I have download a Fortran library (https://github.com/brianlockwood/ForK), I want to compile these files into a library and call the library by other Fortran file wrote by myself. Here are my questions: 1. How should I compile the library(in this case,Fork), and what command should I use; 2. How can I use the library and my own Fortran source fileI( All.f90 ) with the f2py command to generate the module I can use py Python Thanks! Shishijie

2 1