Mailman 3 April 2015 - NumPy-Discussion

performance of numpy.array()
by simona bellavista 30 Apr '15

30 Apr '15

I work on two distinct scientific clusters. I have run the same python code on the two clusters and I have noticed that one is faster by an order of magnitude than the other (1min vs 10min, this is important because I run this function many times). I have investigated with a profiler and I have found that the cause of this is that (same code and same data) is the function numpy.array that is being called 10^5 times. On cluster A it takes 2 s in total, whereas on cluster B it takes ~6 min. For what regards the other functions, they are generally faster on cluster A. I understand that the clusters are quite different, both as hardware and installed libraries. It strikes me that on this particular function the performance is so different. I would have though that this is due to a difference in the available memory, but actually by looking with `top` the memory seems to be used only at 0.1% on cluster B. In theory numpy is compiled with atlas on cluster B, and on cluster A it is not clear, because numpy.__config__.show() returns NOT AVAILABLE for anything. Does anybody has any insight on that, and if I can improve the performance on cluster B?

6 9

Weighted covariance.
by Charles R Harris 29 Apr '15

29 Apr '15

The weighted covariance function in PR #4960 <https://github.com/numpy/numpy/pull/4960> is evolving to the following, where frequency weights are `f` and reliability weights are `a`. Assume that the observations are in the columns of the observation matrix. the steps to compute the weighted covariance are as follows:: >>> w = f * a >>> v1 = np.sum(w) >>> v2 = np.sum(a * w) >>> m -= np.sum(m * w, axis=1, keepdims=True) / v1 >>> cov = np.dot(m * w, m.T) * v1 / (v1**2 - ddof * v2) Note that when ``a == 1``, the normalization factor ``v1 / (v1**2 - ddof * v2)`` goes over to ``1 / (np.sum(f) - ddof)`` as it should. This is probably a good time for comments from all the kibitzers out there. Chuck

1 0

ANN: numexpr 2.4.3 released
by Francesc Alted 29 Apr '15

29 Apr '15

Announcing Numexpr 2.4.3 ========================= Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. What's new ========== This is a maintenance release to cope with an old bug affecting comparisons with empty strings. Fixes #121 and PyTables #184. In case you want to know more in detail what has changed in this version, see: https://github.com/pydata/numexpr/wiki/Release-Notes or have a look at RELEASE_NOTES.txt in the tarball. Where I can find Numexpr? ========================= The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Alted

5 13

scipy.fftpack.diff operation question
by R Schumacher 28 Apr '15

28 Apr '15

We are looking to plot to time series accelerometer data as velocity and displacement. To this end we tried scipy.fftpack.diff, but in looking at the test code direct_diff() we get odd results, and, why is the doc using "sqrt(-1)*j" in its explanation? So, I tried a few different integration methods. We were first looking at the doc at http://docs.scipy.org/doc/scipy-0.15.1/reference/generated/scipy.fftpack.di… and code at https://github.com/scipy/scipy/blob/v0.15.1/scipy/fftpack/pseudo_diffs.py#L… as well as the tests at https://github.com/scipy/scipy/blob/master/benchmarks/benchmarks/fftpack_ps… The direct_diff() test function in bench_pseudo_diffs.py seems odd, since I have to pass period=-1 to get a matching sign plot for the first integration. And why the scaling differences required, even for diff() and direct_diff()? Is my understanding fundamentally flawed? Emacs! Ray

1 0

how to set a fixed sized dtype suitable for bitwise operations
by Benjamin Root 28 Apr '15

28 Apr '15

I have a need to have a numpy array of 17 byte (more specifically, at least 147 bits) values that I would be doing some bit twiddling on. I have found that doing a dtype of "i17" yields a dtype of int32, which is completely not what I intended. Doing 'u17' gets an "data type not understood". I have tried 'a17', but then bitwise_or() and left_shift() do not work (returns "NotImplemented"). How should I be going about this? Cheers! Ben Root

2 2

GSoC'15 accepted students for Scipy/Numpy
by Ralf Gommers 27 Apr '15

27 Apr '15

Hi all, Google has just announced which students got accepted for this year's GSoC. For Scipy these are: - Nikolay Mayorov, "Improve nonlinear least squares minimization functionality in SciPy" mentors: Chuck & Evgeni - Abraham Escalante, "SciPy: scipy.stats improvements" mentor: Ralf (Evgeni is backup mentor) Furthermore, this proposal was accepted for Scikit-image: - Aman Singh, "Scikit-Image: rewriting scipy.ndimage to cython" mentors: Jaime, Ralf & the scikit-image devs Congratulations to all of you! We had a lot of interest this year, which is great to see. GSoC applications are competitive, and unfortunately there are students who didn't make it. To those students I would say: please stay involved, and you're very welcome to apply again next year! Today is also the start of the "community bonding period", where the students aren't yet expected to start working on their project but do get time to further figure out how things work, interact with the community and ensure that they can hit the ground running on day 1 of the coding period: http://googlesummerofcode.blogspot.nl/2007/04/so-what-is-this-community-bon…. It looks like it'll be an interesting and productive summer! Cheers, Ralf P.S. all proposals are linked on https://github.com/scipy/scipy/wiki/GSoC-project-ideas#student-applications… for who's interested in the details. P.P.S. some students have asked to get some feedback about why they were/weren't accepted, in order to learn from it for a next time. Until today we weren't allowed to say much, but now that Google has announced the results I'd be happy to give some feedback - please contact me in private if you want.

1 0

EuroScipy 2015: Submission deadline in 3 days !!!
by Nicolas P. Rougier 27 Apr '15

27 Apr '15

--------------------------------- Submission deadline in 3 days !!! --------------------------------- EuroScipy 2015, the annual conference on Python in science will take place in Cambridge, UK on 26-30 August 2015. The conference features two days of tutorials followed by two days of scientific talks & posters and an extra day dedicated to developer sprints. It is the major event in Europe in the field of technical/scientific computing within the Python ecosystem. Data scientists, analysts, quants, PhD's, scientists and students from more than 20 countries attended the conference last year. The topics presented at EuroSciPy are very diverse, with a focus on advanced software engineering and original uses of Python and its scientific libraries, either in theoretical or experimental research, from both academia and the industry. Submissions for posters, talks & tutorials (beginner and advanced) are welcome on our website at http://www.euroscipy.org/2015/ Sprint proposals should be addressed directly to the organisation at euroscipy-org(a)python.org Important dates =============== Mar 24, 2015 Call for talks, posters & tutorials Apr 30, 2015 Talk and tutorials submission deadline May 1, 2015 Registration opens May 30, 2015 Final program announced Jun 15, 2015 Early-bird registration ends Aug 26-27, 2015 Tutorials Aug 28-29, 2015 Main conference Aug 30, 2015 Sprints We look forward to an exciting conference and hope to see you in Cambridge The EuroSciPy 2015 Team - http://www.euroscipy.org/2015/

1 0

ANN: PyTables 3.2.0 release candidate 1 is out
by Francesc Alted 21 Apr '15

21 Apr '15

=========================== Announcing PyTables 3.2.0rc1 =========================== We are happy to announce PyTables 3.2.0rc1. ******************************* IMPORTANT NOTICE: If you are a user of PyTables, it needs your help to keep going. Please read the next thread as it contains important information about the future of the project: https://groups.google.com/forum/#!topic/pytables-users/yY2aUa4H7W4 Thanks! ******************************* What's new ========== This is a major release of PyTables and it is the result of more than a year of accumulated patches, but most specially it fixes a nasty problem with indexed queries not returning the correct results in some scenarios. There are many usablity and performance improvements too. In case you want to know more in detail what has changed in this version, please refer to: http://pytables.github.io/release_notes.html You can download a source package with generated PDF and HTML docs, as well as binaries for Windows, from: http://sourceforge.net/projects/pytables/files/pytables/3.2.0rc1 For an online version of the manual, visit: http://pytables.github.io/usersguide/index.html What it is? =========== PyTables is a library for managing hierarchical datasets and designed to efficiently cope with extremely large amounts of data with support for full 64-bit file addressing. PyTables runs on top of the HDF5 library and NumPy package for achieving maximum throughput and convenient use. PyTables includes OPSI, a new indexing technology, allowing to perform data lookups in tables exceeding 10 gigarows (10**10 rows) in less than a tenth of a second. Resources ========= About PyTables: http://www.pytables.org About the HDF5 library: http://hdfgroup.org/HDF5/ About NumPy: http://numpy.scipy.org/ Acknowledgments =============== Thanks to many users who provided feature improvements, patches, bug reports, support and suggestions. See the ``THANKS`` file in the distribution package for a (incomplete) list of contributors. Most specially, a lot of kudos go to the HDF5 and NumPy makers. Without them, PyTables simply would not exist. Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. ---- **Enjoy data!** -- The PyTables Developers

1 0

Automatic number of bins for numpy histograms
by Varun 18 Apr '15

18 Apr '15

http://nbviewer.ipython.org/github/nayyarv/matplotlib/blob/master/examples/… tistics/A utomating%20Binwidth%20Choice%20for%20Histogram.ipynb Long story short, histogram visualisations that depend on numpy (such as matplotlib, or nearly all of them) have poor default behaviour as I have to constantly play around with the number of bins to get a good idea of what I'm looking at. The bins=10 works ok for up to 1000 points or very normal data, but has poor performance for anything else, and doesn't account for variability either. I don't have a method easily available to scale the number of bins given the data. R doesn't suffer from these problems and provides methods for use with it's hist method. I would like to provide similar functionality for matplotlib, to at least provide some kind of good starting point, as histograms are very useful for initial data discovery. The notebook above provides an explanation of the problem as well as some proposed alternatives. Use different datasets (type and size) to see the performance of the suggestions. All of the methods proposed exist in R and literature. I've put together an implementation to add this new functionality, but am hesitant to make a pull request as I would like some feedback from a maintainer before doing so. https://github.com/numpy/numpy/compare/master...nayyarv:master I've provided them as functions for easy refactoring, as it can be argued that it should be in it's own function/file/class, or alternatively can be turned into simple if, elif statements. I believe this belongs in numpy as it is where the functionality exists for histogram methods that most libraries build on, and it would useful for them to not require scipy for example. I will update the documentation accordingly before making a pull request, and add in more tests to show it's functionality. I can adapt my ipython notebook into a quick tutorial/help file if need be. I've already attempted to add this into matplotlib before being redirected here https://github.com/matplotlib/matplotlib/issues/4316

11 22

Consider improving numpy.outer's behavior with zero-dimensional vectors
by Neil Girdhar 17 Apr '15

17 Apr '15

Numpy's outer product works fine with vectors. However, I seem to always want len(outer(a, b).shape) to be equal to len(a.shape) + len(b.shape). Wolfram-alpha seems to agree https://reference.wolfram.com/language/ref/Outer.html with respect to matrix outer products. My suggestion is to define outer as defined below. I've contrasted it with numpy's current outer product. In [36]: def a(n): return np.ones(n) In [37]: b = a(()) In [38]: c = a(4) In [39]: d = a(5) In [40]: np.outer(b, d).shape Out[40]: (1, 5) In [41]: np.outer(c, d).shape Out[41]: (4, 5) In [42]: np.outer(c, b).shape Out[42]: (4, 1) In [43]: def outer(a, b): return a[(...,) + len(b.shape) * (np.newaxis,)] * b ....: In [44]: outer(b, d).shape Out[44]: (5,) In [45]: outer(c, d).shape Out[45]: (4, 5) In [46]: outer(c, b).shape Out[46]: (4,) Best, Neil

5 46