Mailman 3 March 2016 - NumPy-Discussion

ANN: numexpr 2.5.1 released
by Francesc Alted 31 Mar '16

31 Mar '16

========================= Announcing Numexpr 2.5.1 ========================= Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It wears multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. What's new ========== Fixed a critical bug that caused wrong evaluations of log10() and conj(). These produced wrong results when numexpr was compiled with Intel's MKL (which is a popular build since Anaconda ships it by default) and non-contiguous data. This is considered a *critical* bug and upgrading is highly recommended. Thanks to Arne de Laat and Tom Kooij for reporting and providing a test unit. In case you want to know more in detail what has changed in this version, see: https://github.com/pydata/numexpr/blob/master/RELEASE_NOTES.rst Where I can find Numexpr? ========================= The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Share your experience ===================== Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Alted

1 0

change to memmap subclass propagation
by Allan Haldane 30 Mar '16

30 Mar '16

Hi all, This is a warning for a planned change to np.memmap in https://github.com/numpy/numpy/pull/7406. The return values of ufuncs and fancy slices of a memmap will now be plain ndarrays, since those return values don't point to mem-mapped memory. There is a possibility that if you are subclassing memmap using multiple inheritance something may break. We don't think anyone will be affected, but please reply if it does affect you. Allan

1 0

Blog-post that explains what Blaze actually is and where Pluribus project now lives.
by Travis Oliphant 29 Mar '16

29 Mar '16

I have emailed this list in the past explaining what is driving my open source efforts now. Here is a blog-post that may help some of you understand at little bit of the history of Blaze, DyND Numba, and other related developments as they relate to scaling up and scaling out array-computing in Python. http://technicaldiscovery.blogspot.com/2016/03/anaconda-and-hadoop-story-of… This post and these projects do not have anything to do with the future of the NumPy and/or SciPy projects which are now in great hands guiding their community-driven development. The post is however, a discussion of additional projects that will hopefully benefit some of you as well, and for which your feedback and assistance is welcome. Best, -Travis -- *Travis Oliphant, PhD* *Co-founder and CEO* @teoliphant 512-222-5440 http://www.continuum.io

1 0

Re: [Numpy-discussion] Reflect array?
by Peter Creasey 29 Mar '16

29 Mar '16

> >> On Tue, Mar 29, 2016 at 1:46 PM, Benjamin Root <ben.v.root(a)gmail.com> > >> wrote: > >> > Is there a quick-n-easy way to reflect a NxM array that represents a > >> > quadrant into a 2Nx2M array? Essentially, I am trying to reduce the size > >> > of > >> > an expensive calculation by taking advantage of the fact that the first > >> > part > >> > of the calculation is just computing gaussian weights, which is radially > >> > symmetric. > >> > > >> > It doesn't seem like np.tile() could support this (yet?). Maybe we could > >> > allow negative repetitions to mean "reflected"? But I was hoping there > >> > was > >> > some existing function or stride trick that could accomplish what I am > >> > trying. > >> > > >> > x = np.linspace(-5, 5, 20) > >> > y = np.linspace(-5, 5, 24) > >> > z = np.hypot(x[None, :], y[:, None]) > >> > zz = np.hypot(x[None, :int(len(x)//2)], y[:int(len(y)//2), None]) > >> > zz = some_mirroring_trick(zz) > >> > > You can avoid the allocation with preallocation: > > nx = len(x) // 2 > ny = len(y) // 2 > zz = np.zeros((len(y), len(x))) > zz[:ny,-nx:] = np.hypot.outer(y[:ny], x[:nx]) > zz[:ny, :nx] = zz[:ny,:-nx-1:-1] > zz[-ny:, :] = zz[ny::-1, :] > > if nx * 2 != len(x): > zz[:ny, nx] = y[::-1] > zz[-ny:, nx] = y > if ny * 2 != len(y): > zz[ny, :nx] = x[::-1] > zz[ny, -nx:] = x > > All of the steps after the call to `hypot.outer` create views. This is > untested, so you may need to tweak the indices a little. > A couple of months ago I wrote a C-code with ctypes to do this sort of mirroring trick on an (N,N,N) numpy array of fft weights (where you can exploit the 48-fold symmetry of using interchangeable axes), which was pretty useful since I had N^3 >> 1e9 and the weight function was quite expensive. Obviously the (N,M) case doesn't allow quite so much optimization but if it could be interesting then PM me. Best, Peter

1 0

Reflect array?
by Benjamin Root 29 Mar '16

29 Mar '16

Is there a quick-n-easy way to reflect a NxM array that represents a quadrant into a 2Nx2M array? Essentially, I am trying to reduce the size of an expensive calculation by taking advantage of the fact that the first part of the calculation is just computing gaussian weights, which is radially symmetric. It doesn't seem like np.tile() could support this (yet?). Maybe we could allow negative repetitions to mean "reflected"? But I was hoping there was some existing function or stride trick that could accomplish what I am trying. x = np.linspace(-5, 5, 20) y = np.linspace(-5, 5, 24) z = np.hypot(x[None, :], y[:, None]) zz = np.hypot(x[None, :int(len(x)//2)], y[:int(len(y)//2), None]) zz = some_mirroring_trick(zz) assert np.all(z == zz) What can be my "some_mirroring_trick()"? I am hoping for something a little better than using hstack()/vstack(). Thanks, Ben Root

2 3

ANN: LAST CALL for SciPy (Scientific Python) 2016 Conference Talk / Poster Proposals - Due Friday 4/1
by Courtenay Godshall (Enthought) 29 Mar '16

29 Mar '16

**ANN: LAST CALL for SciPy 2016 Conference Talk / Poster Proposals (Scientific Computing with Python) - Final Deadline - Friday April 1st** <http://www.scipy2016.scipy.org/> SciPy 2016, the 15th annual Scientific Computing with Python conference, will be held July 11-17, 2016 in Austin, Texas. SciPy is a community dedicated to the advancement of scientific computing through open source Python software for mathematics, science, and engineering. The annual SciPy Conference brings together over 650 participants from industry, academia, and government to showcase their latest projects, learn from skilled users and developers, and collaborate on code development. The full program will consist of <http://scipy2016.scipy.org/ehome/146062/332967/?&&> 2 days of tutorials (July 11-12), <http://scipy2016.scipy.org/ehome/146062/332968/?&&> 3 days of talks (July 13-15), and 2 days of <http://scipy2016.scipy.org/ehome/146062/332969/?&&> developer sprints (July 16-17). More info is available on the conference website at <http://www.scipy2016.scipy.org/> http://scipy2016.scipy.org (where you can sign up for the mailing list); or follow <https://twitter.com/scipyconf> @scipyconf on Twitter. ---------------------------------------------------------------------------- --------------------------------- **SUBMIT A SCIPY 2016 TALK / POSTER PROPOSAL - DUE APRIL 1, 2016* ---------------------------------------------------------------------------- --------------------------------- Submissions for talks and posters are welcome on our website ( <http://scipy2016.scipy.org/> http://scipy2016.scipy.org). In your abstract, please provide details on what Python tools are being employed, and how. The talk and poster submission deadline is March 25th, 2016 SciPy 2016 will include 3 major topic tracks and 8 mini-symposia tracks. Major topic tracks include: - Scientific Computing in Python - Python in Data Science (Big data and not so big data) - High Performance Computing Mini-symposia will include the applications of Python in: - Earth and Space Science - Engineering - Medicine and Biology - Social Sciences - Special Purpose Databases - Case Studies in Industry - Education - Reproducibility If you have any questions or comments, feel free to contact us at: scipy-organizers(a)scipy.org ----------------------------------------------------------- **SCIPY 2016 REGISTRATION IS OPEN** ----------------------------------------------------------- Please register early. SciPy early bird registration until May 22, 2016. Register at <http://scipy2016.scipy.org/> http://scipy2016.scipy.org. Plus, enter our t-shirt design contest to win a free registration. (Send a vector art file to scipy(a)enthought.com by March 31 to enter). Important dates: April 1: Talk and Poster Proposals Due May 11: Plotting Contest Submissions Due Apr 22: Tutorials Announced Apr 22: Financial Aid Submissions Due May 4: Talk and Posters Announced May 22: Early Bird Registration Deadline Jul 11-12: SciPy 2016 Tutorials Jul 13-15: SciPy 2016 General Conference Jul 16-17: SciPy 2016 Sprints We look forward to an exciting conference and hope to see you in Austin in July! The Scipy 2016 Committee <http://scipy2016.scipy.org/> http://scipy2016.scipy.org/

1 0

ATLAS build errors
by Matthew Brett 29 Mar '16

29 Mar '16

Hi, I'm workon on building manylinux wheels for numpy, and I ran into unexpected problems with a numpy built against the ATLAS 3.8 binaries supplied by CentOS 5. I'm working on the manylinux docker container [1] To get ATLAS, I'm doing `yum install atlas-devel` which gets the default CentOS 5 ATLAS packages. I then build numpy. Local tests work fine, but when I test on travis, I get these errors [2]: ====================================================================== ERROR: test_svd_build (test_regression.TestRegression) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/travis/build/matthew-brett/manylinux-testing/venv/lib/python2.7/site-packages/numpy/linalg/tests/test_regression.py", line 56, in test_svd_build u, s, vh = linalg.svd(a) File "/home/travis/build/matthew-brett/manylinux-testing/venv/lib/python2.7/site-packages/numpy/linalg/linalg.py", line 1359, in svd u, s, vt = gufunc(a, signature=signature, extobj=extobj) ValueError: On entry to DGESDD parameter number 12 had an illegal value ====================================================================== FAIL: test_lapack (test_build.TestF77Mismatch) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/travis/build/matthew-brett/manylinux-testing/venv/lib/python2.7/site-packages/numpy/testing/decorators.py", line 146, in skipper_func return f(*args, **kwargs) File "/home/travis/build/matthew-brett/manylinux-testing/venv/lib/python2.7/site-packages/numpy/linalg/tests/test_build.py", line 56, in test_lapack information.""") AssertionError: Both g77 and gfortran runtimes linked in lapack_lite ! This is likely to cause random crashes and wrong results. See numpy INSTALL.txt for more information. Sure enough, scipy built the same way segfaults or fails to import (see [2]). I get no errors for an openblas build. Does anyone recognize these? How should I modify the build to avoid them? Cheers, Matthew [1] https://github.com/pypa/manylinux [2] https://travis-ci.org/matthew-brett/manylinux-testing/jobs/118712090

3 2

Re: [Numpy-discussion] Make np.bincount output same dtype as weights
by Joseph Fox-Rabinovitz 29 Mar '16

29 Mar '16

Would it make sense to just make the output type large enough to hold the cumulative sum of the weights? - Joseph Fox-Rabinovitz ------ Original message------From: Jaime Fernández del RíoDate: Sat, Mar 26, 2016 16:16To: Discussion of Numerical Python;Subject:[Numpy-discussion] Make np.bincount output same dtype as weightsHi all, I have just submitted a PR (#7464) that fixes an enhancement request (#6854), making np.bincount return an array of the same type as the weights parameter. This is an important deviation from current behavior, which always casts weights to double, and always returns a double array, so I would like to hear what others think about the worthiness of this. Main discussion points:np.bincount now works with complex weights (yay!), I guess this should be a pretty uncontroversial enhancement.The return is of the same type as weights, which means that small integers are very likely to overflow. This is exactly what #6854 requested, but perhaps we should promote the output for integers to a long, as we do in np.sum?Boolean arrays stay boolean, and OR, rather than sum, the weights. Is this what one would want? If we decide that integer promotion is the way to go, perhaps booleans should go in the same pack?This new implementation currently supports all of the reasonable native types, but has no fallback for user defined types. I guess we should attempt to cast the array to double as before if no native loop can be found? It would be good to have a way of testing this though, any thoughts on how to go about this?Does a behavior change like this require some deprecation period? What would that look like?I have also added broadcasting of weights to the full size of list, so that one can do e.g. np.bincount([1, 2, 3], weights=2j) without having to tile the single weight to the size of the bins list. Any other thoughts are very welcome as well! Jaime -- (__/) ( O.o) ( > <) Este es Conejo. Copia a Conejo en tu firma y ayúdale en sus planes de dominación mundial.

5 4

numpy in python callback from threaded c++
by Burlen Loring 28 Mar '16

28 Mar '16

Hi All, in my c++ code I've added Python binding via swig. one scenario is to pass a python function to do some computational work. the Python program runs in serial in the main thread but work is handled by a thread pool, the callback is invoked from another thread on unique data. Before a thread invokes the Python callback it acquires Python's GIL. Also I PyEval_InitThreads during module initialization, and have swig'd with -threads flag. However, I'm experiencing frequent crashes when thread pool size is greater than 1, and valgrind is reporting errors from numpy even in case where thread pool size is 1. Here's the essence of the error reported by valgrind: ==10316== Invalid read of size 4 ==10316== at 0x4ED7D73: PyObject_Free (obmalloc.c:1013) ==10316== by 0x10D540B0: NpyIter_Deallocate (nditer_constr.c:699) .... ==10316== Address 0x20034020 is 3,856 bytes inside a block of size 4,097 free'd ==10316== at 0x4C29E00: free (vg_replace_malloc.c:530) ==10316== by 0x4F57B22: import_module_level (import.c:2278) ==10316== by 0x4F57B22: PyImport_ImportModuleLevel (import.c:2292) ==10316== by 0x4F36597: builtin___import__ (bltinmodule.c:49) ==10316== by 0x4E89AC2: PyObject_Call (abstract.c:2546) ==10316== by 0x4E89C1A: call_function_tail (abstract.c:2578) ==10316== by 0x4E89C1A: PyObject_CallFunction (abstract.c:2602) ==10316== by 0x4F58735: PyImport_Import (import.c:2890) ==10316== by 0x4F588B9: PyImport_ImportModule (import.c:2133) ==10316== by 0x10D334C2: get_forwarding_ndarray_method (methods.c:57) ==10316== by 0x10D372C0: array_mean (methods.c:1932) ==10316== by 0x4F40AC7: call_function (ceval.c:4350) There are a few of these reported. I'll attach the full output. This is from the simplest scenario, where the thread pool has a size of 1. Although there are 2 threads, the program is serial as the main thread passes work tasks to the thread pool and waits for work to finish. Here is the work function where above occurs: def execute(port, data_in, req): sys.stderr.write('descriptive_stats::execute MPI %d\n'%(rank)) mesh = as_teca_cartesian_mesh(data_in[0]) table = teca_table.New() table.declare_columns(['step','time'], ['ul','d']) table << mesh.get_time_step() << mesh.get_time() for var_name in var_names: table.declare_columns(['min '+var_name, 'avg '+var_name, \ 'max '+var_name, 'std '+var_name, 'low_q '+var_name, \ 'med '+var_name, 'up_q '+var_name], ['d']*7) var = mesh.get_point_arrays().get(var_name).as_array() table << float(np.min(var)) << float(np.average(var)) \ << float(np.max(var)) << float(np.std(var)) \ << map(float, np.percentile(var, [25.,50.,75.])) return table Again, I'm acquiring the GIL so this should be executed in serial. What am I doing wrong? Have I missed some key aspect of using numpy in this scenario? Any documentation on using numpy in a scenario like this? Any help is greatly appreciated! Thanks Burlen

1 0

Numpy 1.11.0 released
by Charles R Harris 28 Mar '16

28 Mar '16

Hi All, I'm pleased to announce the release of Numpy 1.11.0. This release is the result of six months of work comprising 371 merged pull requests submitted by 112 authors containing many bug fixes and improvements. Highlights are: - The datetime64 type is now timezone naive. - A dtype parameter has been added to ``randint``. - Improved detection of two arrays possibly sharing memory. - Automatic bin size estimation for ``np.histogram``. - Speed optimization of A @ A.T and dot(A, A.T). - New function ``np.moveaxis`` for reordering array axes. Source files and documentation can be found on Sourceforge <https://sourceforge.net/projects/numpy/files/NumPy/1.11.0/>, while source files and OS X wheels for Python 2.7, 3.3, 3.4, and 3.5 can be installed from Pypi. Note that this is the last release to support Python 2.6, 3.2, and 3.3. Contributors are listed below in alphabetical order with the names of new contributors starred. Abdullah Alrasheed* Aditya Panchal* Alain* Alex Griffing Alex Rogozhnikov* Alex Willmer Alexander Heger* Allan Haldane Anatoly Techtonik* Andrew Nelson Anne Archibald Antoine Pitrou Antony Lee* Behzad Nouri Bertrand Lefebvre Blake Griffith Boxiang Sun* Brigitta Sipocz* Carl Kleffner Charles Harris Chris Hogan* Christoph Gohlke Colin Jermain* Cong Ma* Daniel David Freese David Sanders* Dmitry Odzerikho* Dmitry Zagorny* Eric Larson* Eric Moore Ethan Kruse* Eugene Krokhalev* Evgeni Burovski* Francis T. O'Donovan* François Boulogne* Gabi Davar* Gerrit Holl Gopal Singh Meena* Greg Yang* Greg Young* Gregory R. Lee Griffin Hosseinzadeh* Hassan Kibirige* Holger Kohr* Ian Henriksen Iceman9* Jaime Fernandez James Camel* Jason King* John Bjorn Nelson* John Kirkham Jonathan Helmus Jonathan Underwood* Joseph Fox-Rabinovitz* Julian Taylor Julien Dubois* Julien Lhermitte* Julien Schueller* Jörn Hees* Konstantinos Psychas* Lars Buitinck Luke Zoltan Kelley* MaPePeR* Mad Physicist* Mark Wiebe Marten van Kerkwijk Matthew Brett Matthias Geier* Maximilian Trescher* Michael K. Tran* Michael Behrisch* Michael Currie* Michael Löffler* Nathaniel Hellabyte* Nathaniel J. Smith Nick Papior* Nicolas Calle* Nicolás Della Penna* Olivier Grisel Pauli Virtanen Peter Iannucci Phaiax* Ralf Gommers Rehas Sachdeva* Ronan Lamy Ruediger Meier* Ryan Grout* Ryosuke Okuta* Rémy Léone* Sam Radhakrishnan* Samuel St-Jean* Sebastian Berg Simon Conseil* Stephan Hoyer Stuart Archibald* Stuart Berg Sumith* Tapasweni Pathak* Thomas Robitaille Tobias Megies* Tushar Gautam* Varun Nayyar* Vincent Legoll* Warren Weckesser Wendell Smith Yash Mehrotra* Yifan Li* endolith floatingpointstack* ldoddema* yolanda15* Thanks to all who worked on this release. Chuck

1 0