Mailman 3 January 2019 - NumPy-Discussion

summation along a non-fast axis
by Keith Goodman 11 Jan '19

11 Jan '19

I remember back when a.sum(axis=0) was much slower than a.sum(axis=1) for something like a=np.ones((1000, 1000)). But now it runs in about the same time. How does numpy do it? Does numpy do something like for i in range(a.shape[0]): for j in range(x.shape[1]): result[j] += a[i, j]

2 1

[ANN] 2019 Scipy Conference: Call for Proposals
by Nathan Goldbaum 08 Jan '19

08 Jan '19

SciPy 2019, the 18th annual Scientific Computing with Python conference, will be held July 8-14, 2019 in Austin, Texas. The annual SciPy Conference brings together over 800 participants from industry, academia, and government to showcase their latest projects, learn from skilled users and developers, and collaborate on code development. The call for abstracts for SciPy 2019 for talks, posters and tutorials is now open. The deadline for submissions is February 10, 2019. Conference Website: https://www.scipy2019.scipy.org/ Submission Website: https://easychair.org/conferences/?conf=scipy2019 Talks and Posters (July 10-12, 2019) In addition to the general track, this year will have specialized tracks focused on: - Data Driven Discoveries (including Machine Learning and Data Science) - Open Source Communities (Sustainability) Mini Symposia - Science Communication through Visualization - Neuroscience and Cognitive Science - Image Processing - Earth, Ocean, Geo and Atmospheric Science There will also be a SciPy Tools Plenary Session each day with 2 to 5 minute updates on tools and libraries. Tutorials (July 8-9, 2019) Tutorials should be focused on covering a well-defined topic in a hands-on manner. We are looking for useful techniques or packages, helping new or advanced Python programmers develop better or faster scientific applications. We encourage submissions to be designed to allow at least 50% of the time for hands-on exercises even if this means the subject matter needs to be limited. Tutorials will be 4 hours in duration. In your tutorial application, you can indicate what prerequisite skills and knowledge will be needed for your tutorial, and the approximate expected level of knowledge of your students (i.e., beginner, intermediate, advanced). Instructors of accepted tutorials will receive a stipend.

1 0

Reminder: weekly status meeting Jan 9 at 12:00 pacific time
by Matti Picus 08 Jan '19

08 Jan '19

The draft agenda is at https://hackmd.io/D_0X2QCjRpS-ENiFWokc-g?both# There is a section for community suggested topics, feel free to join the conversation and add in topics that need attention. The BIDS team

1 0

timedelta64 remainder behavior with div by 0
by Tyler Reddy 08 Jan '19

08 Jan '19

We are now at the stage of implementing the timedelta64 divmod inner loop given very recent additions of floordiv and remainder inner loops for this data type. However, there is some contention about a previous decision regarding modulus behavior that we'd like to resolve before we bake it in to divmod. Currently, a modulus operation with two timedelta64 operands with a 0 denominator returns 0. For example: np.timedelta64(5) % np.timedelta64(0) -> numpy.timedelta64(0) In contrast, np.float64(1) % np.float64(0) -> nan There's a suggestion that we should switch to returning NaT for the timedelta64 case for consistency, and that this probably isn't too harmful given how recent these additions are. Do we have consensus on this? Ref: https://github.com/numpy/numpy/pull/12683 Thanks! Tyler

4 4

Pairwise summation
by Keith Goodman 07 Jan '19

07 Jan '19

Numpy uses pairwise summation along the fast axis if that axis contains no more than 8192 elements. How was 8192 chosen? Doubling to 16384 would result in a lot more function call overhead due to the recursion. Is it a speed issue? Memory? Or something else entirely?

2 1

Issue with setup_requires and 1.16 release candidates
by Thomas Robitaille 07 Jan '19

07 Jan '19

Hi all, Back in December I started getting failures in continuous integration as well as reports of failures from users of installation issues for a couple of packages. The problem can be easily reproduced in a Docker container with: FROM ubuntu:16.04 RUN apt-get update RUN apt-get install -y python3 python3-dev python3-pip python3-wheel RUN pip3 install fast-histogram RUN python3 -c 'import fast_histogram' Doing this results in the following traceback: ImportError: No module named 'numpy.core._multiarray_umath' Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/local/lib/python3.5/dist-packages/fast_histogram/__init__.py", line 1, in <module> from .histogram import * File "/usr/local/lib/python3.5/dist-packages/fast_histogram/histogram.py", line 7, in <module> from ._histogram_core import (_histogram1d, ImportError: numpy.core.multiarray failed to import I've seen similar issues with other packages too. The key is that the fast-histogram package defines: setup_requires=['numpy'] in the setup.py (since the package has a C extension that uses the Numpy C API) and numpy is needed before the install_requires dependencies are installed: https://github.com/astrofrog/fast-histogram/blob/master/setup.py Now this normally works fine, but the issues I saw started when the first 1.16 RC was made available, when installing into an environment in which numpy is not already installed. My understanding is that setup_requires is honored by easy_install (even if installing the main package with pip), which doesn't ignore pre-releases. Thus, the package is built against the 1.16 RC but then 1.15 is installed due to: install_requires=['numpy'] which is honored by pip. I think that the correct solution is to make sure that: [build-system] requires = ["setuptools", "wheel", "numpy"] is added to the pyproject.toml file (as per PEP 518). This then works properly with recent versions of pip (>10 I think). I think removing setup_requires then makes sense because it'd be better to have an error with older versions of pip that numpy is not installed rather than having the package built against the wrong version. I know this is a temporary issue in the sense that it will go away once the final version of 1.16 is out, but I just wanted to share this as a heads-up in case you get reports of issues from other people, and also to check whether there are any other solutions/workarounds to be aware of? (to avoid a similar situation in future). Thanks, Tom

2 2

NumPy 1.16.0rc2 released
by Charles R Harris 05 Jan '19

05 Jan '19

Hi All, On behalf of the NumPy team I'm pleased to announce the release of NumPy 1.16.0rc2. This is the last NumPy release to support Python 2.7 and will be maintained as a long term release with bug fixes until 2020. This release has seen a lot of refactoring and features many bug fixes, improved code organization, and better cross platform compatibility. Not all of these improvements will be visible to users, but they should help make maintenance easier going forward. Highlights are - Experimental support for overriding numpy functions in downstream projects. - The matmul function is now a ufunc and can be overridden using __array_ufunc__. - Improved support for the ARM and POWER architectures. - Improved support for AIX and PyPy. - Improved interoperation with ctypes. - Improved support for PEP 3118. The supported Python versions are 2.7 and 3.5-3.7, support for 3.4 has been dropped. The wheels on PyPI are linked with OpenBLAS v0.3.4+, which should fix the known threading issues found in previous OpenBLAS versions. Downstream developers building this release should use Cython >= 0.29.2 and, if linking OpenBLAS, OpenBLAS > v0.3.4. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.16.0rc2/>, source archives are available from Github <https://github.com/numpy/numpy/releases/tag/v1.16.0rc2>. *Contributors* A total of 112 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Alan Fontenot + * Allan Haldane * Alon Hershenhorn + * Alyssa Quek + * Andreas Nussbaumer + * Anner + * Anthony Sottile + * Antony Lee * Ayappan P + * Bas van Schaik + * C.A.M. Gerlach + * Charles Harris * Chris Billington * Christian Clauss * Christoph Gohlke * Christopher Pezley + * Daniel B Allan + * Daniel Smith * Dawid Zych + * Derek Kim + * Dima Pasechnik + * Edgar Giovanni Lepe + * Elena Mokeeva + * Elliott Sales de Andrade + * Emil Hessman + * Eric Schles + * Eric Wieser * Giulio Benetti + * Guillaume Gautier + * Guo Ci * Heath Henley + * Isuru Fernando + * J. Lewis Muir + * Jack Vreeken + * Jaime Fernandez * James Bourbeau * Jeff VanOss * Jeffrey Yancey + * Jeremy Chen + * Jeremy Manning + * Jeroen Demeyer * John Darbyshire + * John Kirkham * John Zwinck * Jonas Jensen + * Joscha Reimer + * Juan Azcarreta + * Julian Taylor * Kevin Sheppard * Krzysztof Chomski + * Kyle Sunden * Lars Grüter * Lilian Besson + * MSeifert04 * Mark Harfouche * Marten van Kerkwijk * Martin Thoma * Matt Harrigan + * Matthew Bowden + * Matthew Brett * Matthias Bussonnier * Matti Picus * Max Aifer + * Michael Hirsch, Ph.D + * Michael James Jamie Schnaitter + * MichaelSaah + * Mike Toews * Minkyu Lee + * Mircea Akos Bruma + * Mircea-Akos Brumă + * Moshe Looks + * Muhammad Kasim + * Nathaniel J. Smith * Nikita Titov + * Paul Müller + * Paul van Mulbregt * Pauli Virtanen * Pierre Glaser + * Pim de Haan * Ralf Gommers * Robert Kern * Robin Aggleton + * Rohit Pandey + * Roman Yurchak + * Ryan Soklaski * Sebastian Berg * Sho Nakamura + * Simon Gibbons * Stan Seibert + * Stefan Otte * Stefan van der Walt * Stephan Hoyer * Stuart Archibald * Taylor Smith + * Tim Felgentreff + * Tim Swast + * Tim Teichmann + * Toshiki Kataoka * Travis Oliphant * Tyler Reddy * Uddeshya Singh + * Warren Weckesser * Weitang Li + * Wenjamin Petrenko + * William D. Irons * Yannick Jadoul + * Yaroslav Halchenko * Yug Khanna + * Yuji Kanagawa + * Yukun Guo + * ankokumoyashi + * lerbuke + Cheers, Charles Harris

1 0

Ia `-fno-strict-aliasing` still required?
by Charles R Harris 04 Jan '19

04 Jan '19

Hi All, Just asking if the `-fno-strict-aliasing` flag is still required for gcc. Supposedly `-fstrict-aliasing` is enabled by default with optimization levels >= `-O2` and that used to result in errors. However, I have noticed that numpy wheels are being built without the `-fno-strict-aliasing` flag and doing fine. Same on my local machine. So maybe the compiler has gotten better at knowing when pointers may be aliased? I cannot find informative documentation on how `-fstrict-aliasing` is actually implemented, so thought I'd ask here if someone knows more about it. Chuck

3 4

Adding more detailed exception types to numpy
by Eric Wieser 04 Jan '19

04 Jan '19

PR #12593 <https://github.com/numpy/numpy/pull/12593> adds a handful of new exception types for . Some consequences of this change are that: 1. The string formatting is moved to python, allowing us to give better error messages without a lot of work 2. The formatting is dispatched lazily, meaning that users trying ufuncs, but catching unsupported loop types don’t have to pay the cost of string formatting 3. Users can catch a specific exception type. This might expose more of our internals than we’re willing to though. 4. We need to expose these new exception names in the public API. 3 & 4 raise some questions, which I’d like some feedback on: - Should we actually expose the detailed exception types to the user, or should they be kept an implementation detail? One way to hide the implementation details would be class TypeError(builtins.TypeError): """actually UFuncCastingError""" _UFuncCastingError = TypeError # for internal use when raising _UFuncCastingError.__module__ = None class TypeError(builtins.TypeError): """actually UFuncLoopError""" _UFuncLoopError= TypeError # for internal use when raising _UFuncLoopError .__module__ = None del TypeError This gives us all the advantages of 1 & 2 without the user noticing that they’re receiving anything other than TypeError, which their tracebacks will continue to show. - If we do expose them, where should these exceptions go? In the past, we also added AxisError and TooHardError - it would be nice to start being consistent about where to expose these things 1. np.UFuncCastingError (expands the global namespace even further) 2. np.core._exceptions.UFuncCastingError (shouldn’t really be private, since users won’t know how to catch it) 3. np.core.UFuncCastingError 4. np.core.umath.CastingError 5. A dedicated namespace for exceptions: - np.errors.UFuncCastingError (matches pandas) - np.exceptions.UFuncCastingError - np.exc.UFuncCastingError (matches sqlalchemy) Eric

3 2

Testing radix sort on big-endian architectures
by Hameer Abbasi 03 Jan '19

03 Jan '19

Hello. I’m writing a PR (#12586 (https://github.com/numpy/numpy/pull/12586)) for radix sort, and since some operations involve bit-shifts, the concern came up that big-endian architectures might not support it. If someone has access to a machine with a big-endian architecture, it would be nice if they could run the test suite on that PR. Best Regards, Hameer Abbasi

1 0