Mailman 3 June 2022 - NumPy-Discussion

Speeding up isin1d and adding a "method" or similar
by Sebastian Berg 23 Jun '22

23 Jun '22

Hi all, there is a PR to add a faster path to `np.isin`, that uses a look-up- table for all the elements that are included in the haystack (`test_elements`): https://github.com/numpy/numpy/pull/12065/files Such a table means that the memory overhead can be very significant, but the speedup as well, so there was the idea of adding an option to pick which version is used. The current documentation for this new `method` keyword argument would be. So the main questions are: * Is there any concern about adding such a new kwarg? * Is `method` the best name? Sorts uses `kind` which may also be good There is also the smaller question of what heuristic 'auto' would use, but that can be tweaked at any time. ``` method : {'auto', 'sort', 'dictionary'}, optional The algorithm to use. This will not affect the final result, but will affect the speed. Default is 'auto'. - If 'sort', will use a mergesort-based approach. This will have a memory usage of roughly 6 times the sum of the sizes of `ar1` and `ar2`, not accounting for size of dtypes. - If 'dictionary', will use a key-dictionary approach similar to a counting sort. This is only available for boolean and integer arrays. This will have a memory usage of the size of `ar1` plus the max-min value of `ar2`. This tends to be the faster method if the following formula is true: `log10(len(ar2)) > (log10(max(ar2)-min(ar2)) - 2.27) / 0.927`, but may use greater memory. - If 'auto', will automatically choose the method which is expected to perform the fastest, using the above formula. For larger sizes or smaller range, 'dictionary' is chosen. For larger range or smaller sizes, 'sort' is chosen.` ``` Cheers, Sebastian

2 3

NumPy 1.23.0 released
by Charles R Harris 23 Jun '22

23 Jun '22

Hi All, On behalf of the NumPy team, I'm pleased to announce the release of NumPy 1.23.0. The NumPy 1.23.0 release continues the ongoing work to improve the handling and promotion of dtypes, increase the execution speed, clarify the documentation, and expire old deprecations. The highlights are: - Implementation of ``loadtxt`` in C, greatly improving its performance. - Exposing DLPack at the Python level for easy data exchange. - Changes to the promotion and comparisons of structured dtypes. - Improvements to f2py. The Python versions supported in this release are 3.8-3.10, 3.11 will be supported when it comes out. Note that 32 bit wheels are only provided for Windows, all other wheels are 64 bits on account of Ubuntu, Fedora, and other Linux distributions dropping 32 bit support. All 64 bit wheels are also linked with 64 bit OpenBLAS. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.23.0/>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.23.0>. *Contributors* A total of 151 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - @DWesl - @GalaxySnail + - @code-review-doctor + - @h-vetinari - Aaron Meurer - Alexander Shadchin - Alexandre de Siqueira - Allan Haldane - Amrit Krishnan - Andrei Batomunkuev - Andrew J. Hesford + - Andrew Murray + - Andrey Andreyevich Bienkowski + - André Elimelek de Weber + - Andy Wharton + - Arryan Singh - Arushi Sharma - Bas van Beek - Bharat Raghunathan - Bhavuk Kalra + - Brigitta Sipőcz - Brénainn Woodsend + - Burlen Loring + - Caio Agiani + - Charles Harris - Chiara Marmo - Cornelius Roemer + - Dahyun Kim + - Damien Caliste - David Prosin + - Denis Laxalde - Developer-Ecosystem-Engineering - Devin Shanahan + - Diego Wang + - Dimitri Papadopoulos Orfanos - Ding Liu + - Diwakar Gupta + - Don Kirkby + - Emma Simon + - Eric Wieser - Evan Miller + - Evgeni Burovski - Evgeny Posenitskiy + - Ewout ter Hoeven + - Felix Divo - Francesco Andreuzzi + - Ganesh Kathiresan - Gaëtan de Menten - Geoffrey Gunter + - Hans Meine - Harsh Mishra + - Henry Schreiner - Hood Chatham + - I-Shen Leong - Ilhan Polat - Inessa Pawson - Isuru Fernando - Ivan Gonzalez + - Ivan Meleshko + - Ivan Yashchuk + - Janus Heide + - Jarrod Millman - Jason Thai + - Jeremy Volkman + - Jesús Carrete Montaña + - Jhong-Ken Chen (陳仲肯) + - John Kirkham - John-Mark Gurney + - Jonathan Deng + - Joseph Fox-Rabinovitz - Jouke Witteveen + - Junyan Ou + - Jérôme Richard + - Kassian Sun + - Kazuki Sakamoto + - Kenichi Maehashi - Kevin Sheppard - Kilian Lieret + - Kushal Beniwal + - Leo Singer - Logan Thomas + - Lorenzo Mammana + - Margret Pax - Mariusz Felisiak + - Markus Mohrhard + - Mars Lee - Marten van Kerkwijk - Masamichi Hosoda + - Matthew Barber - Matthew Brett - Matthias Bussonnier - Matthieu Darbois - Matti Picus - Melissa Weber Mendonça - Michael Burkhart + - Morteza Mirzai + - Motahhar Mokf + - Muataz Attaia + - Muhammad Motawe + - Mukulika Pahari - Márton Gunyhó + - Namami Shanker + - Nihaal Sangha + - Niyas Sait - Omid Rajaei + - Oscar Gustafsson + - Ovee Jawdekar + - P. L. Lim + - Pamphile Roy + - Pantelis Antonoudiou + - Pearu Peterson - Peter Andreas Entschev - Peter Hawkins - Pierre de Buyl - Pieter Eendebak + - Pradipta Ghosh + - Rafael Cardoso Fernandes Sousa + - Raghuveer Devulapalli - Ralf Gommers - Raphael Kruse - Raúl Montón Pinillos - Robert Kern - Rohit Goswami - Ross Barnowski - Ruben Garcia + - Sadie Louise Bartholomew + - Saswat Das + - Sayed Adel - Sebastian Berg - Serge Guelton - Simon Surland Andersen + - Siyabend Ürün + - Somasree Majumder + - Soumya + - Stefan van der Walt - Stefano Miccoli + - Stephan Hoyer - Stephen Worsley + - Tania Allard - Thomas Duvernay + - Thomas Green + - Thomas J. Fan - Thomas Li + - Tim Hoffmann - Ting Sun + - Tirth Patel - Toshiki Kataoka - Tyler Reddy - Warren Weckesser - Yang Hau - Yoon, Jee Seok + Cheers, Charles Harris

1 1

Numpy binary wheels and CI for win/arm64 platform
by Niyas Sait 22 Jun '22

22 Jun '22

Hello, Numpy v1.21.2 <https://github.com/numpy/numpy/releases/tag/v1.21.2> added support for windows/arm64 platforms but we still don't have any systems in place to produce binary wheels or test win/arm64 packages. I think it will be good to start looking into this. CPython has an official buildbot worker running for win/arm64 and official python support for the platform will be available from the 3.11 release. It is not yet clear to me how the build and CI system for numpy is deployed and how to enable support for a new platform like win/arm64. One of the main issues in supporting win/arm64 build would be due to the lack of win/arm64 VMs available on the cloud. But I see we have been producing binary wheels for Apple M1 platforms on pypi and conda repository for some time which also lacks the cloud VM support. I think we could take some learnings from Apple M1 support and look at how a similar strategy can be used for win/arm64. I would like to hear if anyone has any thoughts on this topic. Also, any pointers to understand numpy wheel generation and CI flow for similar platforms would be helpful as well. Thanks, Niyas

4 9

seed for new random number generator
by Pieter Eendebak 21 Jun '22

21 Jun '22

Hi everyone, The new numpy random interface (e.g. r=numpy.random.default_rng; r.random) is much faster than the old one (e.g. np.random.random). When converting code from the old style to the new style I miss having a way to set the seed of the RNG I tried: rng.bit_generator = np.random.PCG64(seed=42) # fails, with good error message rng.bit_generator.state['state']['state']=42 # has no effect, perhaps make this dict read-only? Is there a way to set the seed without creating a new RNG object? With kind regards, Pieter Eendebak

3 4

next NumPy Documentation Team meeting
by Inessa Pawson 19 Jun '22

19 Jun '22

The next NumPy Documentation Team meeting will be held this Monday, June 20th at 16:00 (4 pm) UTC. Join us via Zoom: https://zoom.us/j/96219574921?pwd=VTRNeGwwOUlrYVNYSENpVVBRRjlkZz09 Everyone is welcome to attend and contribute to a conversation. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/oB_boakvRqKR-_2jRV-Qjg Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ Twitter: @inessapawson

1 0

numpy build question
by Jesse Peet 17 Jun '22

17 Jun '22

I am trying to build numpy 1.22.3 in container using a rhel8 host and the container being rhel7. My version of gcc is 4.8.5-44. Here is the error I get building library "npymath" sourcesWARN: Could not locate executable armflang _configtest.cxx: In function ‘int main()’: _configtest.cxx:1:17: warning: ‘auto’ changes meaning in C++11; please remove it [-Wc++0x-compat] int main(void){ auto x = 0.0; return static_cast<int>(x); } ^ _configtest.cxx:1:22: error: ‘x’ does not name a type int main(void){ auto x = 0.0; return static_cast<int>(x); } ^ _configtest.cxx:1:55: error: ‘x’ was not declared in this scope int main(void){ auto x = 0.0; return static_cast<int>(x); } ^ Traceback (most recent call last): File "setup.py", line 461, in <module> setup_package() File "setup.py", line 453, in setup_package setup(**metadata) File "/opt/rh/rh-python38/root/usr/lib64/python3.8/distutils/core.py", line 148, in setup dist.run_commands() File "/opt/rh/rh-python38/root/usr/lib64/python3.8/distutils/dist.py", line 966, in run_commands self.run_command(cmd) File "/opt/rh/rh-python38/root/usr/lib64/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "/opt/rh/rh-python38/root/usr/lib64/python3.8/distutils/command/build.py", line 135, in run self.run_command(cmd_name) File "/opt/rh/rh-python38/root/usr/lib64/python3.8/distutils/cmd.py", line 313, in run_command self.distribution.run_command(command) File "/opt/rh/rh-python38/root/usr/lib64/python3.8/distutils/dist.py", line 985, in run_command cmd_obj.run() File "numpy/core/setup.py", line 708, in get_mathlib_info raise RuntimeError( RuntimeError: Broken toolchain: cannot link a simple C++ program. note: A compiler with support for C++11 language features is required. V/R Jesse S. Peet Configuration Manager Specialist/Engineer Email: jesse.peet(a)raytheon.com Cell: 719-322-4257

4 6

assert_array_equal scalar broadcast behaviour
by Jon Morris 17 Jun '22

17 Jun '22

Hello all, I was recently tripped up by issue #9542<https://github.com/numpy/numpy/issues/9542>, where a call to assert_array_equal unexpectedly passed because a single scalar can be declared equal to an array if every value in the array is the same. I've created pull #21595<https://github.com/numpy/numpy/pull/21595> to address the problem - what does everyone think? Is this an acceptable solution or is there a better way to resolve this issue? Many thanks, Jon Jon Morris Software Developer e: Jon.Morris(a)jbarisk.com d:+44 (0)1756 587229 t: +44 (0)1756 799919 www.jbarisk.com Celebrating 10 Years of The Flood People All JBA Risk Management's email messages contain confidential information and are intended only for the individual(s) named. If you are not the named addressee you should not disseminate, distribute or copy this e-mail. Please notify the sender immediately by email if you have received this email by mistake and delete this email from your system. JBA Risk Management Limited is registered in England, company number 07732946, 1 Broughton Park, Old Lane North, Broughton, Skipton, North Yorkshire, BD23 3FD, England.

2 1

argmin/argmax underspecification?
by TODD ANDERSON 16 Jun '22

16 Jun '22

The docs say "Returns the indices of the maximum values along an axis." When axis=None, I presume the implementation is actually "Returns the first index (assuming ravel ordering) of the maximum value." I also presume that people are now relying on the fact that it always returns the first such index. So, maybe the docs should be updated?

1 0

NumPy Development Meeting Wednesday - Triage Focus
by Sebastian Berg 14 Jun '22

14 Jun '22

Hi all, Our bi-weekly triage-focused NumPy development meeting is Wednesday, June 15st at 16:00 UTC (9:00am Pacific Time). Everyone is invited to join in and edit the work-in-progress meeting topics and notes: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg I encourage everyone to notify us of issues or PRs that you feel should be prioritized, discussed, or reviewed. A calendar to can be found at: https://scientific-python.org/calendars/ please make sure to not import the static `ics` but the url. Best regards Sebastian

1 0

Future of numpy.distutils
by Jerome Kieffer 13 Jun '22

13 Jun '22

Dear Numpy developpers, We are developing a set of scientific tools (https://github.com/silx-kit) and all our build infrastructure is based on `numpy.distutils` which apparently is going to disappear in the coming years. Beside us, the `scipy` project was using it ... Ralf Gommers has ported `scipy` to build with `meson-python` and there are apparently some sharp edges remaining, especially under windows and macos. I wonder if you can comment on the sustainability of this approach or if you would advice us another build tool. Thanks for your insights Cheers, Jerome

4 7