Mailman 3 June 2022 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker July 11, 2023

July 11, 2023

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem Feb. 18, 2023

Feb. 18, 2023

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How … [View More]

5 4

Documentation Team meeting - Monday June 8th
by Melissa Mendonça Dec. 3, 2022

Dec. 3, 2022

Hi all! A reminder that on Monday, June 8, we have another documentation team meeting at 3PM UTC**. If you wish to join on Zoom, you need to use this link https://zoom.us/j/420005230 Here's the permanent hackmd document with the meeting notes: https://hackmd.io/oB_boakvRqKR-_2jRV-Qjg <https://www.google.com/url?q=https%3A%2F%2Fhackmd.io%2FoB_boakvRqKR-_2jRV-Q…> Hope to see you around (especially if you want to introduce yourself or discuss ideas for Google Season of Docs). ** You … [View More]

6 74

next NumPy Newcomers' Hour
by Inessa Pawson Oct. 2, 2022

Oct. 2, 2022

Our next Newcomers' Hour will be held this Thursday, June 16th, at 4 pm UTC. Stop by to ask questions or just to say hi. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/3f3otyyuTte3FU9y3QzsLg?both Join the meeting via Zoom: https://us02web.zoom.us/j/87192457898 Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ Twitter: @inessapawson

1 3

An extension of the .npy file format
by Michael Siebert Aug. 25, 2022

Aug. 25, 2022

Dear all, originally, I have planned to make an extension of the .npy file format a dedicated follow-up pull request, but I have upgraded my current request instead, since it was not as difficult to implement as I initially thought and probably a more straight-forward solution: https://github.com/numpy/numpy/pull/20321/ What is this pull request about? It is about appending to Numpy .npy files. Why? I see two main use cases: 1. creating .npy files larger than the main memory. They can, … [View More]once finished, be loaded as memory maps 2. creating binary log files, which can be processed very efficiently without parsing Are there not other good file formats to do this? Theoretically yes, but practically they can be pretty complex and with very little tweaking .npy could do efficient appending too. Use case 1 is already covered by the Pip/Conda package npy-append-array I have created and getting the functionality directly into Numpy was the original goal of the pull request. This would have been possible without introducing a new file format version, just by adding some spare space in the header. During the pull request discussion it turned out that rewriting the header after each append would be desirable in case the writing program crashes to minimize data loss. Use case 2 however would highly profit from a new file format version as it would make rewriting the header unnecessary: since efficient appending can only take place along one axis, setting shape[-1] = -1 in case of Fortran order or shape[0] = -1 otherwise (default) in the .npy header on file creation could indicate that the array size is determined by the file size: when np.load (typically with memory mapping on) gets called, it constructs the ndarray with the actual shape by replacing the -1 in the constructor call. Otherwise, the header is not modified anymore, neither on append nor on file write finish. Concurrent appends to a single file would not be advisable and should be channeled through a single AppendArray instance. Concurrent reads while writes take place however should work relatively smooth: every time an np.load (ideally with mmap) is called, the ndarray would provide access to all data written until that time. Currently, my pull request provides: 1. A definition of .npy version 4.0 that supports -1 in the shape 2. implementations for fortran order and non-fortran order (default) including test cases 3. Updated np.load 4. The AppendArray class that does the actual appending Although there is a certain hassle with introducing a new .npy version, the changes themselves are very small. I could also implement a fallback mode for older Numpy installations, if someone is interested. What do you think about such a feature, would it make sense? Anyone available for some more code review? Best from Berlin, Michael PS thank you so far, I could improve my npy-append-array module as well and from what I have seen so far the Numpy code readability exceeded my already high expectations. [View Less]

5 7

NumPy community meeting
by Inessa Pawson July 31, 2022

July 31, 2022

The next NumPy community meeting will be held this Wednesday, May 25th at 18:00 (6 pm) UTC. Join us via Zoom: https://berkeley.zoom.us/j/762261535 Everyone is welcome and encouraged to attend. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy

1 5

copy="never" discussion and no deprecation cycle?
by Sebastian Berg July 21, 2022

July 21, 2022

Hi all, (sorry for the length, details/discussion below) On the triage call, there seemed a preference to just try to skip the deprecation and introduce `copy="never"`, `copy="if_needed"`, and `copy="always"` (i.e. string options for the `copy` keyword argument). Strictly speaking, this is against the typical policy (one year of warning/errors). But nobody could think of a reasonable chance that anyone actually uses it. (For me just "policy" will be enough of an argument to just take it … [View More]slow.) BUT: If nobody has *any* concerns at all, I think we may just end up introducing the change right away. The PR is: https://github.com/numpy/numpy/pull/19173 ## The Feature There is the idea to add `copy=never` (or similar). This would modify the existing `copy` argument to make it a 3-way decision: * `copy=always` or `copy=True` to force a copy * `copy=if_needed` or `copy=False` to prefer no-copy behavior * `copy=never` to error when no-copy behavior is not possible (this ensures that a view is returned) this would affect the functions: * np.array(object, copy=...) * arr.astype(new_dtype, copy=...) * np.reshape(arr, new_shape, copy=...), and the method arr.reshape() * np.meshgrid and possibly Where `reshape` currently does not have the option and would benefit by allowing for `arr.reshape(-1, copy=never)`, which would guarantee a view. ## The Options We have three options that are currently being discussed: 1. We introduce a new `np.CopyMode` or `np.<something>.Copy` Enum with values `np.CopyMode.NEVER`, `np.CopyMode.IF_NEEDED`, and `np.CopyMode.ALWAYS` * Plus: No compatibility concerns * Downside(?): This would be a first in NumPy, and is untypical API due to that. 2. We introduce `copy="never"`, `copy="if_needed"` and `copy="always"` as strings (all other strings will be a `TypeError`): * Problem: `copy="never"` currently means `copy=True` (the opposite) Which means new code has to take care when it may run on older NumPy versions. And in theory could make old code return the wrong thing. * Plus: Strings are the typical for options in NumPy currently. 3. Same as 2. But we take it very slow: Make strings an error right now and only introduce the new options after two releases as per typical deprecation policy. ## Discussion We discussed it briefly today in the triage call and we were leaning towards strings. I was honestly expecting to converge to option 3 to avoid compatibility issues (mainly surprises with `copy="never"` on older versions). But considering how weird it is to currently pass `copy="never"`, the question was whether we should not change it with a release note. The probability of someone currently passing exactly one of those three (and no other) strings seems exceedingly small. Personally, I don't have a much of an opinion. But if *nobody* voices any concern about just changing the meaning of the string inputs, I think the current default may be to just do it. Cheers, Sebastian [View Less]

12 28

NumPy Newcomers' Hour – Thursday, March 24th
by Inessa Pawson July 10, 2022

July 10, 2022

Our next Newcomers' Hour will be held tomorrow, March 24th, at 4 pm UTC. We have no agenda this time. Stop by to ask questions or just to say hi. Join the meeting via Zoom: https://us02web.zoom.us/j/87192457898 Cheers, Inessa Inessa Pawson NumPy Contributor Experience Lead

1 4

NEP 50: Promotion rules for Python scalars
by Sebastian Berg July 6, 2022

July 6, 2022

Hi all, I would like to share the first formal draft of NEP 50: Promotion rules for Python scalars with everyone. The full text can be found here: https://numpy.org/neps/nep-0050-scalar-promotion.html NEP 50 is an attempt to remove value-based casting/promotion. We wish to replace it with clearer rules for the resulting dtype when mixing NumPy arrays and Python scalars. As a brief example, the proposal allows the following (unchanged): >>> np.array([1, 2, 3], dtype=… [View More]

3 5

next NumPy Newcomers Hour
by Inessa Pawson July 5, 2022

July 5, 2022

The next NumPy Newcomers Hour will be held this Thursday, June 30th at 4 pm UTC. Ryan C. Cooper, an assistant professor-in-residence at the University of Connecticut (Mansfield, Connecticut, USA), will share how he uses NumPy in his Engineering classes, from individual student research to semester-long courses. We will talk about lessons learned and key strategies to motivate and engage new Python users. Join us via Zoom: https://us02web.zoom.us/j/87192457898 Cheers, Inessa Inessa Pawson … [View More]

1 1

Fuzzing integration of Numpy into OSS-Fuzz
by david korczynski July 2, 2022

July 2, 2022

Hi Numpy maintainers, Would you be interested in integrating continuous fuzzing by way of OSS-Fuzz? Fuzzing is a way to automate test-case generation and has been heavily used for memory unsafe languages. Recently efforts have been put into fuzzing memory safe languages and Python is one of the languages where it would be great to use fuzzing. In this PR: https://github.com/google/oss-fuzz/pull/7681 I did an initial integration into OSS-Fuzz. Essentially, OSS-Fuzz is a free service run by … [View More]

6 7

Feature request: function to get minimum and maximum values simultaneously (as a tuple)
by Ewout ter Hoeven July 1, 2022

July 1, 2022

A function to get the minimum and maximum values of an array simultaneously could be very useful, from both a convenience and performance point of view. Especially when arrays get larger the performance benefit could be significant, and even more if the array doesn't fit in L2/L3 cache or even memory. There are many cases where not either the minimum or the maximum of an array is required, but both. Think of clipping an array, getting it's range, checking for outliers, normalizing, making a … [View More]

5 4

Setting C and linker flags for Windows build
by Matthew Brett June 30, 2022

June 30, 2022

Hi, I am very sorry - I feel I should know this, or be able to work it out, but is there a way of setting the flags to the C compiler and the linker, for the Numpy build, on Windows? I'm trying to set the flags for a build with Windows mingw-w64 - but I believe Numpy is ignoring $env:LDFLAGS, $env:CFLAGS and $env:OPT - and I can't see any way of setting these options from the command line. Am I missing something? Cheers, Matthew

4 14

Changes to `np.loadtxt`
by Sebastian Berg June 29, 2022

June 29, 2022

Hi all, just a brief heads up that: https://github.com/numpy/numpy/pull/20580 is now merged. This moves `np.loadtxt` to C. Mainly making it much faster. There are also some other improvements and changes though: * It now supports `quotechar='"'` to support Excel dialect CSV. * Parsing some numbers is stricter (e.g. removed support for `_` or hex float parsing by default). * `max_rows` now actually counts rows and not lines. A warning is given if this makes a difference (blank … [View More]

1 1

Speeding up `unique` and adding "kind" parameter
by Miles Cranmer June 29, 2022

June 29, 2022

Dear all, There is a PR that adds a lookup table approach to `unique`, shown below. You can get up to ~16x speedup for large integer arrays, at the cost of potentially greater memory usage. https://github.com/numpy/numpy/pull/21843 This is controlled by a new `kind` parameter, which is described below. The current approach uses "sort" while the proposed change implements the "table" method. The default option None will automatically select "table" when possible and memory usage is not too … [View More]

4 11

NumPy Development Meeting Wednesday - Triage Focus
by Sebastian Berg June 27, 2022

June 27, 2022

Hi all, Our bi-weekly triage-focused NumPy development meeting is Wednesday, June 29th at 16:00 UTC (9:00am Pacific Time). Everyone is invited to join in and edit the work-in-progress meeting topics and notes: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg I encourage everyone to notify us of issues or PRs that you feel should be prioritized, discussed, or reviewed. A calendar to can be found at: https://scientific-python.org/calendars/ please make sure to not import the static `ics` but the url. … [View More]

1 0

ANN: NumExpr 2.8.3
by Robert McLeod June 25, 2022

June 25, 2022

Hi everyone, Please find here another maintenance release of NumExpr. Support for Python 3.6 has been dropped to enable support for NumPy 1.23 (and by extension Python 3.11 when it is released). Wheels for ARM64 multilinux should be available again after troubles with GitHub Actions and Apple Silicon wheels are also now available on PyPi for download. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.8.1 to 2.8.2 --------------------------- * Support for … [View More]Python 3.6 has been dropped due to the need to substitute the flag `NPY_ARRAY_WRITEBACKIFCOPY` for `NPY_ARRAY_UPDATEIFCOPY`. This flag change was initiated in NumPy 1.14 and finalized in 1.23. The only changes were made to cases where an unaligned constant was passed in with a pre-allocated output variable: ``` x = np.empty(5, dtype=np.uint8)[1:].view(np.int32) ne.evaluate('3', out=x) ``` We think the risk of issues is very low, but if you are using NumExpr as a expression evaluation tool you may want to write a test for this edge case. * Thanks to Matt Einhorn (@matham) for improvements to the GitHub Actions build process to add support for Apple Silicon and aarch64. * Thanks to Biswapriyo Nath (@biswa96) for a fix to allow `mingw` builds on Windows. * There have been some changes made to not import `platform.machine()` on `sparc` but it is highly advised to upgrade to Python 3.9+ to avoid this issue with the Python core package `platform`. What's Numexpr? --------------- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? ------------------------- The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience --------------------- Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod robbmcleod(a)gmail.com robert.mcleod(a)hitachi-hightech.com [View Less]

1 0

No PDF documentation
by Charles R Harris June 24, 2022

June 24, 2022

Hi All, There is a Makefile PR <https://github.com/numpy/numpy/pull/21849> that, among other things, removes the generation of PDF files. This will require changes to the documentation files and I'm wondering exactly how we want to proceed with that. Note that the zipped html files are considerably larger than the two generated PDF file together, about 30 MB, and users will need to download that, unzip it, and click on the index file. Is there anyway to make that simpler? Do we need to? Chuck

1 0

ANN: SciPy 1.9.0rc1 -- please test
by Tyler Reddy June 24, 2022

June 24, 2022

Hi all, On behalf of the SciPy development team I'm pleased to announce the release candidate SciPy 1.9.0rc1. Please help us test this pre-release. Sources and binary wheels can be found at: https://pypi.org/project/scipy/ and at: https://github.com/scipy/scipy/releases/tag/v1.9.0rc1 One of a few ways to install the release candidate with pip: pip install scipy==1.9.0rc1 ========================== SciPy 1.9.0 Release Notes ========================== Note: Scipy 1.9.0 is not released yet! … [View More]

1 0

StackOverflow Developer Survey
by Michael Siebert June 23, 2022

June 23, 2022

Hi all, just found some more survey data on Numpy, see (need to scroll down a little to „Other frameworks and libraries“): https://survey.stackoverflow.co/2022 Numpy seems to enjoy an exceptional position as a library over a wide spectrum of programming languages: rank 2 overall and among professionals and even rank 1 with beginners. Well deserved! Best, Michael

1 0

Speeding up isin1d and adding a "method" or similar
by Sebastian Berg June 23, 2022

June 23, 2022

Hi all, there is a PR to add a faster path to `np.isin`, that uses a look-up- table for all the elements that are included in the haystack (`test_elements`): https://github.com/numpy/numpy/pull/12065/files Such a table means that the memory overhead can be very significant, but the speedup as well, so there was the idea of adding an option to pick which version is used. The current documentation for this new `method` keyword argument would be. So the main questions are: * Is there any … [View More]

2 3

NumPy 1.23.0 released
by Charles R Harris June 22, 2022

June 22, 2022

Hi All, On behalf of the NumPy team, I'm pleased to announce the release of NumPy 1.23.0. The NumPy 1.23.0 release continues the ongoing work to improve the handling and promotion of dtypes, increase the execution speed, clarify the documentation, and expire old deprecations. The highlights are: - Implementation of ``loadtxt`` in C, greatly improving its performance. - Exposing DLPack at the Python level for easy data exchange. - Changes to the promotion and comparisons of … [View More]structured dtypes. - Improvements to f2py. The Python versions supported in this release are 3.8-3.10, 3.11 will be supported when it comes out. Note that 32 bit wheels are only provided for Windows, all other wheels are 64 bits on account of Ubuntu, Fedora, and other Linux distributions dropping 32 bit support. All 64 bit wheels are also linked with 64 bit OpenBLAS. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.23.0/>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.23.0>. *Contributors* A total of 151 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - @DWesl - @GalaxySnail + - @code-review-doctor + - @h-vetinari - Aaron Meurer - Alexander Shadchin - Alexandre de Siqueira - Allan Haldane - Amrit Krishnan - Andrei Batomunkuev - Andrew J. Hesford + - Andrew Murray + - Andrey Andreyevich Bienkowski + - André Elimelek de Weber + - Andy Wharton + - Arryan Singh - Arushi Sharma - Bas van Beek - Bharat Raghunathan - Bhavuk Kalra + - Brigitta Sipőcz - Brénainn Woodsend + - Burlen Loring + - Caio Agiani + - Charles Harris - Chiara Marmo - Cornelius Roemer + - Dahyun Kim + - Damien Caliste - David Prosin + - Denis Laxalde - Developer-Ecosystem-Engineering - Devin Shanahan + - Diego Wang + - Dimitri Papadopoulos Orfanos - Ding Liu + - Diwakar Gupta + - Don Kirkby + - Emma Simon + - Eric Wieser - Evan Miller + - Evgeni Burovski - Evgeny Posenitskiy + - Ewout ter Hoeven + - Felix Divo - Francesco Andreuzzi + - Ganesh Kathiresan - Gaëtan de Menten - Geoffrey Gunter + - Hans Meine - Harsh Mishra + - Henry Schreiner - Hood Chatham + - I-Shen Leong - Ilhan Polat - Inessa Pawson - Isuru Fernando - Ivan Gonzalez + - Ivan Meleshko + - Ivan Yashchuk + - Janus Heide + - Jarrod Millman - Jason Thai + - Jeremy Volkman + - Jesús Carrete Montaña + - Jhong-Ken Chen (陳仲肯) + - John Kirkham - John-Mark Gurney + - Jonathan Deng + - Joseph Fox-Rabinovitz - Jouke Witteveen + - Junyan Ou + - Jérôme Richard + - Kassian Sun + - Kazuki Sakamoto + - Kenichi Maehashi - Kevin Sheppard - Kilian Lieret + - Kushal Beniwal + - Leo Singer - Logan Thomas + - Lorenzo Mammana + - Margret Pax - Mariusz Felisiak + - Markus Mohrhard + - Mars Lee - Marten van Kerkwijk - Masamichi Hosoda + - Matthew Barber - Matthew Brett - Matthias Bussonnier - Matthieu Darbois - Matti Picus - Melissa Weber Mendonça - Michael Burkhart + - Morteza Mirzai + - Motahhar Mokf + - Muataz Attaia + - Muhammad Motawe + - Mukulika Pahari - Márton Gunyhó + - Namami Shanker + - Nihaal Sangha + - Niyas Sait - Omid Rajaei + - Oscar Gustafsson + - Ovee Jawdekar + - P. L. Lim + - Pamphile Roy + - Pantelis Antonoudiou + - Pearu Peterson - Peter Andreas Entschev - Peter Hawkins - Pierre de Buyl - Pieter Eendebak + - Pradipta Ghosh + - Rafael Cardoso Fernandes Sousa + - Raghuveer Devulapalli - Ralf Gommers - Raphael Kruse - Raúl Montón Pinillos - Robert Kern - Rohit Goswami - Ross Barnowski - Ruben Garcia + - Sadie Louise Bartholomew + - Saswat Das + - Sayed Adel - Sebastian Berg - Serge Guelton - Simon Surland Andersen + - Siyabend Ürün + - Somasree Majumder + - Soumya + - Stefan van der Walt - Stefano Miccoli + - Stephan Hoyer - Stephen Worsley + - Tania Allard - Thomas Duvernay + - Thomas Green + - Thomas J. Fan - Thomas Li + - Tim Hoffmann - Ting Sun + - Tirth Patel - Toshiki Kataoka - Tyler Reddy - Warren Weckesser - Yang Hau - Yoon, Jee Seok + Cheers, Charles Harris [View Less]

1 1

Numpy binary wheels and CI for win/arm64 platform
by Niyas Sait June 22, 2022

June 22, 2022

Hello, Numpy v1.21.2 <https://github.com/numpy/numpy/releases/tag/v1.21.2> added support for windows/arm64 platforms but we still don't have any systems in place to produce binary wheels or test win/arm64 packages. I think it will be good to start looking into this. CPython has an official buildbot worker running for win/arm64 and official python support for the platform will be available from the 3.11 release. It is not yet clear to me how the build and CI system for numpy is deployed … [View More]

4 9

seed for new random number generator
by Pieter Eendebak June 21, 2022

June 21, 2022

Hi everyone, The new numpy random interface (e.g. r=numpy.random.default_rng; r.random) is much faster than the old one (e.g. np.random.random). When converting code from the old style to the new style I miss having a way to set the seed of the RNG I tried: rng.bit_generator = np.random.PCG64(seed=42) # fails, with good error message rng.bit_generator.state['state']['state']=42 # has no effect, perhaps make this dict read-only? Is there a way to set the seed without creating a new RNG … [View More]

3 4

next NumPy Documentation Team meeting
by Inessa Pawson June 18, 2022

June 18, 2022

The next NumPy Documentation Team meeting will be held this Monday, June 20th at 16:00 (4 pm) UTC. Join us via Zoom: https://zoom.us/j/96219574921?pwd=VTRNeGwwOUlrYVNYSENpVVBRRjlkZz09 Everyone is welcome to attend and contribute to a conversation. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/oB_boakvRqKR-_2jRV-Qjg Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ Twitter: @inessapawson

1 0