Mailman 3 October 2023 - NumPy-Discussion

next NumPy triage meeting - now starting at 5 pm UTC
by Inessa Pawson 26 Nov '23

26 Nov '23

The next NumPy triage meeting will be held this Wednesday, January 11th at 5 pm UTC. (Please note the new time!) This is a meeting where we synchronously triage prioritized PRs and issues. Join us via Zoom: https://numfocus-org.zoom.us/j/82096749952?pwd=MW9oUmtKQ1c3a2gydGk1RTdYUUVX… Everyone is welcome to attend and contribute to a conversation. Please notify us of issues or PRs that you’d like to have reviewed by adding a GitHub link to them in the meeting agenda: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg. -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

1 23

*New Time* Next Documentation team meeting
by Mukulika Pahari 20 Nov '23

20 Nov '23

Hi all, sorry for the late notice. Our next Documentation Team meeting will happen on *Monday, October 9* at *11PM UTC*. If this time slot is inconvenient for you to join, please let me know in the replies or Slack and we will try to add another time slot. All are welcome - you don't need to already be a contributor to join. If you have questions or are curious about what we're doing, we'll be happy to meet you! If you wish to join on Zoom, use this (updated) link: https://numfocus-org.zoom.us/j/85016474448?pwd=TWEvaWJ1SklyVEpwNXUrcHV1YmFJQ... Here's the permanent hackmd document with the meeting notes (still being updated): https://hackmd.io/oB_boakvRqKR-_2jRV-Qjg Hope to see you around! Best wishes, Mukulika

2 3

Function that searches arrays for the first element that satisfies a condition
by rosko37 01 Nov '23

01 Nov '23

I know this question has been asked before, both on this list as well as several threads on Stack Overflow, etc. It's a common issue. I'm NOT asking for how to do this using existing Numpy functions (as that information can be found in any of those sources)--what I'm asking is whether Numpy would accept inclusion of a function that does this, or whether (possibly more likely) such a proposal has already been considered and rejected for some reason. The task is this--there's a large array and you want to find the next element after some index that satisfies some condition. Such elements are common, and the typical number of elements to be searched through is small relative to the size of the array. Therefore, it would greatly improve performance to avoid testing ALL elements against the conditional once one is found that returns True. However, all built-in functions that I know of test the entire array. One can obviously jury-rig some ways, like for instance create a "for" loop over non-overlapping slices of length slice_length and call something like np.where(cond) on each--that outer "for" loop is much faster than a loop over individual elements, and the inner loop at most will go slice_length-1 elements past the first "hit". However, needing to use such a convoluted piece of code for such a simple task seems to go against the Numpy spirit of having one operation being one function of the form func(arr)". A proposed function for this, let's call it "np.first_true(arr, start_idx, [stop_idx])" would be best implemented at the C code level, possibly in the same code file that defines np.where. I'm wondering if I, or someone else, were to write such a function, if the Numpy developers would consider merging it as a standard part of the codebase. It's possible that the idea of such a function is bad because it would violate some existing broadcasting or fancy indexing rules. Clearly one could make it possible to pass an "axis" argument to np.first_true() that would select an axis to search over in the case of multi-dimensional arrays, and then the result would be an array of indices of one fewer dimension than the original array. So np.first_true(np.array([1,5],[2,7],[9,10],cond) would return [1,1,0] for cond(x): x>4. The case where no elements satisfy the condition would need to return a "signal value" like -1. But maybe there are some weird cases where there isn't a sensible return value, hence why such a function has not been added. -Andrew Rosko

7 13

next NumPy Newcomers' Hour - 12 pm UTC
by Inessa Pawson 31 Oct '23

31 Oct '23

Our next Newcomers' Hour will be held tomorrow, December 1st, at 12 pm UTC. Stop by to ask questions or just to say hi. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/3f3otyyuTte3FU9y3QzsLg?both Join the meeting via Zoom: https://us06web.zoom.us/j/82563808729?pwd=ZFU3Z2dMcXBGb05YemRsaGE1OW5nQT09 -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

2 5

Merging very limited weights support for quantiles/percentiles
by Sebastian Berg 27 Oct '23

27 Oct '23

Hi all, there is a PR to merge very limited support for weights in quantiles, which given no further input I will probably merge based on sklearn devs saying that they will use it. This means, adding a `weights` kwarg [1]. See: https://github.com/numpy/numpy/pull/24254 Limited here means that it would only work for the "inverted_cdf" method (which is not the default one). Why is it very limited? Because this limited version is the only form we/I am pretty confident about getting it right. There are various problems with making it more broad: 1. Weights are not clearly defined and can have many meanings, e.g.: * frequency weights (repeated observations) * probability weights (removing sample biases) * "analytic"/"precision" weights (encoding observation precision/variance). 2. There is very little to no literature on how to deal with the subtleties of dealing with (in the context of the various types of weights: * Interpolation (relevant to all interpolating methods) * Unbiasing (the main difference between the methods) The PR adds the most minimal thing, where weights are largly equivalent (no unbiasing issues, no interpolation). [2] Due to these complexities (and the lack of many statistic specialists looking at it) there is a point to be made that we just shouldn't add this in NumPy, but if nobody else has an opinion, I will go with the sklearn devs who want it :). (Also with weights we have to rely on full sorting for now, which can be slow, which I can live with personally.) - Sebastian [1] There are different styles of weights and for some method that clearly matters. Thus, if we ever expand the definition, it may be that `weights` has to be mapped to one of these, or that the the generic `weights` kwarg would raise an error for these that you need to pick a specific one like `pweights=`, or `fweights=`. [2] I am not quite sure about "analytic weights" here, but to me these do not really make sense in the context of a discrete interpolation method.

1 0

next NumPy community meeting
by Inessa Pawson 22 Oct '23

22 Oct '23

The next NumPy community meeting will be held this Wednesday, August 17th at 18:00 (6 pm) UTC. Join us via Zoom: https://us06web.zoom.us/j/83278611437?pwd=ekhoLzlHRjdWc0NOY2FQM0NPemdkZz09 Everyone is welcome and encouraged to attend. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both For the notes from the previous meeting, visit: https://github.com/numpy/archive/blob/main/community_meetings/community-202… Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ Twitter: @inessapawson

1 30

Seeking feedback: design doc for `namedarray`, a lightweight array data structure with named dimensions
by Anderson Banihirwe 20 Oct '23

20 Oct '23

:wave:t5: folks, [there has been growing interest in a lightweight array structure](https://github.com/pydata/xarray/issues/3981) that's in the same vein as [xarray's Variable](https://docs.xarray.dev/en/stable/generated/xarray.Variable.html). we've put together a design doc for `namedarray`, and we could use your feedback/input. ## what is `namedarray`? in essence, `namedarray` aims to be a lighter version of xarray's Variable—shedding some of the heavier dependencies (e.g. Pandas) but still retaining the goodness of named dimensions. ## what makes it special? * **Array Protocol Compatibility**: we are planning to make it compatible with existing array protocols and the new [Python array API standard](https://data-apis.org/array-api/latest/). * **Duck-Array Objects**: designed to wrap around multiple duck-array objects, like NumPy, Dask, Sparse, Pint, CuPy, and PyTorch. ## why are we doing this? the goal is to bridge the gap between power and simplicity, providing a lightweight alternative for scientific computing tasks that don't require the full firepower of Xarray (`DataArray` and `Dataset`). ## share your thoughts We've put together a design doc that goes into the nitty-gritty of `namedarray`. your insights could be invaluable in making this initiative a success. please give it a read and share your thoughts [here](https://github.com/pydata/xarray/discussions/8080) * **Design Doc**: [namedarray Design Document](https://github.com/pydata/xarray/blob/main/design_notes/named_arr… cross posting from [Scientifc Python Discourse](https://discuss.scientific-python.org/t/seeking-feedback-design-…

2 1

Preserving subclass on concatenation
by Dom Grigonis 15 Oct '23

15 Oct '23

Hi everyone, Quick question. Is there a way to concatenate arrays while preserving subclasses or is it left for custom implementation due to ambiguities of this operation? Regards, DG

1 0

NumPy 1.26.1 released
by Charles R Harris 14 Oct '23

14 Oct '23

Hi All, On behalf of the NumPy team, I'm pleased to announce the release of NumPy 1.26.1. NumPy 1.26.1 is a maintenance release that fixes bugs and regressions discovered after the 1.26.0 release. In addition, it adds new functionality for detecting BLAS and LAPACK when building from source. Highlights are: - Improved detection of BLAS and LAPACK libraries for meson builds - Pickle compatibility with the upcoming NumPy 2.0. The Python versions supported by this release are 3.9-3.12. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.26.1/>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.26.1>. The release notes have documentation of the new meson functionality. *Contributors* A total of 13 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Andrew Nelson - Anton Prosekin + - Charles Harris - Chongyun Lee + - Ivan A. Melnikov + - Jake Lishman + - Mahder Gebremedhin + - Mateusz Sokół - Matti Picus - Munira Alduraibi + - Ralf Gommers - Rohit Goswami - Sayed Adel *Pull requests merged* A total of 20 pull requests were merged for this release. - #24742: MAINT: Update cibuildwheel version - #24748: MAINT: fix version string in wheels built with setup.py - #24771: BLD, BUG: Fix build failure for host flags e.g. ``-march=native``... - #24773: DOC: Updated the f2py docs to remove a note on -fimplicit-none - #24776: BUG: Fix SIMD f32 trunc test on s390x when baseline is none - #24785: BLD: add libquadmath to licences and other tweaks (#24753) - #24786: MAINT: Activate ``use-compute-credits`` for Cirrus. - #24803: BLD: updated vendored-meson/meson for mips64 fix - #24804: MAINT: fix licence path win - #24813: BUG: Fix order of Windows OS detection macros. - #24831: BUG, SIMD: use scalar cmul on bad Apple clang x86_64 (#24828) - #24840: BUG: Fix DATA statements for f2py - #24870: API: Add ``NumpyUnpickler`` for backporting - #24872: MAINT: Xfail test failing on PyPy. - #24879: BLD: fix math func feature checks, fix FreeBSD build, add CI... - #24899: ENH: meson: implement BLAS/LAPACK auto-detection and many CI... - #24902: DOC: add a 1.26.1 release notes section for BLAS/LAPACK build... - #24906: MAINT: Backport ``numpy._core`` stubs. Remove ``NumpyUnpickler`` - #24911: MAINT: Bump pypa/cibuildwheel from 2.16.1 to 2.16.2 - #24912: BUG: loongarch doesn't use REAL(10) Cheers, Charles Harris

1 0

Adding NumpyUnpickler to Numpy 1.26 and future Numpy 2.0
by Nathan 12 Oct '23

12 Oct '23

Hi all, As part of the ongoing work on NEP 52 we are getting close to merging the pull request that changes numpy.core to numpy._core. While working on this we realized that numpy pickle files include paths to np.core in the pickle data. If we do nothing, switching np.core to np._core will generate deprecation warnings when loading pickle files generated by Numpy 1.x in Numpy 2.x and Numpy 1.x will be unable to read Numpy 2.x pickle files. Eventually, when Numpy 2.x completely removes the stub np.core module, loading old pickle files will break. The fix we have come up with is to add a new public NumpyUnpickler class to both the main branch and the Numpy 1.26 maintenance branch. This allows loading pickle files that were generated by Numpy 1.x and 2.x in either version without any warnings or errors. Users who are loading old pickle files will need to update their code to use NumpyUnpickler or create new pickle files and users who generate pickles with numpy 2.x will need to use NumpyUnpickler to read the files in numpy 1.x. We are using NumpyUnpickler internally for loading files in the npy file format. Users with pickle data saved in npy files won't see warnings. Only users who are storing data in pickle files directly and who want pickle files written in one numpy version to load correctly in another numpy version will run into trouble. The I/O docs already explicitly discourage using pickles to share data files between people and organizations like this. An alternate approach which would require less work for users would be to leave a limited subset of functionality in `np.core` needed for loading pickle files undeprecated. We would prefer to avoid doing this both because it would leave behind a publicly visible `np.core` module in NumPy's public API and because we're not sure if we can come up with a complete set of imports that should be allowed without warning from `np.core` without missing some corner cases and users will see deprecation warnings when loading pickles anyway. See https://github.com/numpy/numpy/pull/24866, https://github.com/numpy/numpy/issues/24844, and the discussion in https://github.com/numpy/numpy/pull/24634 for more context. -Nathan

7 20