Mailman 3 February 2024 - NumPy-Discussion

next NumPy community meeting
by Inessa Pawson 21 Apr '24

21 Apr '24

The next NumPy community meeting will be held this Wednesday, November 8th at 5pm UTC. Join us via Zoom: https://numfocus-org.zoom.us/s/83278611437?pwd=ekhoLzlHRjdWc0NOY2FQM0NPemdk… . Everyone is welcome and encouraged to attend. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both. For the notes from the previous meetings, visit: https://github.com/numpy/archive/tree/main/community_meetings. -- Cheers, Inessa Inessa Pawson GitHub: inessapawson

1 12

JSON format for multi-dimensional data
by philippe＠loco-labs.io 17 Apr '24

17 Apr '24

Hi community, This memo is a proposal to implement a compact and reversible (lossless round-trip) JSON interface for multi-dimensional data and in particular for Numpy (see issue #12481). The links to the documents are at the end of the memo. The JSON-NTV (Named and Typed value) format is a JSON format which integrates a notion of type. This format has also been implemented for tabular data (see NTV-pandas package available in the pandas ecosystem and the PDEP12 specification). . The use of this format has the following advantages: - Taking into account data types not known to Numpy, - Reversible format (lossless round-trip) - Interoperability with other tools for tabular or multi-dimensional data (e.g. pandas, Xarray) - Ease of sharing Json format - Binary coding possible (e.g. CBOR format) - Format integrating data of different nature The associated Jupyter Notebook presents some key points of this proposal (first draft): Summary: - introduction - benefits - multi-dimensionnal data - Multi-dimensional types - Format JSON - Using the NTV format - Equivalence of tabular format and multidimensional format - Astropy specific points - Units and quantities - Coordinates - Tables - Other structures This subject seems important to me (in particular for interoperability issues) and I would like to have your feedback before working on the implementation. Especially, - do you think this “semantic” format is interesting to use? - do you have any particular expectations or subjects that I need to study beforehand? - do you have any examples or test cases to offer me? And of course, any type of remark and comment is welcome. Thanks in advance ! links: - Jupyter notebook : https://nbviewer.org/github/loco-philippe/Environmental-Sensing/blob/main/p… - JSON-NTV format : https://www.ietf.org/archive/id/draft-thomy-json-ntv-02.html - JSON-NTV overview : https://nbviewer.org/github/loco-philippe/NTV/blob/main/example/example_ntv… - NTV tabular format : https://www.ietf.org/archive/id/draft-thomy-ntv-tab-00.html#name-tabular-st… - NTV-pandas package : https://github.com/loco-philippe/ntv-pandas/blob/main/README.md - NTV-pandas examples : https://nbviewer.org/github/loco-philippe/ntv-pandas/blob/main/example/exam… - Pandas specification - PDEP12 : https://pandas.pydata.org/pdeps/0012-compact-and-reversible-JSON-interface.…

5 12

next Documentation team meeting
by Mukulika Pahari 08 Apr '24

08 Apr '24

Hi all, Our next Documentation Team meeting will happen on *Monday, December 4* at *11PM UTC*. If this time slot is inconvenient for you to join, please let me know in the replies or Slack and we will try to add another time slot. All are welcome - you don't need to already be a contributor to join. If you have questions or are curious about what we're doing, we'll be happy to meet you! If you wish to join on Zoom, use this (updated) link: https://numfocus-org.zoom.us/j/85016474448?pwd=TWEvaWJ1SklyVEpwNXUrcHV1YmFJQ... Here's the permanent hackmd document with the meeting notes (still being updated): https://hackmd.io/oB_boakvRqKR-_2jRV-Qjg Hope to see you around! Best wishes, Mukulika

2 4

Numpy wheels: who maintains the builds and who pays for it?
by sean.gillies＠gmail.com 04 Apr '24

04 Apr '24

Hi folks, My name is Sean and I'm the author of several GIS packages using Numpy: Fiona, Rasterio, and Shapely. I've followed Numpy's trail when it comes to wheel building for many years and now I'm seeking advice on how to prioritize platforms to support and how to pay for the labor and computing that it takes to build wheels and maintain the infrastructure over time. Fiona and Rasterio have an order of magnitude more C library dependencies than Numpy, via GDAL (https://gdal.org/), which is almost more of an OS than a library. I found a thread in the archive about adding musllinux wheels, but it wasn't clear to me how the work gets done, who does, and how it gets paid for. Does NumFOCUS support pay a maintainer to do it? Are Numpy maintainers adding new platform builds as part of their day jobs? Are they donating their own time to the effort? Does the Numpy project aspire to provide wheels for all of the top N platforms? Is it more than an aspiration? Is support for one or more platforms part of any sponsorship agreement? Maybe users and sponsorship cover the cost of building wheels and maintenance entirely for Numpy, but it's not so in my case. I've got so many questions in this vein, and I'm grateful for any answers or insights or more discussion. Thanks!

2 2

Add to NumPy a function to compute cumulative sums from 0.
by john.dawson＠camlingroup.com 20 Mar '24

20 Mar '24

`cumsum` computes the sum of the first k summands for every k from 1. Judging by my experience, it is more often useful to compute the sum of the first k summands for every k from 0, as `cumsum`'s behaviour leads to fencepost-like problems. https://en.wikipedia.org/wiki/Off-by-one_error#Fencepost_error For example, `cumsum` is not the inverse of `diff`. I propose adding a function to NumPy to compute cumulative sums beginning with 0, that is, an inverse of `diff`. It might be called `cumsum0`. The following code is probably not the best way to implement it, but it illustrates the desired behaviour. ``` def cumsum0(a, axis=None, dtype=None, out=None): """ Return the cumulative sum of the elements along a given axis, beginning with 0. cumsum0 does the same as cumsum except that cumsum computes the sum of the first k summands for every k from 1 and cumsum, from 0. Parameters ---------- a : array_like Input array. axis : int, optional Axis along which the cumulative sum is computed. The default (None) is to compute the cumulative sum over the flattened array. dtype : dtype, optional Type of the returned array and of the accumulator in which the elements are summed. If `dtype` is not specified, it defaults to the dtype of `a`, unless `a` has an integer dtype with a precision less than that of the default platform integer. In that case, the default platform integer is used. out : ndarray, optional Alternative output array in which to place the result. It must have the same shape and buffer length as the expected output but the type will be cast if necessary. See :ref:`ufuncs-output-type` for more details. Returns ------- cumsum0_along_axis : ndarray. A new array holding the result is returned unless `out` is specified, in which case a reference to `out` is returned. If `axis` is not None the result has the same shape as `a` except along `axis`, where the dimension is smaller by 1. See Also -------- cumsum : Cumulatively sum array elements, beginning with the first. sum : Sum array elements. trapz : Integration of array values using the composite trapezoidal rule. diff : Calculate the n-th discrete difference along given axis. Notes ----- Arithmetic is modular when using integer types, and no error is raised on overflow. ``cumsum0(a)[-1]`` may not be equal to ``sum(a)`` for floating-point values since ``sum`` may use a pairwise summation routine, reducing the roundoff-error. See `sum` for more information. Examples -------- >>> a = np.array([[1, 2, 3], [4, 5, 6]]) >>> a array([[1, 2, 3], [4, 5, 6]]) >>> np.cumsum0(a) array([ 0, 1, 3, 6, 10, 15, 21]) >>> np.cumsum0(a, dtype=float) # specifies type of output value(s) array([ 0., 1., 3., 6., 10., 15., 21.]) >>> np.cumsum0(a, axis=0) # sum over rows for each of the 3 columns array([[0, 0, 0], [1, 2, 3], [5, 7, 9]]) >>> np.cumsum0(a, axis=1) # sum over columns for each of the 2 rows array([[ 0, 1, 3, 6], [ 0, 4, 9, 15]]) ``cumsum(b)[-1]`` may not be equal to ``sum(b)`` >>> b = np.array([1, 2e-9, 3e-9] * 1000000) >>> np.cumsum0(b)[-1] 1000000.0050045159 >>> b.sum() 1000000.0050000029 """ empty = a.take([], axis=axis) zero = empty.sum(axis, dtype=dtype, keepdims=True) later_cumsum = a.cumsum(axis, dtype=dtype) return concatenate([zero, later_cumsum], axis=axis, dtype=dtype, out=out) ```

14 25

next NumPy triage meeting - now starting at 5 pm UTC
by Inessa Pawson 17 Mar '24

17 Mar '24

The next NumPy triage meeting will be held this Wednesday, January 11th at 5 pm UTC. (Please note the new time!) This is a meeting where we synchronously triage prioritized PRs and issues. Join us via Zoom: https://numfocus-org.zoom.us/j/82096749952?pwd=MW9oUmtKQ1c3a2gydGk1RTdYUUVX… Everyone is welcome to attend and contribute to a conversation. Please notify us of issues or PRs that you’d like to have reviewed by adding a GitHub link to them in the meeting agenda: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg. -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

1 30

Improved 2DFFT Approach
by camrymanjr＠me.com 15 Mar '24

15 Mar '24

Good day! My name is Alexander Levin. My colleague and I did a project on optimisation of two-dimensional Fourier transform algorithm six months ago. We took your implementation of numpy fft2d as a unit of quality. In the course of our research we found out that initially mathematically the method uses a far from optimal algorithm. As you may know, your operation goes first by rows then by columns applying a one-dimensional transformation. After spending some time researching mathematical papers on this topic, we found and implemented a new approach, the so-called Cooley-Tukey butterfly, optimised by Russian mathematicians in 2016. The output is a completely new approach, where we immediately apply a two-dimensional operation and save a serious proportion of time on it. As a result, we wrote a C++ package for Python, using Cython as a wrapper. The result was the package and an article on Medium describing the process. On tests for matrices ranging in size from 2048x512 to 8192x8192, our algorithm outperformed the NumPy transformation by an average of 50% in time. After discussing this matter with my colleague with whom we did the above development, we came to a common desire to share our results with you. Your company has been making a huge contribution to the IT community for almost 20 years on a pro bono basis. We share your philosophy and so we want to make the CS world a better place by providing you with our code to optimise the approach in an existing operation in your package. We would like to offer you our development version, though it'll need some minor improvements, we'd love to finish them collaboratively with you and hear your thoughts regarding what we have discovered and done so far. We'd be honored to help you and become NumPy contributors. I trust that our vision will resonate with you and that you will agree with us. I invite you to read our quick article about the process, which I have linked below, and our fully functioning package and share your views on our contribution to NumPy. We are willing to edit the code to fit your standards, as we believe that the best use of our work will be to contribute to the development of technology in the world. Thank you for your time. We look forward to your response and opinion. Medium Article about the process: https://medium.com/p/7963c3b2f3c9 GitHub Source code Repository: https://github.com/2D-FFT-Project/2d-fft

6 11

next NumPy Newcomers' Hour - 10pm UTC
by Inessa Pawson 04 Mar '24

04 Mar '24

Our next Newcomers' Hour will be held this Thursday, November 16th at 10pm UTC. Stop by to ask questions, share your progress, celebrate success, or just to say hi. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/3f3otyyuTte3FU9y3QzsLg?both. Join the meeting via Zoom: https://us06web.zoom.us/j/82563808729?pwd=ZFU3Z2dMcXBGb05YemRsaGE1OW5nQT09. -- Cheers, Inessa Inessa Pawson GitHub: inessapawson

3 8

next NumPy Optimization Team meeting
by Inessa Pawson 01 Mar '24

01 Mar '24

The next NumPy Optimization Team meeting will be held on Monday, November 20th at 4pm UTC. Join us via Zoom: https://numfocus-org.zoom.us/j/88162122074?pwd=a3h0cTZubzlLcTZzMVhCL1AxUVFM… . Everyone is welcome and encouraged to attend. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/dVdSlQ0TThWkOk0OkmGsmw?both. For the notes from the previous meetings, visit: https://github.com/numpy/archive/tree/main/optim_team_meetings. -- Cheers, Inessa Inessa Pawson GitHub: inessapawson

1 5

ENH: Add 'axis' kwarg to np.tile
by evan＠evanw.org 01 Mar '24

01 Mar '24

Hoping to get some more feedback on my recent PR [0] which has stagnated a bit for the past few weeks. This adds an `axis` keyword argument to np.tile which may be an int or tuple of ints, much like np.sum or np.roll. This is my first contribution to numpy :) Thanks, Evan [0]: https://github.com/numpy/numpy/pull/25703

3 2