Mailman 3 August 2022 - NumPy-Discussion

next NumPy community meeting
by Inessa Pawson 22 Oct '23

22 Oct '23

The next NumPy community meeting will be held this Wednesday, August 17th at 18:00 (6 pm) UTC. Join us via Zoom: https://us06web.zoom.us/j/83278611437?pwd=ekhoLzlHRjdWc0NOY2FQM0NPemdkZz09 Everyone is welcome and encouraged to attend. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both For the notes from the previous meeting, visit: https://github.com/numpy/archive/blob/main/community_meetings/community-202… Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ Twitter: @inessapawson

1 30

reminder: put an upper bound on setuptools if you use numpy.distutils!
by Ralf Gommers 31 Aug '23

31 Aug '23

Hi all, After setuptools 65.0 was released a few days ago, all users of numpy.distutils had their builds broken. This is already fixed in setuptools 65.0.2 because the breakage was particularly bad. However, the next breakage may not be fixed anymore (and more breakages *are* expected). So this is a good time to remind you all that you should put an upper bound on the setuptools version you allow in the releases of your package - to the last version that is known to work with your package. Our official stance here is that setuptools versions >=60 are not supported - see the "deprecating numpy.distutils" thread: https://mail.python.org/archives/list/numpy-discussion@python.org/message/P…. Newer versions may work for you, depending on what features you use. They don't for NumPy and for SciPy; both projects pin to 59.X to avoid problems. For the recent issue with setuptools 65.0.0, see https://github.com/numpy/numpy/issues/22135. We have also made the warnings about this topic in our docs more explicit, see https://github.com/numpy/numpy/pull/22154. Cheers, Ralf

2 2

mixed mode arithmetic
by Neal Becker 11 Jul '23

11 Jul '23

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Documentation Team meeting - Monday June 8th
by Melissa Mendonça 04 Dec '22

04 Dec '22

Hi all! A reminder that on Monday, June 8, we have another documentation team meeting at 3PM UTC**. If you wish to join on Zoom, you need to use this link https://zoom.us/j/420005230 Here's the permanent hackmd document with the meeting notes: https://hackmd.io/oB_boakvRqKR-_2jRV-Qjg <https://www.google.com/url?q=https%3A%2F%2Fhackmd.io%2FoB_boakvRqKR-_2jRV-Q…> Hope to see you around (especially if you want to introduce yourself or discuss ideas for Google Season of Docs). ** You can click this link to get the correct time at your timezone: https://www.timeanddate.com/worldclock/fixedtime.html?msg=NumPy+Documentati… - Melissa

6 74

next NumPy triage meeting
by Inessa Pawson 27 Nov '22

27 Nov '22

The next NumPy triage meeting will be held this Wednesday, August 10th at 16:00 (4 pm) UTC. This is a meeting where we synchronously triage prioritized PRs and issues. Join us via Zoom: https://us06web.zoom.us/j/82096749952?pwd=MW9oUmtKQ1c3a2gydGk1RTdYUUVXZz09 Everyone is welcome to attend and contribute to a conversation. Please notify us of issues or PRs that you’d like to have reviewed by adding a GitHub link to them in the meeting agenda: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg. Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy email: inessa(a)albuscode.org

1 8

next NumPy Newcomers' Hour
by Inessa Pawson 02 Oct '22

02 Oct '22

Our next Newcomers' Hour will be held this Thursday, June 16th, at 4 pm UTC. Stop by to ask questions or just to say hi. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/3f3otyyuTte3FU9y3QzsLg?both Join the meeting via Zoom: https://us02web.zoom.us/j/87192457898 Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ Twitter: @inessapawson

1 3

f2py compile error with full path of signature file on Windows
by Johann Rohwer 04 Sep '22

04 Sep '22

2 1

Need help extending a NumPy array in C
by lpsmith＠uw.edu 03 Sep '22

03 Sep '22

Hello! I am maintaining a C++ codebase with extensive ties to Python bindings (via SWIG). One of the features of the code is that we define (in C) a subclass of a NumPy Array. Everything worked until we started getting this message with numpy 1.23: RuntimeError: Object of type <class 'NamedArray'> appears to be C subclassed NumPy array, void scalar, or allocated in a non-standard way.NumPy reserves the right to change the size of these structures. Projects are required to take this into account by either recompiling against a specific NumPy version or padding the struct and enforcing a maximum NumPy version. My problem is that I don't know how do to either of those things. I would have assumed that whatever I compiled against, it would always be compiled against a specific NumPy version, and I also assumed that 'enforcing a maximum NumPy version' would happen in requirements.txt for the Python package, but that also seems to not be the case. Any hints? Thank you! -Lucian Smith

3 6

Exporting numpy arrays to binary JSON (BJData) for better portability
by Qianqian Fang 01 Sep '22

01 Sep '22

To avoid derailing the other thread <https://mail.python.org/archives/list/numpy-discussion@python.org/thread/A4…> on extending .npy files, I am going to start a new thread on alternative array storage file formats using binary JSON - in case there is such a need and interest among numpy users specifically, i want to first follow up with Bill's question below regarding loading time On 8/25/22 11:02, Bill Ross wrote: > > |Can you give load times for these?| > as I mentioned in the earlier reply to Robert, the most memory-efficient (i.e. fast loading, disk-mmap-able) but not necessarily disk-efficient (i.e. may result in the largest data file sizes) BJData construct to store an ND array using BJData's ND-array container. I have to admit that both jdata and bjdata modules have not been extensively optimized for speed. with the current implementation, here are the loading time for a larger diagonal matrix (eye(10000)) a BJData file storing a single eye(10000) array using the ND-array container can be downloaded from here <http://neurojson.org/wiki/upload/eye1e4_bjd_raw_ndsyntax.jdb.zip>(file size: 1MB with zip, if decompressed, it is ~800MB, as the npy file) - this file was generated from a matlab encoder, but can be loaded using Python (see below Re Robert). |800000128 eye1e4.npy|| ||800000014 eye1e4_bjd_raw_ndsyntax.jdb|| || 813721 eye1e4_bjd_zlib.jdb|| || 113067 eye1e4_bjd_lzma.jdb| the loading time (from an nvme drive, Ubuntu 18.04, python 3.6.9, numpy 1.19.5) for each file is listed below: |0.179s eye1e4.npy (mmap_mode=None)|| ||0.001s eye1e4.npy (mmap_mode=r)|| ||0.718s eye1e4_bjd_raw_ndsyntax.jdb|| ||1.474s eye1e4_bjd_zlib.jdb|| ||0.635s eye1e4_bjd_lzma.jdb| clearly, mmapped loading is the fastest option without a surprise; it is true that the raw bjdata file is about 5x slower than npy loading, but given the main chunk of the data are stored identically (as contiguous buffer), I suppose with some optimization of the decoder, the gap between the two can be substantially shortened. The longer loading time of zlib/lzma (and similarly saving times) reflects a trade-off between smaller file sizes and time for compression/decompression/disk-IO. by no means I am saying the binary JSON format is readily available to deliver better speed with its current non-optimized implementation. I just want to bright the attention to this class of formats, and highlight that it's flexibility gives abundant mechanisms to create fast, disk-mapped IO, while allowing additional benefits such as compression, unlimited metadata for future extensions etc. > |> 8000128 eye5chunk.npy|| > ||> 5004297 eye5chunk_bjd_raw.jdb|| > ||> 10338 eye5chunk_bjd_zlib.jdb|| > ||> 2206 eye5chunk_bjd_lzma.jdb| > > For my case, I'd be curious about the time to add one 1T-entries file > to another. as I mentioned in the previous reply, bjdata is appendable <https://github.com/NeuroJSON/bjdata/blob/master/images/BJData_Diagram.pdf>, so you can simply append another array (or a slice) to the end of the file. > Thanks, > Bill also related, Re @Robert's question below > Are any of them supported by a Python BJData implementation? I didn't > see any option to get that done in the `bjdata` package you > recommended, for example. > https://github.com/NeuroJSON/pybj/blob/a46355a0b0df0bec1817b04368a5a5733586… the bjdata module currently only support nd-array in the decoder <https://github.com/NeuroJSON/pybj/blob/a46355a0b0df0bec1817b04368a5a5733586…> (i.e. map such buffer to a numpy.ndarray) - should be relatively trivial to add it to the encoder though. on the other side, the annotated format is currently supported. one can call jdata module (responsible for annotation-level encoding/decoding) as shown in my sample code, then call bjdata internally for data serialization. > Okay. Given your wording, it looked like you were claiming that the > binary JSON was supported by the whole ecosystem. Rather, it seems > like you can either get binary encoding OR the ecosystem support, but > not both at the same time. all in relative terms of course - json has ~100 listed parsers on it's website <https://www.json.org/json-en.html>, MessagePack - another flavor of binary JSON - listed <https://msgpack.org/index.html> ~50/60 parsers, and UBJSON listed <https://ubjson.org/libraries/> ~20 parsers. I am not familiar with npy parsers, but googling it returns only a few. also, most binary JSON implementations provided tools to convert to JSON and back, so, in that sense, whatever JSON has in its ecosystem can be "potentially" used for binary JSON files if one wants to. There are also recent publications comparing differences between various binary JSON formats in case anyone is interested https://github.com/ubjson/universal-binary-json/issues/115

5 10

presentation topics for NumPy Newcomers’ Hour
by Inessa Pawson 29 Aug '22

29 Aug '22

Presentations by active NumPy contributors and core developers at NumPy Newcomers’ Hour have become very popular amongst our community members. To suggest a topic for future presentations, please leave a comment in this tracking issue: https://github.com/numpy/archive/issues/58. P.S. Visit the official NumPy YouTube channel to view video recordings of the previous events: https://www.youtube.com/c/NumPy_team. -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

1 0

ANN: SciPy 1.9.1
by Tyler Reddy 27 Aug '22

27 Aug '22

Hi all, On behalf of the SciPy development team, I'm pleased to announce the release of SciPy 1.9.1, which is a bug fix release that includes some important meson build fixes. Sources and binary wheels can be found at: https://pypi.org/project/scipy/ and at: https://github.com/scipy/scipy/releases/tag/v1.9.1 One of a few ways to install this release with pip: pip install scipy==1.9.1 ===================== SciPy 1.9.1 Release Notes ===================== SciPy 1.9.1 is a bug-fix release with no new features compared to 1.9.0. Notably, some important meson build fixes are included. Authors ====== * Anirudh Dagar (1) * Ralf Gommers (12) * Matt Haberland (2) * Andrew Nelson (1) * Tyler Reddy (14) * Atsushi Sakai (1) * Eli Schwartz (1) * Warren Weckesser (2) A total of 8 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 1.9.1 ------------------------------ * `#14517 <https://github.com/scipy/scipy/issues/14517>`__: scipy/linalg/tests/test_decomp.py::TestSchur::test_sort test... * `#16765 <https://github.com/scipy/scipy/issues/16765>`__: DOC: \`scipy.stats.skew\` no longer returns 0 on constant input * `#16787 <https://github.com/scipy/scipy/issues/16787>`__: BUG: Can't build 1.10 with mingw-w64 toolchain and numpy 1.21.6... * `#16813 <https://github.com/scipy/scipy/issues/16813>`__: BUG: scipy.interpolate interp1d extrapolate behaviour change... * `#16878 <https://github.com/scipy/scipy/issues/16878>`__: BUG: optimize.milp fails to execute when given exactly 3 constraints Pull requests for 1.9.1 ------------------------------ * `#16709 <https://github.com/scipy/scipy/pull/16709>`__: BLD: make the way we count commits for version numbering more... * `#16736 <https://github.com/scipy/scipy/pull/16736>`__: REL: prep for SciPy 1.9.1 * `#16749 <https://github.com/scipy/scipy/pull/16749>`__: BLD: install missing \`.pxd\` files, and update TODOs/FIXMEs... * `#16750 <https://github.com/scipy/scipy/pull/16750>`__: BLD: make OpenBLAS detection work with CMake * `#16755 <https://github.com/scipy/scipy/pull/16755>`__: TST: sparse.linalg: Loosen tolerance for the lobpcg test 'test_tolerance_float32' * `#16760 <https://github.com/scipy/scipy/pull/16760>`__: BLD: use a bit more idiomatic approach to constructing paths... * `#16768 <https://github.com/scipy/scipy/pull/16768>`__: DOC: stats.skew/kurtosis: returns NaN when input has only one... * `#16794 <https://github.com/scipy/scipy/pull/16794>`__: BLD/REL: on Windows use numpy 1.22.3 as the version to build... * `#16822 <https://github.com/scipy/scipy/pull/16822>`__: BUG/TST: linalg: Check the results of 'schur' more carefully. * `#16825 <https://github.com/scipy/scipy/pull/16825>`__: BUG: interpolate: fix "previous" and "next" extrapolate logic... * `#16862 <https://github.com/scipy/scipy/pull/16862>`__: BUG, DOC: Fix sphinx autosummary generation for \`odr\` and \`czt\` * `#16881 <https://github.com/scipy/scipy/pull/16881>`__: MAINT: optimize.milp: fix input validation when three constraints... * `#16901 <https://github.com/scipy/scipy/pull/16901>`__: MAINT: 1.9.1 backports * `#16904 <https://github.com/scipy/scipy/pull/16904>`__: BLD: update dependency ranges for meson-python and pybind11 for... Checksums ========= MD5 ~~~ 65d2b065c09d40b07272ae30332f6202 Changelog 69dd050c058c46e67d05956bf06e0566 README.txt 6108e7119fd6ae6d6338adede1904bb7 scipy-1.9.1-cp310-cp310-macosx_10_9_x86_64.whl 301053451a09b81268734133ce6d7bda scipy-1.9.1-cp310-cp310-macosx_12_0_arm64.whl 43a2fa03b951738d0deed6ddbc0a9f4d scipy-1.9.1-cp310-cp310-macosx_12_0_universal2.macosx_10_9_x86_64.whl 2339afc328d3c92cf85d1d86c5be4e19 scipy-1.9.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl d95584ddfe0703ba3b4624001a1f84f7 scipy-1.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 01937d372de92b3bcf9196b81bf8041d scipy-1.9.1-cp310-cp310-win_amd64.whl 3f85c3d508fced31f420ea31dccb1f83 scipy-1.9.1-cp38-cp38-macosx_10_9_x86_64.whl 3e07b61125901375f3ab99766465a428 scipy-1.9.1-cp38-cp38-macosx_12_0_arm64.whl 5cb2848a03030d3cec82a406b6d032de scipy-1.9.1-cp38-cp38-macosx_12_0_universal2.macosx_10_9_x86_64.whl cffdc85788b0d11e00f5c055e303538e scipy-1.9.1-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl ad0b411b353608b9fdd2fb3da25dd9ef scipy-1.9.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl fa1872fdcc63b57b95bffdcd2142a54e scipy-1.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 673905ce31524d16e5d44e600c48ff4d scipy-1.9.1-cp38-cp38-win32.whl bc421013525427036272cafa47c41f71 scipy-1.9.1-cp38-cp38-win_amd64.whl 19aa6604a3c39cbcc58f7a626e3fad89 scipy-1.9.1-cp39-cp39-macosx_10_9_x86_64.whl 0f6312acdf5edf5a6a5d2bf453e1c6cc scipy-1.9.1-cp39-cp39-macosx_12_0_arm64.whl 94ab8626a5d2aa5ed936f8571593567b scipy-1.9.1-cp39-cp39-macosx_12_0_universal2.macosx_10_9_x86_64.whl 4a9c22a4a059383cfb96cc2373156a30 scipy-1.9.1-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl 156f552987ff57cc347326b1df0a5916 scipy-1.9.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 8937a4cece2682585ceaf4115e526e21 scipy-1.9.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 9e1d2a5368f20c120da578a9f99b3986 scipy-1.9.1-cp39-cp39-win32.whl 3571be3d0e79cd46b0ab4c6418f6d44b scipy-1.9.1-cp39-cp39-win_amd64.whl SHA256 ~~~~~~ b6b1343d400fb775a7aaefe51654f56478a795318f11c29762a949a024cef9ca Changelog 1f18fbebfc36e5f6b3971b399fda124bd0c307308b407475d37059ba6a66be99 README.txt c61b4a91a702e8e04aeb0bfc40460e1f17a640977c04dda8757efb0199c75332 scipy-1.9.1-cp310-cp310-macosx_10_9_x86_64.whl d79da472015d0120ba9b357b28a99146cd6c17b9609403164b1a8ed149b4dfc8 scipy-1.9.1-cp310-cp310-macosx_12_0_arm64.whl 825951b88f56765aeb6e5e38ac9d7d47407cfaaeb008d40aa1b45a2d7ea2731e scipy-1.9.1-cp310-cp310-macosx_12_0_universal2.macosx_10_9_x86_64.whl f950a04b33e17b38ff561d5a0951caf3f5b47caa841edd772ffb7959f20a6af0 scipy-1.9.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 8cc81ac25659fec73599ccc52c989670e5ccd8974cf34bacd7b54a8d809aff1a scipy-1.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 8d3faa40ac16c6357aaf7ea50394ea6f1e8e99d75e927a51102b1943b311b4d9 scipy-1.9.1-cp310-cp310-win_amd64.whl 7a412c476a91b080e456229e413792bbb5d6202865dae963d1e6e28c2bb58691 scipy-1.9.1-cp38-cp38-macosx_10_9_x86_64.whl eb954f5aca4d26f468bbebcdc5448348eb287f7bea536c6306f62ea062f63d9a scipy-1.9.1-cp38-cp38-macosx_12_0_arm64.whl 3c6f5d1d4b9a5e4fe5e14f26ffc9444fc59473bbf8d45dc4a9a15283b7063a72 scipy-1.9.1-cp38-cp38-macosx_12_0_universal2.macosx_10_9_x86_64.whl bc4e2c77d4cd015d739e75e74ebbafed59ba8497a7ed0fd400231ed7683497c4 scipy-1.9.1-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl 0419485dbcd0ed78c0d5bf234c5dd63e86065b39b4d669e45810d42199d49521 scipy-1.9.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 34441dfbee5b002f9e15285014fd56e5e3372493c3e64ae297bae2c4b9659f5a scipy-1.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl b97b479f39c7e4aaf807efd0424dec74bbb379108f7d22cf09323086afcd312c scipy-1.9.1-cp38-cp38-win32.whl e8fe305d9d67a81255e06203454729405706907dccbdfcc330b7b3482a6c371d scipy-1.9.1-cp38-cp38-win_amd64.whl 39ab9240cd215a9349c85ab908dda6d732f7d3b4b192fa05780812495536acc4 scipy-1.9.1-cp39-cp39-macosx_10_9_x86_64.whl 71487c503e036740635f18324f62a11f283a632ace9d35933b2b0a04fd898c98 scipy-1.9.1-cp39-cp39-macosx_12_0_arm64.whl 3bc1ab68b9a096f368ba06c3a5e1d1d50957a86665fc929c4332d21355e7e8f4 scipy-1.9.1-cp39-cp39-macosx_12_0_universal2.macosx_10_9_x86_64.whl f7c39f7dbb57cce00c108d06d731f3b0e2a4d3a95c66d96bce697684876ce4d4 scipy-1.9.1-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl 47d1a95bd9d37302afcfe1b84c8011377c4f81e33649c5a5785db9ab827a6ade scipy-1.9.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 96d7cf7b25c9f23c59a766385f6370dab0659741699ecc7a451f9b94604938ce scipy-1.9.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 09412eb7fb60b8f00b328037fd814d25d261066ebc43a1e339cdce4f7502877e scipy-1.9.1-cp39-cp39-win32.whl 90c805f30c46cf60f1e76e947574f02954d25e3bb1e97aa8a07bc53aa31cf7d1 scipy-1.9.1-cp39-cp39-win_amd64.whl

1 0

Grace Hopper Celebration Open Source Day 2022 - Friday, September 16th - call for mentors
by Inessa Pawson 26 Aug '22

26 Aug '22

Hi, everyone! NumPy was selected as a featured project for Grace Hopper Celebration Open Source Day 2022 ( https://ghc.anitab.org/programs-and-awards/open-source-day/) held on Friday, September 16th, 8:00 am - 3:00 pm Pacific Time. I’m currently looking for mentors who would be helping NumPy newcomers during this event. If you are available, please sign up here: https://docs.google.com/spreadsheets/d/1Y1t7-Yu2INR-Unl277TCJmlKOSNWok8fX2N… . You may choose to help for 2 hours only or stay longer. -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

1 0

An extension of the .npy file format
by Michael Siebert 25 Aug '22

25 Aug '22

Dear all, originally, I have planned to make an extension of the .npy file format a dedicated follow-up pull request, but I have upgraded my current request instead, since it was not as difficult to implement as I initially thought and probably a more straight-forward solution: https://github.com/numpy/numpy/pull/20321/ What is this pull request about? It is about appending to Numpy .npy files. Why? I see two main use cases: 1. creating .npy files larger than the main memory. They can, once finished, be loaded as memory maps 2. creating binary log files, which can be processed very efficiently without parsing Are there not other good file formats to do this? Theoretically yes, but practically they can be pretty complex and with very little tweaking .npy could do efficient appending too. Use case 1 is already covered by the Pip/Conda package npy-append-array I have created and getting the functionality directly into Numpy was the original goal of the pull request. This would have been possible without introducing a new file format version, just by adding some spare space in the header. During the pull request discussion it turned out that rewriting the header after each append would be desirable in case the writing program crashes to minimize data loss. Use case 2 however would highly profit from a new file format version as it would make rewriting the header unnecessary: since efficient appending can only take place along one axis, setting shape[-1] = -1 in case of Fortran order or shape[0] = -1 otherwise (default) in the .npy header on file creation could indicate that the array size is determined by the file size: when np.load (typically with memory mapping on) gets called, it constructs the ndarray with the actual shape by replacing the -1 in the constructor call. Otherwise, the header is not modified anymore, neither on append nor on file write finish. Concurrent appends to a single file would not be advisable and should be channeled through a single AppendArray instance. Concurrent reads while writes take place however should work relatively smooth: every time an np.load (ideally with mmap) is called, the ndarray would provide access to all data written until that time. Currently, my pull request provides: 1. A definition of .npy version 4.0 that supports -1 in the shape 2. implementations for fortran order and non-fortran order (default) including test cases 3. Updated np.load 4. The AppendArray class that does the actual appending Although there is a certain hassle with introducing a new .npy version, the changes themselves are very small. I could also implement a fallback mode for older Numpy installations, if someone is interested. What do you think about such a feature, would it make sense? Anyone available for some more code review? Best from Berlin, Michael PS thank you so far, I could improve my npy-append-array module as well and from what I have seen so far the Numpy code readability exceeded my already high expectations.

5 7

writing a known-size 1D ndarray serially as it's calced
by bross_phobrain＠sonic.net 25 Aug '22

25 Aug '22

I want to calc multiple ndarrays at once and lack memory, so want to write in chunks (here sized to GPU batch capacity). It seems there should be an interface to write the header, then write a number of elements cyclically, then add any closing rubric and close the file. Is it as simple as lib.format.write_array_header_2_0(fp, d) then writing multiple shape(N,) arrays of float by fp.write(item.tobytes())?

4 4

help wanted to review Portuguese translation
by Inessa Pawson 25 Aug '22

25 Aug '22

Angélica Cardozo took initiative to translate into Portuguese the subtitles for the video “Find your way in the NumPy codebase” ( https://www.youtube.com/watch?v=mTWpBf1zewc) posted on the official NumPy YouTube channel (https://www.youtube.com/c/NumPy_team).🎉 We are currently looking for a contributor who could review the translation. If you are available, please respond to this message in a thread. -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

2 2

Support for Multiple Interpreters (Subinterpreters) in numpy
by Eric Snow 24 Aug '22

24 Aug '22

Hi all, CPython has supported multiple interpreters (in the same process) for a long time, but only through the C-API. I'm working on exposing that functionality to Python code (see PEP 554), aiming for 3.12. I expect that users will find the feature useful (particularly with a per-interpreter GIL--see PEP 684) and that it will be used a lot more over the coming years. This has the potential to impact extension module projects, especially large ones like numpy, which is why I'm reaching out to you. Use of multiple interpreters depends on isolation between them. When an extension module is imported in multiple interpreters, it is loaded separately into a new module object in each. Extensions often store module data/state in C globals, which means the the multiple instances end up sharing data. This causes problems, more so once we have one GIL per interpreter. Over the years we have added machinery to help extensions get the necessary isolation, moving away from global variables. This includes PEPs 384, 3121, and 489. This has culminated in the guide you can find in PEP 630. Note that nothing should change when only a single interpreter is in use (basically the status quo). With PEP 684, importing an incompatible extension outside the main (initial) interpreter will now be an ImportError. (Currently the behavior is undefined and too often results in hard-to-debug failures and crashes.) Thus extension module maintainers do have the option to *not* support multiple interpreters. Unfortunately, that doesn't mean their users won't pester them about adding support. We all recognize how that dynamic can be draining on a project. The potential burden on maintainers is a serious factor for these upcoming changes. numpy is likely to be affected more than any other project. That's why I'm starting this thread. PEP 684 discusses all of the above. What I'm after with this thread is: * to make sure the numpy maintainers are clear on what interpreter isolation requires of the project * a clear picture of what changes numpy would need (and how much work that would be) * feedback on what the CPython team can do to minimize that work (incl. adding new C APIs) I'm fine with having the discussion here, but I will probably create a new category on discuss.python.org for a variety of similar threads related to multiple interpreters and supporting them. Having our discussion there may lead to more participation from more CPython core devs than just me. Do you have any preference for or against any particular venue? Thanks! -eric

4 10

ENH: ndarray.__format__ implementation for numeric dtypes
by Inessa Pawson 23 Aug '22

23 Aug '22

We are looking for more feedback on this PR: https://github.com/numpy/numpy/pull/19550 If you would like to contribute to the discussion, please leave a comment in the Conversation section of the PR. -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

1 0

Display runtime information through a new function: `show_runtime`
by Ganesh Kathiresan 21 Aug '22

21 Aug '22

Hi Team, We are planning on adding a new function called `np.show_runtime` to display the runtime information of the machine in addition to `np.show_config` which displays the build-related information. Relevant PR: ENH: Use threadpoolctl in show_runtime (a new function) #21468 <https://github.com/numpy/numpy/pull/21468> Reasons for new function: 1. Will be helpful in root-causing issuing with information from the acceleration frameworks. 2. Build time information alone is not always helpful [1] Sample Output of new function >>> np.show_runtime() [{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'], 'found': ['SSSE3', 'SSE41', 'POPCNT', 'SSE42', 'AVX', 'F16C', 'FMA3', 'AVX2'], 'not_found': ['AVX512F', 'AVX512CD', 'AVX512_KNL', 'AVX512_KNM', 'AVX512_SKX', 'AVX512_CLX', 'AVX512_CNL', 'AVX512_ICL']}}, {'architecture': 'Zen', 'filepath': '/usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so', 'internal_api': 'openblas', 'num_threads': 12, 'prefix': 'libopenblas', 'threading_layer': 'pthreads', 'user_api': 'blas', 'version': '0.3.20'}] Under the hood `np.show_runtime` will use `threadpoolctl` [2] to get the runtime information of the frameworks. In addition to this, we will also display useful SIMD extensions that are supported. Future Scope 1. When we move towards meson, we can leverage the additional information and revamp `np.show_config` to utilize it. 2. If needed, we can gradually merge `show_runtime` and `show_config` to display information in sections and various other formats. Please do let us know if you have any suggestions on improving this function or concerns about the information provided by the new function. Thanks, Ganesh [1] show_config displays useless info when installed through conda-forge channel <https://github.com/numpy/numpy/issues/14492> (#14492) [2] threadpoolctl <https://pypi.org/project/threadpoolctl/>

4 5

NumPy 1.23.2 release
by Charles R Harris 14 Aug '22

14 Aug '22

Hi All, On behalf of the NumPy team, I am pleased to announce the release of NumPy 1.23.2. NumPy 1.23.2 is a maintenance release that fixes bugs discovered after the 1.23.1 release. Notable features are: - Typing changes needed for Python 3.11 - Wheels for Python 3.11.0rc1 The Python versions supported for this release are 3.8-3.11. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.23.2>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.23.2>. *Contributors* A total of 9 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Alexander Grund + - Bas van Beek - Charles Harris - Jon Cusick + - Matti Picus - Michael Osthege + - Pal Barta + - Ross Barnowski - Sebastian Berg *Pull requests merged* A total of 15 pull requests were merged for this release. - #22030: ENH: Add ``__array_ufunc__`` typing support to the ``nin=1`` ufuncs - #22031: MAINT, TYP: Fix ``np.angle`` dtype-overloads - #22032: MAINT: Do not let ``_GenericAlias`` wrap the underlying classes'... - #22033: TYP,MAINT: Allow ``einsum`` subscripts to be passed via integer... - #22034: MAINT,TYP: Add object-overloads for the ``np.generic`` rich comparisons - #22035: MAINT,TYP: Allow the ``squeeze`` and ``transpose`` method to... - #22036: BUG: Fix subarray to object cast ownership details - #22037: BUG: Use ``Popen`` to silently invoke f77 -v - #22038: BUG: Avoid errors on NULL during deepcopy - #22039: DOC: Add versionchanged for converter callable behavior. - #22057: MAINT: Quiet the anaconda uploads. - #22078: ENH: reorder includes for testing on top of system installations... - #22106: TST: fix test_linear_interpolation_formula_symmetric - #22107: BUG: Fix skip condition for test_loss_of_precision[complex256] - #22115: BLD: Build python3.11.0rc1 wheels. Cheers, Charles Harris

1 0

should random.standard_t allow df=inf?
by Sam Mason 12 Aug '22

12 Aug '22

Hi, I've just stumbled across a somewhat strange behaviour when I evaluated: np.random.standard_t(np.inf) and got NaN back. The limit of df going to infinity in Student's t-distribution is a Gaussian, so I'd expect a Gaussian variate back (but I suppose a warning or error could be argued to be valid). Note that R explicitly supports this[1] and results in Gaussian behaviour[2]. It looks relatively easy to modify random_standard_t (in distributions.c) to get similar behaviour, but wanted to check if this behaviour was deliberate before opening an issue. In fact it looks like the precision of doubles limit df > ~1e35 to be Gaussian[3], which might compile down to something smaller than a call to isfinite() as well as catching more cases[4]. Thanks! Sam [1] https://search.r-project.org/R/refmans/stats/html/TDist.html [2] R source, src/nmath/rt.c, src/nmath/dt.c, etc. [3] np.random.gamma(1e35, size=100) - 1e35 [4] https://godbolt.org/z/KcvY8sKd3

1 0

I need help on the issue of replacing the original C math library with an external C math library
by 454626566＠qq.com 12 Aug '22

12 Aug '22

Hello developers, The external math library contains some dynamic link libraries of C language (.so files) and some C header files (.h files), I am tring to link my custom library to NumPy and replace the original bottom math library. I need to add several compile options to the C compiler (gcc etc.), but I have not found the corrosponding part of NumPy source code. Merely modifying 'site.cfg' does not work at all, and then adding lib and include directory to '~/numpy/distutils/system_info.py' still does not work. When compiling NumPy source code, it gives a fatal error: myBLAS.h: No such file or directory. My doubts: Which code file should I focus on? How to modify configurations and add compile options like '-I /path/to/includes -L /path/to/libs -lmyBLAS'? Could you please give me a hint? Looking forward to your reply. Thanks!

2 1

Request to review a PR
by Ivan Tsybulin 09 Aug '22

09 Aug '22

Hello, I kindly ask someone to join as a reviewer to my PR https://github.com/numpy/numpy/pull/21814 about handling np.float32 arrays in ma.mean and ma.std functions. The PR is aimed to fix different behaviour between np.core and ma variants of these functions. -- Ivan Tsybulin

1 0

Python 3.11.0rc1 is out
by Charles R Harris 08 Aug '22

08 Aug '22

Hi All, Just to note that Python 3.11.0rc1 has been released. I plan to make a NumPy 1.23.2 release with support for that as soon as it is available on our build platforms. Chuck

1 0

Additional aliases for MaskedArray
by Matti Picus 08 Aug '22

08 Aug '22

PR 21998 [0] adds methods "ones_like", "zeros_like", "dsplit","split", and "vsplit" for MaskedArrays. The motivation was to clean up broken documentation links, and it seemed logical to add the interfaces. As far as I can tell we do not have users asking for these methods. Should we add them (expanding the np.ma namespace) or find another way to fix the broken documentation links? Matti [0] https://github.com/numpy/numpy/pull/21998

2 1

aarch64 wheels on github actions
by Andrew Nelson 07 Aug '22

07 Aug '22

At the moment Linux aarch64 wheels are being made in a TravisCI job. I've been experimenting recently about making scipy wheels for that target on Github Actions. cibuildwheel + github actions does quite well for that purpose. It takes around 2 hours for scipy wheels to be created (but not tested). I would expect numpy wheels to take less time to build. I think it would be a relatively minor modification of the .github/workflows/wheels script. cheers, Andrew. -- _____________________________________ Dr. Andrew Nelson _____________________________________

2 1

generate a random sample based on independent priors in `random.choice` #22082
by Rodo-Singh 04 Aug '22

04 Aug '22

Proposed new feature or change: Objective: Sample elements from the given iterator (a list, a numpy array, etc.) based upon pre-defined probabilities associated with each element which may not sums upto 1. Overview * In the numpy.random.choice function the cases where we're explicitly providing the list of probabilistic values for an input sample, the requirement expects the sum of the whole list to be 1. This makes sense when all the elements are possible observation for a single random variable whose pmf(probability mass function) is nothing but the p list. * But when every element in that list can be regarded as observation of separate independent Bernoulli r.v. (random variable), then the problem falls into the category of multi-label. Intuitively speaking, for each element in the list or array given to us, we'll toss a coin (may be biased or unbiased) ~B(p) (i.e., follows Bernoulli with p as success probability for head). * The output array or list would probably be a proper subset or can be a full list or can be an empty one. * Plus, here the argument size should automatically get shut down as we just want which all elements got selected into the output list after n coin tossess (where len(list) = n). Also it may happen that each element is independent as argued above, but the sum(p) = 1. Then we can probably put an extra argument independence = True or independence = False.

3 2

Speedup NumPy conjugate,square and CPU cache
by tim-berberich＠arcor.de 04 Aug '22

04 Aug '22

Hi all, I hope this is the right place to ask my questions. Recently I did some simple benchmarks on the numpy.conjugate routine for an 128 MB array, lets call it 'A'. It turned out that on my test machines there is a speed up of around 2.5 times if instead of simply calling A.conj() one loops over sub-matrices that fit into L1 cache of my CPU, like: for i in range(0,A.shape[0],size): A[i:i+size].conj() such that each A[i:i+size] fits into L1 cache. I posted example code and some graphs over at stackoverflow (https://stackoverflow.com/questions/73209565/strange-behaviour-during-multi…) I quickly checked and found a similar behavior for numpy.square. Now for my questions. 1. Is this a known/expected behavior of NumPy ? 2. Would it be possible/sensible to make simple numerical operations like numpy.conjugate & numpy.square cache aware ? Best, Tim

1 0