Mailman 3 August 2022 - NumPy-Discussion

presentation topics for NumPy Newcomers’ Hour
by Inessa Pawson 29 Aug '22

29 Aug '22

Presentations by active NumPy contributors and core developers at NumPy Newcomers’ Hour have become very popular amongst our community members. To suggest a topic for future presentations, please leave a comment in this tracking issue: https://github.com/numpy/archive/issues/58. P.S. Visit the official NumPy YouTube channel to view video recordings of the previous events: https://www.youtube.com/c/NumPy_team. -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

1 0

ANN: SciPy 1.9.1
by Tyler Reddy 27 Aug '22

27 Aug '22

Hi all, On behalf of the SciPy development team, I'm pleased to announce the release of SciPy 1.9.1, which is a bug fix release that includes some important meson build fixes. Sources and binary wheels can be found at: https://pypi.org/project/scipy/ and at: https://github.com/scipy/scipy/releases/tag/v1.9.1 One of a few ways to install this release with pip: pip install scipy==1.9.1 ===================== SciPy 1.9.1 Release Notes ===================== SciPy 1.9.1 is a bug-fix release with no new features compared to 1.9.0. Notably, some important meson build fixes are included. Authors ====== * Anirudh Dagar (1) * Ralf Gommers (12) * Matt Haberland (2) * Andrew Nelson (1) * Tyler Reddy (14) * Atsushi Sakai (1) * Eli Schwartz (1) * Warren Weckesser (2) A total of 8 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 1.9.1 ------------------------------ * `#14517 <https://github.com/scipy/scipy/issues/14517>`__: scipy/linalg/tests/test_decomp.py::TestSchur::test_sort test... * `#16765 <https://github.com/scipy/scipy/issues/16765>`__: DOC: \`scipy.stats.skew\` no longer returns 0 on constant input * `#16787 <https://github.com/scipy/scipy/issues/16787>`__: BUG: Can't build 1.10 with mingw-w64 toolchain and numpy 1.21.6... * `#16813 <https://github.com/scipy/scipy/issues/16813>`__: BUG: scipy.interpolate interp1d extrapolate behaviour change... * `#16878 <https://github.com/scipy/scipy/issues/16878>`__: BUG: optimize.milp fails to execute when given exactly 3 constraints Pull requests for 1.9.1 ------------------------------ * `#16709 <https://github.com/scipy/scipy/pull/16709>`__: BLD: make the way we count commits for version numbering more... * `#16736 <https://github.com/scipy/scipy/pull/16736>`__: REL: prep for SciPy 1.9.1 * `#16749 <https://github.com/scipy/scipy/pull/16749>`__: BLD: install missing \`.pxd\` files, and update TODOs/FIXMEs... * `#16750 <https://github.com/scipy/scipy/pull/16750>`__: BLD: make OpenBLAS detection work with CMake * `#16755 <https://github.com/scipy/scipy/pull/16755>`__: TST: sparse.linalg: Loosen tolerance for the lobpcg test 'test_tolerance_float32' * `#16760 <https://github.com/scipy/scipy/pull/16760>`__: BLD: use a bit more idiomatic approach to constructing paths... * `#16768 <https://github.com/scipy/scipy/pull/16768>`__: DOC: stats.skew/kurtosis: returns NaN when input has only one... * `#16794 <https://github.com/scipy/scipy/pull/16794>`__: BLD/REL: on Windows use numpy 1.22.3 as the version to build... * `#16822 <https://github.com/scipy/scipy/pull/16822>`__: BUG/TST: linalg: Check the results of 'schur' more carefully. * `#16825 <https://github.com/scipy/scipy/pull/16825>`__: BUG: interpolate: fix "previous" and "next" extrapolate logic... * `#16862 <https://github.com/scipy/scipy/pull/16862>`__: BUG, DOC: Fix sphinx autosummary generation for \`odr\` and \`czt\` * `#16881 <https://github.com/scipy/scipy/pull/16881>`__: MAINT: optimize.milp: fix input validation when three constraints... * `#16901 <https://github.com/scipy/scipy/pull/16901>`__: MAINT: 1.9.1 backports * `#16904 <https://github.com/scipy/scipy/pull/16904>`__: BLD: update dependency ranges for meson-python and pybind11 for... Checksums ========= MD5 ~~~ 65d2b065c09d40b07272ae30332f6202 Changelog 69dd050c058c46e67d05956bf06e0566 README.txt 6108e7119fd6ae6d6338adede1904bb7 scipy-1.9.1-cp310-cp310-macosx_10_9_x86_64.whl 301053451a09b81268734133ce6d7bda scipy-1.9.1-cp310-cp310-macosx_12_0_arm64.whl 43a2fa03b951738d0deed6ddbc0a9f4d scipy-1.9.1-cp310-cp310-macosx_12_0_universal2.macosx_10_9_x86_64.whl 2339afc328d3c92cf85d1d86c5be4e19 scipy-1.9.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl d95584ddfe0703ba3b4624001a1f84f7 scipy-1.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 01937d372de92b3bcf9196b81bf8041d scipy-1.9.1-cp310-cp310-win_amd64.whl 3f85c3d508fced31f420ea31dccb1f83 scipy-1.9.1-cp38-cp38-macosx_10_9_x86_64.whl 3e07b61125901375f3ab99766465a428 scipy-1.9.1-cp38-cp38-macosx_12_0_arm64.whl 5cb2848a03030d3cec82a406b6d032de scipy-1.9.1-cp38-cp38-macosx_12_0_universal2.macosx_10_9_x86_64.whl cffdc85788b0d11e00f5c055e303538e scipy-1.9.1-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl ad0b411b353608b9fdd2fb3da25dd9ef scipy-1.9.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl fa1872fdcc63b57b95bffdcd2142a54e scipy-1.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 673905ce31524d16e5d44e600c48ff4d scipy-1.9.1-cp38-cp38-win32.whl bc421013525427036272cafa47c41f71 scipy-1.9.1-cp38-cp38-win_amd64.whl 19aa6604a3c39cbcc58f7a626e3fad89 scipy-1.9.1-cp39-cp39-macosx_10_9_x86_64.whl 0f6312acdf5edf5a6a5d2bf453e1c6cc scipy-1.9.1-cp39-cp39-macosx_12_0_arm64.whl 94ab8626a5d2aa5ed936f8571593567b scipy-1.9.1-cp39-cp39-macosx_12_0_universal2.macosx_10_9_x86_64.whl 4a9c22a4a059383cfb96cc2373156a30 scipy-1.9.1-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl 156f552987ff57cc347326b1df0a5916 scipy-1.9.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 8937a4cece2682585ceaf4115e526e21 scipy-1.9.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 9e1d2a5368f20c120da578a9f99b3986 scipy-1.9.1-cp39-cp39-win32.whl 3571be3d0e79cd46b0ab4c6418f6d44b scipy-1.9.1-cp39-cp39-win_amd64.whl SHA256 ~~~~~~ b6b1343d400fb775a7aaefe51654f56478a795318f11c29762a949a024cef9ca Changelog 1f18fbebfc36e5f6b3971b399fda124bd0c307308b407475d37059ba6a66be99 README.txt c61b4a91a702e8e04aeb0bfc40460e1f17a640977c04dda8757efb0199c75332 scipy-1.9.1-cp310-cp310-macosx_10_9_x86_64.whl d79da472015d0120ba9b357b28a99146cd6c17b9609403164b1a8ed149b4dfc8 scipy-1.9.1-cp310-cp310-macosx_12_0_arm64.whl 825951b88f56765aeb6e5e38ac9d7d47407cfaaeb008d40aa1b45a2d7ea2731e scipy-1.9.1-cp310-cp310-macosx_12_0_universal2.macosx_10_9_x86_64.whl f950a04b33e17b38ff561d5a0951caf3f5b47caa841edd772ffb7959f20a6af0 scipy-1.9.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 8cc81ac25659fec73599ccc52c989670e5ccd8974cf34bacd7b54a8d809aff1a scipy-1.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 8d3faa40ac16c6357aaf7ea50394ea6f1e8e99d75e927a51102b1943b311b4d9 scipy-1.9.1-cp310-cp310-win_amd64.whl 7a412c476a91b080e456229e413792bbb5d6202865dae963d1e6e28c2bb58691 scipy-1.9.1-cp38-cp38-macosx_10_9_x86_64.whl eb954f5aca4d26f468bbebcdc5448348eb287f7bea536c6306f62ea062f63d9a scipy-1.9.1-cp38-cp38-macosx_12_0_arm64.whl 3c6f5d1d4b9a5e4fe5e14f26ffc9444fc59473bbf8d45dc4a9a15283b7063a72 scipy-1.9.1-cp38-cp38-macosx_12_0_universal2.macosx_10_9_x86_64.whl bc4e2c77d4cd015d739e75e74ebbafed59ba8497a7ed0fd400231ed7683497c4 scipy-1.9.1-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl 0419485dbcd0ed78c0d5bf234c5dd63e86065b39b4d669e45810d42199d49521 scipy-1.9.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 34441dfbee5b002f9e15285014fd56e5e3372493c3e64ae297bae2c4b9659f5a scipy-1.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl b97b479f39c7e4aaf807efd0424dec74bbb379108f7d22cf09323086afcd312c scipy-1.9.1-cp38-cp38-win32.whl e8fe305d9d67a81255e06203454729405706907dccbdfcc330b7b3482a6c371d scipy-1.9.1-cp38-cp38-win_amd64.whl 39ab9240cd215a9349c85ab908dda6d732f7d3b4b192fa05780812495536acc4 scipy-1.9.1-cp39-cp39-macosx_10_9_x86_64.whl 71487c503e036740635f18324f62a11f283a632ace9d35933b2b0a04fd898c98 scipy-1.9.1-cp39-cp39-macosx_12_0_arm64.whl 3bc1ab68b9a096f368ba06c3a5e1d1d50957a86665fc929c4332d21355e7e8f4 scipy-1.9.1-cp39-cp39-macosx_12_0_universal2.macosx_10_9_x86_64.whl f7c39f7dbb57cce00c108d06d731f3b0e2a4d3a95c66d96bce697684876ce4d4 scipy-1.9.1-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl 47d1a95bd9d37302afcfe1b84c8011377c4f81e33649c5a5785db9ab827a6ade scipy-1.9.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 96d7cf7b25c9f23c59a766385f6370dab0659741699ecc7a451f9b94604938ce scipy-1.9.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 09412eb7fb60b8f00b328037fd814d25d261066ebc43a1e339cdce4f7502877e scipy-1.9.1-cp39-cp39-win32.whl 90c805f30c46cf60f1e76e947574f02954d25e3bb1e97aa8a07bc53aa31cf7d1 scipy-1.9.1-cp39-cp39-win_amd64.whl

1 0

Grace Hopper Celebration Open Source Day 2022 - Friday, September 16th - call for mentors
by Inessa Pawson 27 Aug '22

27 Aug '22

Hi, everyone! NumPy was selected as a featured project for Grace Hopper Celebration Open Source Day 2022 ( https://ghc.anitab.org/programs-and-awards/open-source-day/) held on Friday, September 16th, 8:00 am - 3:00 pm Pacific Time. I’m currently looking for mentors who would be helping NumPy newcomers during this event. If you are available, please sign up here: https://docs.google.com/spreadsheets/d/1Y1t7-Yu2INR-Unl277TCJmlKOSNWok8fX2N… . You may choose to help for 2 hours only or stay longer. -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

1 0

An extension of the .npy file format
by Michael Siebert 26 Aug '22

26 Aug '22

Dear all, originally, I have planned to make an extension of the .npy file format a dedicated follow-up pull request, but I have upgraded my current request instead, since it was not as difficult to implement as I initially thought and probably a more straight-forward solution: https://github.com/numpy/numpy/pull/20321/ What is this pull request about? It is about appending to Numpy .npy files. Why? I see two main use cases: 1. creating .npy files larger than the main memory. They can, once finished, be loaded as memory maps 2. creating binary log files, which can be processed very efficiently without parsing Are there not other good file formats to do this? Theoretically yes, but practically they can be pretty complex and with very little tweaking .npy could do efficient appending too. Use case 1 is already covered by the Pip/Conda package npy-append-array I have created and getting the functionality directly into Numpy was the original goal of the pull request. This would have been possible without introducing a new file format version, just by adding some spare space in the header. During the pull request discussion it turned out that rewriting the header after each append would be desirable in case the writing program crashes to minimize data loss. Use case 2 however would highly profit from a new file format version as it would make rewriting the header unnecessary: since efficient appending can only take place along one axis, setting shape[-1] = -1 in case of Fortran order or shape[0] = -1 otherwise (default) in the .npy header on file creation could indicate that the array size is determined by the file size: when np.load (typically with memory mapping on) gets called, it constructs the ndarray with the actual shape by replacing the -1 in the constructor call. Otherwise, the header is not modified anymore, neither on append nor on file write finish. Concurrent appends to a single file would not be advisable and should be channeled through a single AppendArray instance. Concurrent reads while writes take place however should work relatively smooth: every time an np.load (ideally with mmap) is called, the ndarray would provide access to all data written until that time. Currently, my pull request provides: 1. A definition of .npy version 4.0 that supports -1 in the shape 2. implementations for fortran order and non-fortran order (default) including test cases 3. Updated np.load 4. The AppendArray class that does the actual appending Although there is a certain hassle with introducing a new .npy version, the changes themselves are very small. I could also implement a fallback mode for older Numpy installations, if someone is interested. What do you think about such a feature, would it make sense? Anyone available for some more code review? Best from Berlin, Michael PS thank you so far, I could improve my npy-append-array module as well and from what I have seen so far the Numpy code readability exceeded my already high expectations.

5 7

writing a known-size 1D ndarray serially as it's calced
by bross_phobrain＠sonic.net 25 Aug '22

25 Aug '22

I want to calc multiple ndarrays at once and lack memory, so want to write in chunks (here sized to GPU batch capacity). It seems there should be an interface to write the header, then write a number of elements cyclically, then add any closing rubric and close the file. Is it as simple as lib.format.write_array_header_2_0(fp, d) then writing multiple shape(N,) arrays of float by fp.write(item.tobytes())?

4 4

help wanted to review Portuguese translation
by Inessa Pawson 25 Aug '22

25 Aug '22

Angélica Cardozo took initiative to translate into Portuguese the subtitles for the video “Find your way in the NumPy codebase” ( https://www.youtube.com/watch?v=mTWpBf1zewc) posted on the official NumPy YouTube channel (https://www.youtube.com/c/NumPy_team).🎉 We are currently looking for a contributor who could review the translation. If you are available, please respond to this message in a thread. -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

2 2

Support for Multiple Interpreters (Subinterpreters) in numpy
by Eric Snow 25 Aug '22

25 Aug '22

Hi all, CPython has supported multiple interpreters (in the same process) for a long time, but only through the C-API. I'm working on exposing that functionality to Python code (see PEP 554), aiming for 3.12. I expect that users will find the feature useful (particularly with a per-interpreter GIL--see PEP 684) and that it will be used a lot more over the coming years. This has the potential to impact extension module projects, especially large ones like numpy, which is why I'm reaching out to you. Use of multiple interpreters depends on isolation between them. When an extension module is imported in multiple interpreters, it is loaded separately into a new module object in each. Extensions often store module data/state in C globals, which means the the multiple instances end up sharing data. This causes problems, more so once we have one GIL per interpreter. Over the years we have added machinery to help extensions get the necessary isolation, moving away from global variables. This includes PEPs 384, 3121, and 489. This has culminated in the guide you can find in PEP 630. Note that nothing should change when only a single interpreter is in use (basically the status quo). With PEP 684, importing an incompatible extension outside the main (initial) interpreter will now be an ImportError. (Currently the behavior is undefined and too often results in hard-to-debug failures and crashes.) Thus extension module maintainers do have the option to *not* support multiple interpreters. Unfortunately, that doesn't mean their users won't pester them about adding support. We all recognize how that dynamic can be draining on a project. The potential burden on maintainers is a serious factor for these upcoming changes. numpy is likely to be affected more than any other project. That's why I'm starting this thread. PEP 684 discusses all of the above. What I'm after with this thread is: * to make sure the numpy maintainers are clear on what interpreter isolation requires of the project * a clear picture of what changes numpy would need (and how much work that would be) * feedback on what the CPython team can do to minimize that work (incl. adding new C APIs) I'm fine with having the discussion here, but I will probably create a new category on discuss.python.org for a variety of similar threads related to multiple interpreters and supporting them. Having our discussion there may lead to more participation from more CPython core devs than just me. Do you have any preference for or against any particular venue? Thanks! -eric

4 10

ENH: ndarray.__format__ implementation for numeric dtypes
by Inessa Pawson 24 Aug '22

24 Aug '22

We are looking for more feedback on this PR: https://github.com/numpy/numpy/pull/19550 If you would like to contribute to the discussion, please leave a comment in the Conversation section of the PR. -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

1 0

Display runtime information through a new function: `show_runtime`
by Ganesh Kathiresan 22 Aug '22

22 Aug '22

Hi Team, We are planning on adding a new function called `np.show_runtime` to display the runtime information of the machine in addition to `np.show_config` which displays the build-related information. Relevant PR: ENH: Use threadpoolctl in show_runtime (a new function) #21468 <https://github.com/numpy/numpy/pull/21468> Reasons for new function: 1. Will be helpful in root-causing issuing with information from the acceleration frameworks. 2. Build time information alone is not always helpful [1] Sample Output of new function >>> np.show_runtime() [{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'], 'found': ['SSSE3', 'SSE41', 'POPCNT', 'SSE42', 'AVX', 'F16C', 'FMA3', 'AVX2'], 'not_found': ['AVX512F', 'AVX512CD', 'AVX512_KNL', 'AVX512_KNM', 'AVX512_SKX', 'AVX512_CLX', 'AVX512_CNL', 'AVX512_ICL']}}, {'architecture': 'Zen', 'filepath': '/usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.20.so', 'internal_api': 'openblas', 'num_threads': 12, 'prefix': 'libopenblas', 'threading_layer': 'pthreads', 'user_api': 'blas', 'version': '0.3.20'}] Under the hood `np.show_runtime` will use `threadpoolctl` [2] to get the runtime information of the frameworks. In addition to this, we will also display useful SIMD extensions that are supported. Future Scope 1. When we move towards meson, we can leverage the additional information and revamp `np.show_config` to utilize it. 2. If needed, we can gradually merge `show_runtime` and `show_config` to display information in sections and various other formats. Please do let us know if you have any suggestions on improving this function or concerns about the information provided by the new function. Thanks, Ganesh [1] show_config displays useless info when installed through conda-forge channel <https://github.com/numpy/numpy/issues/14492> (#14492) [2] threadpoolctl <https://pypi.org/project/threadpoolctl/>

4 5

NumPy 1.23.2 release
by Charles R Harris 15 Aug '22

15 Aug '22

Hi All, On behalf of the NumPy team, I am pleased to announce the release of NumPy 1.23.2. NumPy 1.23.2 is a maintenance release that fixes bugs discovered after the 1.23.1 release. Notable features are: - Typing changes needed for Python 3.11 - Wheels for Python 3.11.0rc1 The Python versions supported for this release are 3.8-3.11. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.23.2>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.23.2>. *Contributors* A total of 9 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Alexander Grund + - Bas van Beek - Charles Harris - Jon Cusick + - Matti Picus - Michael Osthege + - Pal Barta + - Ross Barnowski - Sebastian Berg *Pull requests merged* A total of 15 pull requests were merged for this release. - #22030: ENH: Add ``__array_ufunc__`` typing support to the ``nin=1`` ufuncs - #22031: MAINT, TYP: Fix ``np.angle`` dtype-overloads - #22032: MAINT: Do not let ``_GenericAlias`` wrap the underlying classes'... - #22033: TYP,MAINT: Allow ``einsum`` subscripts to be passed via integer... - #22034: MAINT,TYP: Add object-overloads for the ``np.generic`` rich comparisons - #22035: MAINT,TYP: Allow the ``squeeze`` and ``transpose`` method to... - #22036: BUG: Fix subarray to object cast ownership details - #22037: BUG: Use ``Popen`` to silently invoke f77 -v - #22038: BUG: Avoid errors on NULL during deepcopy - #22039: DOC: Add versionchanged for converter callable behavior. - #22057: MAINT: Quiet the anaconda uploads. - #22078: ENH: reorder includes for testing on top of system installations... - #22106: TST: fix test_linear_interpolation_formula_symmetric - #22107: BUG: Fix skip condition for test_loss_of_precision[complex256] - #22115: BLD: Build python3.11.0rc1 wheels. Cheers, Charles Harris

1 0