Mailman 3 August 2023 - NumPy-Discussion

NumPy 1.26.0b1 released
by Charles R Harris Aug. 13, 2023

Aug. 13, 2023

Hi All, On behalf of the NumPy team, I'm pleased to announce the release of NumPy 1.26.0b1. The NumPy 1.26.0 release is a continuation of the 1.25.x release cycle with the addition of Python 3.12.0 support. Python 3.12 dropped distutils, consequently supporting it required finding a replacement for the setup.py/distutils based build system NumPy was using. We have chosen to use the Meson build system instead, and this is the first NumPy release supporting it. This is also the first release that supports Cython 3.0 in addition to retaining 0.29.X compatibility. Supporting those two upgrades was a large project, over 100 files have been touched in this release. The changelog doesn't capture the full extent of the work, special thanks to Ralf Gommers, Sayed Adel, Stéfan van der Walt, and Matti Picus who did much of the work in the main development branch. The highlights of this release are: - Python 3.12.0 support. - Cython 3.0.0 compatibility. - Use of the Meson build system - Updated SIMD support The Python versions supported by this release are 3.9-3.12. Note that no 32 bit wheels are provided. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.26.0b1/>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.26.0b1>. *Contributors* A total of 11 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Bas van Beek - Charles Harris - Matti Picus - Melissa Weber Mendonça - Ralf Gommers - Sayed Adel - Sebastian Berg - Stefan van der Walt - Tyler Reddy - Warren Weckesser *Pull requests merged* A total of 18 pull requests were merged for this release. - #24305: MAINT: Prepare 1.26.x branch for development - #24308: MAINT: Massive update of files from main for numpy 1.26 - #24322: CI: fix wheel builds on the 1.26.x branch - #24326: BLD: update openblas to newer version - #24327: TYP: Trim down the ``_NestedSequence.__getitem__`` signature - #24328: BUG: fix choose refcount leak - #24337: TST: fix running the test suite in builds without BLAS/LAPACK - #24338: BUG: random: Fix generation of nan by dirichlet. - #24340: MAINT: Dependabot updates from main - #24342: MAINT: Add back NPY_RUN_MYPY_IN_TESTSUITE=1 - #24353: MAINT: Update ``extbuild.py`` from main. - #24356: TST: fix distutils tests for deprecations in recent setuptools... - #24375: MAINT: Update cibuildwheel to version 2.15.0 - #24381: MAINT: Fix codespaces setup.sh script - #24403: ENH: Vendor meson for multi-target build support - #24404: BLD: vendor meson-python to make the Windows builds with SIMD... - #24405: BLD, SIMD: The meson CPU dispatcher implementation - #24406: MAINT: Remove versioneer *Build system changes* In this release, NumPy has switched to Meson as the build system and meson-python as the build backend. Installing NumPy or building a wheel can be done with standard tools like ``pip`` and ``pypa/build``. The following are supported: - Regular installs: ``pip install numpy`` or (in a cloned repo) ``pip install .`` - Building a wheel: ``python -m build`` (preferred), or ``pip wheel .`` - Editable installs: ``pip install -e . --no-build-isolation`` - Development builds through the custom CLI implemented with spin <https://github.com/scientific-python/spin>: ``spin build``. All the regular ``pip`` and ``pypa/build`` flags (e.g., ``--no-build-isolation``) should work as expected. *NumPy-specific build customization* Many of the NumPy-specific ways of customizing builds have changed. The ``NPY_*`` environment variables which control BLAS/LAPACK, SIMD, threading, and other such options are no longer supported, nor is a ``site.cfg`` file to select BLAS and LAPACK. Instead, there are command-line flags that can be passed to the build via ``pip``/``build``'s config-settings interface. These flags are all listed in the ``meson_options.txt`` file in the root of the repo. Detailed documented will be available before the final 1.26.0 release; for now please see `the SciPy building from source <http://scipy.github.io/devdocs/building/index.html> docs since most build customization works in an almost identical way in SciPy as it does in NumPy. *Build dependencies* While the runtime dependencies of NumPy have not changed, the build dependencies have. Because we temporarily vendor Meson and meson-python, there are several new dependencies - please see the ``[build-system]`` section of ``pyproject.toml`` for details. *Troubleshooting* This build system change is quite large. In case of unexpected issues, it is still possible to use a ``setup.py``-based build as a temporary workaround (on Python 3.9-3.11, not 3.12), by copying ``pyproject.toml.setuppy`` to ``pyproject.toml``. However, please open an issue with details on the NumPy issue tracker. We aim to phase out ``setup.py`` builds as soon as possible, and therefore would like to see all potential blockers surfaced early on in the 1.26.0 release cycle. Cheers, Charles Harris

1 0

Adding bit_count ufunc
by Doug Turnbull Aug. 13, 2023

Aug. 13, 2023

👋Long time numpy user, and big fan of all your work TL; DR - there's a `bit_count` method on numpy scalars, and Python ints, I'm advocating for a `ufunc` `bit_count` as has been implemented in this PR: https://github.com/numpy/numpy/pull/21429 A number of people have requested this as a numpy feature and there's a lot of great discussion at this issue https://github.com/numpy/numpy/issues/16325 A bit count comes up in certain similarity situations (like hamming distance). This involves an xor with numpy arrays and a bit count, where the bit count is currently the bottleneck by a few orders of magnitude in my local benchmarking. Currently there's a number of not particularly fast, workarounds for doing this with numpy arrays, like the bit-twiddling solutions here https://stackoverflow.com/a/68943135/8123 https://stackoverflow.com/a/109025/8123 So I'd love to be able to continue the work of bit_count -> numpy array ufunc to get performance gains at the array level 🙏

5 7

update on build system changes in NumPy's main branch
by Ralf Gommers Aug. 11, 2023

Aug. 11, 2023

Hey all, We've landed some major changes in `main` this week, so I thought it's a good idea to keep everyone in the loop. First the good news: we now have full SIMD support in the Meson builds on `main`! This was a huge amount of work by Sayed, so I'd like to say thank you to him for doing all that. This was the main missing piece of the puzzle for Python 3.12 support, and for the switch away from `numpy.distutils`. We're still in the process of some backports and other tweaks, but we're basically ready for a first 1.26.0 pre-release. We also need to port some CI jobs over in `main` from setup.py to meson, but that is separate from work for 1.26.x. Now let me give a summary of where we are at, because I anticipate these questions coming up regularly. We had to fork and vendor both Meson and meson-python, because for SIMD support we need a significant new feature in Meson ( https://github.com/mesonbuild/meson/pull/11307) that is not yet merged; once that PR does get merged and ends up in the next Meson feature release, we can drop our vendored versions. For now they live as git submodules under `vendored-meson` in the root dir of the numpy repo. And we will keep the forks at https://github.com/numpy/meson and https://github.com/numpy/meson-python. As a result of that fork, it is no longer possible to run `meson setup` in the root of the repo. This wasn't really recommended anyway, but it's good to know that it doesn't work right now. The two ways of building are: 1. Via `pip` or `pypa/build`; that will trigger the build backend via the hook in `pyproject.toml`. 2. Via the `spin` commands - this is the developer CLI (see https://github.com/scientific-python/spin). Note that editable builds work well too, with `pip install -e . --no-build-isolation`. For the remaining loose ends after the switch to Meson, see https://github.com/numpy/numpy/issues/23981. Once those are all taken care of and we have released 1.26.0 and see that there are no issues, we can remove the `setup.py` based builds. Probably towards the end of this year. We still need to update the docs for the change over to Meson - this will happen before the 1.26.0 release. For now, the SciPy docs at http://scipy.github.io/devdocs/building/index.html are a good reference; those were recently fully rewritten, and everything for NumPy is essentially the same as for SciPy. Another thing worth mentioning is that we have now, in the `main` and `1.26.x` branches, defaulted to failing the build if BLAS/LAPACK cannot be found. This is because all the detection mechanisms changed, and it would otherwise be too easy to silently get a build that is unoptimized (that can cause up to 100x slowdowns in widely used linalg functions). If you get a failure, you can fix it or pass a `-Dallow-noblas` CLI flag to indicate that that is what you intended (or at least, you don't mind a slow build). We may still revert the change for 1.26.0 and default to switching to `lapack_lite`, for the discussion on that and more details, see https://github.com/numpy/numpy/pull/24279. A difference between the meson and setup.py builds is that rather than having a host of `NPY_` environment variables, we now have CLI flags for build options. These options can be queried and the used ones are reported in the build log, so it makes it much easier to understand what is being used. It does mean though that if you have a set of env vars in, for example, your `.bashrc`, those will no longer have an effect. For a full list of options, see the `meson_options.txt` file in the root of the repo. Note that some of the options for BLAS/LAPACK control (e.g., `NPY_BLAS_ORDER`) are not yet available. Those should arrive within the next couple of months. Cheers, Ralf

2 1

GSOD-NumPy 2023: Review at next Community Meeting
by Mars Lee Aug. 10, 2023

Aug. 10, 2023

Hi everyone! I'll be hosting a review session for the '*NumPy Contributor Comic*'. It will be hosted at the next NumPy Community meeting on* August 16*. It will take about 20 minutes, going through the script. You can also comment on the review thread on Github <https://github.com/numpy/numpy/issues/24382>or on the Google Docs <https://docs.google.com/document/d/1fjLTDqSkcKMxo8oSTRThllBFpjo5rXS9/edit?u…>. These will be open for the next two weeks and close on August 23. See you there! - Mars

1 0

Python 3.12.0rc1 released
by Charles R Harris Aug. 8, 2023

Aug. 8, 2023

Hi All, Just a note that Python 3.12.0rc1 was released Aug 6. It looks like manylinux2014 is already updated. Chuck

1 1

Interesting article
by Charles R Harris Aug. 7, 2023

Aug. 7, 2023

Hi All, Just thought I'd pass along a link to an article I found interesting, IBM, Red Hat and Free Software: An old maddog’s view <https://www.lpi.org/blog/2023/07/30/ibm-red-hat-and-free-software-an-old-ma…> . Chuck

2 1

xp.asarray semantics
by Andrew Nelson Aug. 7, 2023

Aug. 7, 2023

The array-api documentation ( https://data-apis.org/array-api/latest/API_specification/generated/array_ap…) for `asarray` has a copy keyword, the numpy version doesn't. Is there a plan to add such a keyword to no.asarray? I guess this is part of a general question, how much numpy convergence there will be to the array-api specifications over time? Andrew.

2 1

ANN: NumExpr 2.8.5
by Robert McLeod Aug. 6, 2023

Aug. 6, 2023

Hi everyone, In 2.8.5 we have added a new function, `validate` which checks an expression `ex` for validity, for usage where the program is parsing a user input. There are also consequences for this sort of usage, since `eval(ex)` is called, and as such we do some string sanitization as described below. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.8.4 to 2.8.5 --------------------------- * A `validate` function has been added. This function checks the inputs, returning `None` on success or raising an exception on invalid inputs. This function was added as numerous projects seem to be using NumExpr for parsing user inputs. `re_evaluate` may be called directly following `validate`. * As an addendum to the use of NumExpr for parsing user inputs, is that NumExpr calls `eval` on the inputs. A regular expression is now applied to help sanitize the input expression string, forbidding '__', ':', and ';'. Attribute access is also banned except for '.r' for real and '.i' for imag. * Thanks to timbrist for a fix to behavior of NumExpr with integers to negative powers. NumExpr was pre-checking integer powers for negative values, which was both inefficient and caused parsing errors in some situations. Now NumExpr will simply return 0 as a result for such cases. While NumExpr generally tries to follow NumPy behavior, performance is also critical. * Thanks to peadar for some fixes to how NumExpr launches threads for embedded applications. * Thanks to de11n for making parsing of the `site.cfg` for MKL consistent among all shared platforms. What's Numexpr? --------------- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? ------------------------- The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience --------------------- Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod robbmcleod(a)gmail.com robert.mcleod(a)hitachi-hightech.com

1 0

[JOB] Astropy Research Software engineer
by Aldcroft, Thomas Aug. 4, 2023

Aug. 4, 2023

Astropy is hiring a Research Software engineer. We are looking for people who can spend 50-100% of their time on Astropy development in the next 6-9 months. Qualified candidates can range from software developers with open source experience to astronomy students with software experience. *If you are interested, please apply! If you know a qualified candidate, please encourage them to apply: https://jobs.numfocus.org/job/2023-07-31_astropy <https://jobs.numfocus.org/job/2023-07-31_astropy>* The role will include solving some long-standing bugs and issues in Astropy and some coordinated packages as well as making progress on items that the community already prioritized on the Astropy roadmap, but that no volunteer has yet found time to tackle. Thanks, Tom Aldcroft, on behalf of the Astropy Project

1 0

numpy.org is now available in Japanese and Portuguese
by Inessa Pawson Aug. 4, 2023

Aug. 4, 2023

We are excited to announce that numpy.org is now available in 2 additional languages: Japanese and Portuguese. This wouldn’t be possible without our dedicated volunteers: Portuguese: Melissa Weber Mendonça (melissawm) Ricardo Prins (ricardoprins) Getúlio Silva (getuliosilva) Julio Batista Silva (jbsilva) Alexandre de Siqueira (alexdesiqueira) Alexandre B A Villares (villares) Vini Salazar (vinisalazar) Japanese: Atsushi Sakai (AtsushiSakai) KKunai Tom Kelly (TomKellyGenetics) Yuji Kanagawa (kngwyu) Looking ahead, we’d love to translate the website into more languages. If you’d like to help, please connect with the NumPy translations team on Slack: https://join.slack.com/t/numpy-team/shared_invite/zt-1gokbq56s-bvEpo10Ef7aH…. (Look for the #translations channel.) We are also building a translations team who will be working on localizing documentation and educational content across the Scientific Python ecosystem. If this piqued your interest, join us on the Scientific Python Discord: https://discord.gg/khWtqY6RKr. (Look for the #translation channel.) The work on the translation infrastructure is supported, in part, with funding from the Chan Zuckerberg Initiative. -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

3 3