Mailman 3 September 2023 - NumPy-Discussion

next NumPy triage meeting - now starting at 5 pm UTC
by Inessa Pawson 26 Nov '23

26 Nov '23

The next NumPy triage meeting will be held this Wednesday, January 11th at 5 pm UTC. (Please note the new time!) This is a meeting where we synchronously triage prioritized PRs and issues. Join us via Zoom: https://numfocus-org.zoom.us/j/82096749952?pwd=MW9oUmtKQ1c3a2gydGk1RTdYUUVX… Everyone is welcome to attend and contribute to a conversation. Please notify us of issues or PRs that you’d like to have reviewed by adding a GitHub link to them in the meeting agenda: https://hackmd.io/68i_JvOYQfy9ERiHgXMPvg. -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

1 23

next NumPy Newcomers' Hour - 12 pm UTC
by Inessa Pawson 31 Oct '23

31 Oct '23

Our next Newcomers' Hour will be held tomorrow, December 1st, at 12 pm UTC. Stop by to ask questions or just to say hi. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/3f3otyyuTte3FU9y3QzsLg?both Join the meeting via Zoom: https://us06web.zoom.us/j/82563808729?pwd=ZFU3Z2dMcXBGb05YemRsaGE1OW5nQT09 -- Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ GitHub: inessapawson

2 5

next NumPy community meeting
by Inessa Pawson 22 Oct '23

22 Oct '23

The next NumPy community meeting will be held this Wednesday, August 17th at 18:00 (6 pm) UTC. Join us via Zoom: https://us06web.zoom.us/j/83278611437?pwd=ekhoLzlHRjdWc0NOY2FQM0NPemdkZz09 Everyone is welcome and encouraged to attend. To add to the meeting agenda the topics you’d like to discuss, follow the link: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both For the notes from the previous meeting, visit: https://github.com/numpy/archive/blob/main/community_meetings/community-202… Cheers, Inessa Inessa Pawson Contributor Experience Lead | NumPy https://numpy.org/ Twitter: @inessapawson

1 30

Curious performance different with np.unique on arrays of characters
by saladi＠caltech.edu 29 Sep '23

29 Sep '23

Hello - In the course of some genomics simulations, I seem to have come across a curious (to me at least) performance difference in np.unique that I wanted to share. (If this is not the right forum for this, please let me know!) With a np.array of characters (U1), np.unique seems to be much faster when doing np.view as int -> np.unique -> np.view as U1 for arrays of decent size. I would not have expected this since np.unique knows what's coming in as S1 and could handle the view-stuff internally. I've played with this a number of ways (e.g. S1 vs U1; int32 vs int64; return_counts = True vs False; 100, 1000, or 10k elements) and seem to notice the same pattern. A short illustration below with U1, int32, return_counts = False, 10 vs 10k. I wonder if this is actually intended behavior, i.e. the view-stuff is actually a good idea for the user to think about and implement if appropriate for their usecase (as it is for me). Best regards, Shyam import numpy as np charlist_10 = np.array(list('ASDFGHJKLZ'), dtype='U1') charlist_10k = np.array(list('ASDFGHJKLZ' * 1000), dtype='U1') def unique_basic(x): return np.unique(x) def unique_view(x): return np.unique(x.view(np.int32)).view(x.dtype) In [27]: %timeit unique_basic(charlist_10) 2.17 µs ± 40.7 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each) In [28]: %timeit unique_view(charlist_10) 2.53 µs ± 38.4 ns per loop (mean ± std. dev. of 7 runs, 100,000 loops each) In [29]: %timeit unique_basic(charlist_10k) 204 µs ± 4.61 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) In [30]: %timeit unique_view(charlist_10k) 66.7 µs ± 2.91 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each) In [31]: np.__version__ Out[31]: '1.25.2' -- Shyam Saladi https://shyam.saladi.org

7 7

ANN: SciPy 1.11.3
by Tyler Reddy 27 Sep '23

27 Sep '23

Hi all, On behalf of the SciPy development team, I'm pleased to announce the release of SciPy 1.11.3, which is a bug fix release. Sources and binary wheels can be found at: https://pypi.org/project/scipy/ and at: https://github.com/scipy/scipy/releases/tag/v1.11.3 One of a few ways to install this release with pip: pip install scipy==1.11.3 ====================== SciPy 1.11.3 Release Notes ====================== SciPy 1.11.3 is a bug-fix release with no new features compared to 1.11.2. Authors ======= * Name (commits) * Jake Bowhay (2) * CJ Carey (1) * Colin Carroll (1) + * Anirudh Dagar (2) * drestebon (1) + * Ralf Gommers (5) * Matt Haberland (2) * Julien Jerphanion (1) * Uwe L. Korn (1) + * Ellie Litwack (2) * Andrew Nelson (5) * Bharat Raghunathan (1) * Tyler Reddy (37) * Søren Fuglede Jørgensen (2) * Hielke Walinga (1) + * Warren Weckesser (1) * Bernhard M. Wiedemann (1) A total of 17 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 1.11.3 -------------------------------- * `#15093 <https://github.com/scipy/scipy/issues/15093>`__: BUG: scipy.optimize's trust-constr algorithm hangs when keep-feasible... * `#15273 <https://github.com/scipy/scipy/issues/15273>`__: freqz: suboptimal performance for worN=2\*\*n+1, include_nyquist=True... * `#17269 <https://github.com/scipy/scipy/issues/17269>`__: Bug in scipy.sparse.csgraph.min_weight_full_bipartite_matching * `#17289 <https://github.com/scipy/scipy/issues/17289>`__: BUG: Different results between numpy.fft.rfft and scipy.signal.freqz * `#18716 <https://github.com/scipy/scipy/issues/18716>`__: Buffer dtype mismatch, expected 'ITYPE_t' but got 'long' * `#18782 <https://github.com/scipy/scipy/issues/18782>`__: BUG: johnsonsu distribution no longer accepts integer \`b\` parameter * `#18922 <https://github.com/scipy/scipy/issues/18922>`__: BUG: dev.py has \`distutils\` usage * `#19101 <https://github.com/scipy/scipy/issues/19101>`__: BUG: mesonpy embeds random path in .pyx files * `#19103 <https://github.com/scipy/scipy/issues/19103>`__: BUG: Regression in 1.11.2: optimize.least_squares with method='trf'... * `#19132 <https://github.com/scipy/scipy/issues/19132>`__: BUG: Build fails on latest commit * `#19149 <https://github.com/scipy/scipy/issues/19149>`__: BUG: scipy.sparse.csgraph.laplacian raises AttributeError on... * `#19197 <https://github.com/scipy/scipy/issues/19197>`__: BUG: Incorrect sampling from zero rank covariance Pull requests for 1.11.3 ------------------------------- * `#17633 <https://github.com/scipy/scipy/pull/17633>`__: BUG: add infeasibility checks to min_weight_full_bipartite_matching * `#18784 <https://github.com/scipy/scipy/pull/18784>`__: BUG: Allow johnsonsu parameters to be floats * `#18913 <https://github.com/scipy/scipy/pull/18913>`__: BUG: sparse.csgraph: Support int64 indices in traversal.pyx * `#18924 <https://github.com/scipy/scipy/pull/18924>`__: BUG: Fix python3.12 distutils dev.py build * `#18956 <https://github.com/scipy/scipy/pull/18956>`__: BUG: trust-constr Bounds exclusive * `#19076 <https://github.com/scipy/scipy/pull/19076>`__: MAINT: should not be using np.float64() on arrays * `#19084 <https://github.com/scipy/scipy/pull/19084>`__: REL, MAINT: prep for 1.11.3 * `#19111 <https://github.com/scipy/scipy/pull/19111>`__: BUG: Fixes #19103 by adding back make_strictly_feasible to lsq... * `#19123 <https://github.com/scipy/scipy/pull/19123>`__: BLD: Avoid absolute pathnames in .pyx files * `#19135 <https://github.com/scipy/scipy/pull/19135>`__: MAINT: signal: Remove the cval parameter from the private function... * `#19139 <https://github.com/scipy/scipy/pull/19139>`__: BLD: revert to using published wheels [wheel build] * `#19156 <https://github.com/scipy/scipy/pull/19156>`__: BUG: Support sparse arrays in scipy.sparse.csgraph.laplacian * `#19199 <https://github.com/scipy/scipy/pull/19199>`__: MAINT: stats.CovViaEigendecomposition: fix \`_LA\` attribute... * `#19200 <https://github.com/scipy/scipy/pull/19200>`__: TST: fix \`TestODR.test_implicit\` test failure with tolerance... * `#19208 <https://github.com/scipy/scipy/pull/19208>`__: BUG: freqz rfft grid fix * `#19280 <https://github.com/scipy/scipy/pull/19280>`__: MAINT: newton, make sure x0 is an inexact type * `#19286 <https://github.com/scipy/scipy/pull/19286>`__: BUG: stats: fix build failure due to incorrect Boost policies... * `#19290 <https://github.com/scipy/scipy/pull/19290>`__: BLD: add float.h include to \`_fpumode.c\`, fixes Clang on Windows... * `#19299 <https://github.com/scipy/scipy/pull/19299>`__: MAINT: fix libquadmath licence Checksums ========= MD5 ~~~ b940b7bab3db8f80d6f79f214ff48ec9 Changelog f744557fb5b0f5094635a35255561195 README.txt df92549e1d2832c0685d851303685301 scipy-1.11.3-cp310-cp310-macosx_10_9_x86_64.whl 989c613e807e1ec39b9ce7aced5b7f6d scipy-1.11.3-cp310-cp310-macosx_12_0_arm64.whl ee99c50f95ce57cb08ea701f5fa9e0b2 scipy-1.11.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 8c5c0a84ab318a916fc0695902e64a1b scipy-1.11.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 84efe7d92207e2bbd659f1b508de95a2 scipy-1.11.3-cp310-cp310-musllinux_1_1_x86_64.whl 860afed0364ae3275ddd96ec8b7c0a4b scipy-1.11.3-cp310-cp310-win_amd64.whl c8d0048c75e8c3bbb89867138446c716 scipy-1.11.3-cp311-cp311-macosx_10_9_x86_64.whl 77cbc738dcb07f61e21371ac0867c6f2 scipy-1.11.3-cp311-cp311-macosx_12_0_arm64.whl 2316f98543b94595e89680a5f07eed8d scipy-1.11.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 49bf268f522be17dc485097084d0e22d scipy-1.11.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 6d3781c9bb920311ba5ef791c62fe057 scipy-1.11.3-cp311-cp311-musllinux_1_1_x86_64.whl 48ca42c3d6e8a380d76baa3ab40268e3 scipy-1.11.3-cp311-cp311-win_amd64.whl 82678417b8a11c4e636764fcfaec5629 scipy-1.11.3-cp312-cp312-macosx_10_9_x86_64.whl ab13a6a0119533426194072ba988d398 scipy-1.11.3-cp312-cp312-macosx_12_0_arm64.whl 3198a523b0bc2c44bcc0d3da38ae431b scipy-1.11.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 4164b74ed3f7b5ff5682d5d0a37fbca0 scipy-1.11.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl e2cc2a1fa3240428402e5a147bd3f6c3 scipy-1.11.3-cp312-cp312-musllinux_1_1_x86_64.whl 22effdc789f3f914f419bad614bc0698 scipy-1.11.3-cp312-cp312-win_amd64.whl 73cde92a345dfa9a71389e1e786e455e scipy-1.11.3-cp39-cp39-macosx_10_9_x86_64.whl 21a4e09de491b366e9f884bbaf773c73 scipy-1.11.3-cp39-cp39-macosx_12_0_arm64.whl e917d5ac9167cb4da4a864c44ff72d61 scipy-1.11.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl c0d7ea4212f34224984a55f95154af39 scipy-1.11.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 1325fc11a69f1f8955b6a3dc455c509b scipy-1.11.3-cp39-cp39-musllinux_1_1_x86_64.whl 21f7168cfd1c85df41b4f4e192c84ca6 scipy-1.11.3-cp39-cp39-win_amd64.whl SHA256 ~~~~~~ b81da8c401e6e0e14d0fc6742d3293f1c6493ef810738e871a4d5d3385397010 Changelog 76cc526909e4f1063632b6a1d623a9d84bc86902c1e9a8c4f44a980eb640d145 README.txt 370f569c57e1d888304052c18e58f4a927338eafdaef78613c685ca2ea0d1fa0 scipy-1.11.3-cp310-cp310-macosx_10_9_x86_64.whl 9885e3e4f13b2bd44aaf2a1a6390a11add9f48d5295f7a592393ceb8991577a3 scipy-1.11.3-cp310-cp310-macosx_12_0_arm64.whl e04aa19acc324a1a076abb4035dabe9b64badb19f76ad9c798bde39d41025cdc scipy-1.11.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 3e1a8a4657673bfae1e05e1e1d6e94b0cabe5ed0c7c144c8aa7b7dbb774ce5c1 scipy-1.11.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 7abda0e62ef00cde826d441485e2e32fe737bdddee3324e35c0e01dee65e2a88 scipy-1.11.3-cp310-cp310-musllinux_1_1_x86_64.whl 033c3fd95d55012dd1148b201b72ae854d5086d25e7c316ec9850de4fe776929 scipy-1.11.3-cp310-cp310-win_amd64.whl 925c6f09d0053b1c0f90b2d92d03b261e889b20d1c9b08a3a51f61afc5f58165 scipy-1.11.3-cp311-cp311-macosx_10_9_x86_64.whl 5664e364f90be8219283eeb844323ff8cd79d7acbd64e15eb9c46b9bc7f6a42a scipy-1.11.3-cp311-cp311-macosx_12_0_arm64.whl 00f325434b6424952fbb636506f0567898dca7b0f7654d48f1c382ea338ce9a3 scipy-1.11.3-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 5f290cf561a4b4edfe8d1001ee4be6da60c1c4ea712985b58bf6bc62badee221 scipy-1.11.3-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 91770cb3b1e81ae19463b3c235bf1e0e330767dca9eb4cd73ba3ded6c4151e4d scipy-1.11.3-cp311-cp311-musllinux_1_1_x86_64.whl e1f97cd89c0fe1a0685f8f89d85fa305deb3067d0668151571ba50913e445820 scipy-1.11.3-cp311-cp311-win_amd64.whl dfcc1552add7cb7c13fb70efcb2389d0624d571aaf2c80b04117e2755a0c5d15 scipy-1.11.3-cp312-cp312-macosx_10_9_x86_64.whl 0d3a136ae1ff0883fffbb1b05b0b2fea251cb1046a5077d0b435a1839b3e52b7 scipy-1.11.3-cp312-cp312-macosx_12_0_arm64.whl bae66a2d7d5768eaa33008fa5a974389f167183c87bf39160d3fefe6664f8ddc scipy-1.11.3-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl d2f6dee6cbb0e263b8142ed587bc93e3ed5e777f1f75448d24fb923d9fd4dce6 scipy-1.11.3-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 74e89dc5e00201e71dd94f5f382ab1c6a9f3ff806c7d24e4e90928bb1aafb280 scipy-1.11.3-cp312-cp312-musllinux_1_1_x86_64.whl 90271dbde4be191522b3903fc97334e3956d7cfb9cce3f0718d0ab4fd7d8bfd6 scipy-1.11.3-cp312-cp312-win_amd64.whl a63d1ec9cadecce838467ce0631c17c15c7197ae61e49429434ba01d618caa83 scipy-1.11.3-cp39-cp39-macosx_10_9_x86_64.whl 5305792c7110e32ff155aed0df46aa60a60fc6e52cd4ee02cdeb67eaccd5356e scipy-1.11.3-cp39-cp39-macosx_12_0_arm64.whl 9ea7f579182d83d00fed0e5c11a4aa5ffe01460444219dedc448a36adf0c3917 scipy-1.11.3-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl c77da50c9a91e23beb63c2a711ef9e9ca9a2060442757dffee34ea41847d8156 scipy-1.11.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 15f237e890c24aef6891c7d008f9ff7e758c6ef39a2b5df264650eb7900403c0 scipy-1.11.3-cp39-cp39-musllinux_1_1_x86_64.whl 4b4bb134c7aa457e26cc6ea482b016fef45db71417d55cc6d8f43d799cdf9ef2 scipy-1.11.3-cp39-cp39-win_amd64.whl

1 0

ANN: NumExpr 2.8.7
by Francesc Alted 26 Sep '23

26 Sep '23

Hi everyone, NumExpr 2.8.7 is a release to deal with issues related to downstream `pandas` and other projects where the sanitization blacklist was triggering issues in their evaluate. Hopefully, the new sanitization code would be much more robust now. For those who do not wish to have sanitization on by default, it can be changed by setting an environment variable, `NUMEXPR_SANITIZE=0`. If you use `pandas` in your packages it is advisable you pin `numexpr >= 2.8.7` in your requirements. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.8.6 to 2.8.7 --------------------------- * More permissive rules in sanitizing regular expression: allow to access digits after the . with scientific notation. Thanks to Thomas Vincent. * Don't reject double underscores that are not at the start or end of a variable name (pandas uses those), or scientific-notation numbers with digits after the decimal point. Thanks to Rebecca Palmer. * Do not use `numpy.alltrue` in the test suite, as it has been deprecated (replaced by `numpy.all`). Thanks to Rebecca Chen. * Wheels for Python 3.12. Wheels for 3.7 and 3.8 are not generated anymore. What's Numexpr? --------------- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? ------------------------- The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience --------------------- Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Francesc Alted

1 0

NEP 55 - Add a UTF-8 Variable-Width String DType to NumPy
by Nathan 20 Sep '23

20 Sep '23

Hello all, I just opened a pull request to add NEP 55, see https://github.com/numpy/numpy/pull/24483. Per NEP 0, I've copied everything up to the "detailed description" section below. I'm looking forward to your feedback on this. -Nathan Goldbaum ========================================================= NEP 55 — Add a UTF-8 Variable-Width String DType to NumPy ========================================================= :Author: Nathan Goldbaum <ngoldbaum(a)quansight.com> :Status: Draft :Type: Standards Track :Created: 2023-06-29 Abstract -------- We propose adding a new string data type to NumPy where each item in the array is an arbitrary length UTF-8 encoded string. This will enable performance, memory usage, and usability improvements for NumPy users, including: * Memory savings for workflows that currently use fixed-width strings and store primarily ASCII data or a mix of short and long strings in a single NumPy array. * Downstream libraries and users will be able to move away from object arrays currently used as a substitute for variable-length string arrays, unlocking performance improvements by avoiding passes over the data outside of NumPy. * A more intuitive user-facing API for working with arrays of Python strings, without a need to think about the in-memory array representation. Motivation and Scope -------------------- First, we will describe how the current state of support for string or string-like data in NumPy arose. Next, we will summarize the last major previous discussion about this topic. Finally, we will describe the scope of the proposed changes to NumPy as well as changes that are explicitly out of scope of this proposal. History of String Support in Numpy ********************************** Support in NumPy for textual data evolved organically in response to early user needs and then changes in the Python ecosystem. Support for strings was added to numpy to support users of the NumArray ``chararray`` type. Remnants of this are still visible in the NumPy API: string-related functionality lives in ``np.char``, to support the obsolete ``np.char.chararray`` class, deprecated since NumPy 1.4 in favor of string DTypes. NumPy's ``bytes_`` DType was originally used to represent the Python 2 ``str `` type before Python 3 support was added to NumPy. The bytes DType makes the most sense when it is used to represent Python 2 strings or other null-terminated byte sequences. However, ignoring data after the first null character means the ``bytes_`` DType is only suitable for bytestreams that do not contain nulls, so it is a poor match for generic bytestreams. The ``unicode`` DType was added to support the Python 2 ``unicode`` type. It stores data in 32-bit UCS-4 codepoints (e.g. a UTF-32 encoding), which makes for a straightforward implementation, but is inefficient for storing text that can be represented well using a one-byte ASCII or Latin-1 encoding. This was not a problem in Python 2, where ASCII or mostly-ASCII text could use the Python 2 ``str`` DType (the current ``bytes_`` DType). With the arrival of Python 3 support in NumPy, the string DTypes were largely left alone due to backward compatibility concerns, although the unicode DType became the default DType for ``str`` data and the old ``string`` DType was renamed the ``bytes_`` DType. This change left NumPy with the sub-optimal situation of shipping a data type originally intended for null-terminated bytestrings as the data type for *all* python ``bytes`` data, and a default string type with an in-memory representation that consumes four times as much memory as needed for ASCII or mostly-ASCII data. Problems with Fixed-Width Strings ********************************* Both existing string DTypes represent fixed-width sequences, allowing storage of the string data in the array buffer. This avoids adding out-of-band storage to NumPy, however, it makes for an awkward user interface. In particular, the maximum string size must be inferred by NumPy or estimated by the user before loading the data into a NumPy array or selecting an output DType for string operations. In the worst case, this requires an expensive pass over the full dataset to calculate the maximum length of an array element. It also wastes memory when array elements have varying lengths. Pathological cases where an array stores many short strings and a few very long strings are particularly bad for wasting memory. Downstream usage of string data in NumPy arrays has proven out the need for a variable-width string data type. In practice, most downstream users employ ``object`` arrays for this purpose. In particular, ``pandas`` has explicitly deprecated support for NumPy fixed-width strings, coerces NumPy fixed-width string arrays to ``object`` arrays, and in the future may switch to only supporting string data via ``PyArrow``, which has native support for UTF-8 encoded variable-width string arrays [1]_. This is unfortunate, since `` object`` arrays have no type guarantees, necessitating expensive sanitization passes and operations using object arrays cannot release the GIL. Previous Discussions -------------------- The project last discussed this topic in depth in 2017, when Julian Taylor proposed a fixed-width text data type parameterized by an encoding [2]_. This started a wide-ranging discussion about pain points for working with string data in NumPy and possible ways forward. In the end, the discussion identified two use-cases that the current support for strings does a poor job of handling: * Loading or memory-mapping scientific datasets with unknown encoding, * Working with string data in a manner that allows transparent conversion between NumPy arrays and Python strings, including support for missing strings. As a result of this discussion, improving support for string data was added to the NumPy project roadmap [3]_, with an explicit call-out to add a DType better suited to memory-mapping bytes with any or no encoding, and a variable-width string DType that supports missing data to replace usages of object string arrays. Proposed work ------------- This NEP proposes adding ``StringDType``, a DType that stores variable-width heap-allocated strings in Numpy arrays, to replace downstream usages of the ``object`` DType for string data. This work will heavily leverage recent improvements in NumPy to improve support for user-defined DTypes, so we will also necessarily be working on the data type internals in NumPy. In particular, we propose to: * Add a new variable-length string DType to NumPy, targeting NumPy 2.0. * Work out issues related to adding a DType implemented using the experimental DType API to NumPy itself. * Support for a user-provided missing data sentinel. * A cleanup of ``np.char``, with the ufunc-like functions moved to a new namespace for functions and types related to string support. * An update to the ``npy`` and ``npz`` file formats to allow storage of arbitrary-length sidecar data. The following is out of scope for this work: * Changing DType inference for string data. * Adding a DType for memory-mapping text in unknown encodings or a DType that attempts to fix issues with the ``bytes_`` DType. * Fully agreeing on the semantics of a missing data sentinels or adding a missing data sentinel to NumPy itself. * Implement fast ufuncs or SIMD optimizations for string operations. While we're explicitly ruling out implementing these items as part of this work, adding a new string DType helps set up future work that does implement some of these items. If implemented this NEP will make it easier to add a new fixed-width text DType in the future by moving string operations into a long-term supported namespace. We are also proposing a memory layout that should be amenable to writing fast ufuncs and SIMD optimization in some cases, increasing the payoff for writing string operations as SIMD-optimized ufuncs in the future. While we are not proposing adding a missing data sentinel to NumPy, we are proposing adding support for an optional, user-provided missing data sentinel, so this does move NumPy a little closer to officially supporting missing data. We are attempting to avoid resolving the disagreement described in :ref:`NEP 26<NEP26>` and this proposal does not require or preclude adding a missing data sentinel or bitflag-based missing data support in the future. Usage and Impact ---------------- The DType is intended as a drop-in replacement for object string arrays. This means that we intend to support as many downstream usages of object string arrays as possible, including all supported NumPy functionality. Pandas is the obvious first user, and substantial work has already occurred to add support in a fork of Pandas. ``scikit-learn`` also uses object string arrays and will be able to migrate to a DType with guarantees that the arrays contains only strings. Both h5py [4]_ and PyTables [5]_ will be able to add first-class support for variable-width UTF-8 encoded string datasets in HDF5. String data are heavily used in machine-learning workflows and downstream machine learning libraries will be able to leverage this new DType. Users who wish to load string data into NumPy and leverage NumPy features like fancy advanced indexing will have a natural choice that offers substantial memory savings over fixed-width unicode strings and better validation guarantees and overall integration with NumPy than object string arrays. Moving to a first-class string DType also removes the need to acquire the GIL during string operations, unlocking future optimizations that are impossible with object string arrays. Performance *********** Here we briefly describe preliminary performance measurements of the prototype version of ``StringDType`` we have implemented outside of NumPy using the experimental DType API. All benchmarks in this section were performed on a Dell XPS 13 9380 running Ubuntu 22.04 and Python 3.11.3 compiled using pyenv. NumPy, Pandas, and the ``StringDType`` prototype were all compiled with meson release builds. Currently, the ``StringDType`` prototype has comparable performance with object arrays and fixed-width string arrays. One exception is array creation from python strings, performance is somewhat slower than object arrays and comparable to fixed-width unicode arrays:: In [1]: from stringdtype import StringDType In [2]: import numpy as np In [3]: data = [str(i) * 10 for i in range(100_000)] In [4]: %timeit arr_object = np.array(data, dtype=object) 3.55 ms ± 51.3 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) In [5]: %timeit arr_stringdtype = np.array(data, dtype=StringDType()) 12.9 ms ± 277 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) In [6]: %timeit arr_strdtype = np.array(data, dtype=str) 11.7 ms ± 150 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) In this example, object DTypes are substantially faster because the objects in the ``data`` list can be directly interned in the array, while ``StrDType`` and ``StringDType`` need to copy the string data and ``StringDType`` needs to convert the data to UTF-8 and perform additional heap allocations outside the array buffer. In the future, if Python moves to a UTF-8 internal representation for strings, the string loading performance of ``StringDType`` should improve. String operations have similar performance:: In [7]: %timeit np.array([s.capitalize() for s in data], dtype=object) 30.2 ms ± 109 µs per loop (mean ± std. dev. of 7 runs, 10 loops each) In [8]: %timeit np.char.capitalize(arr_stringdtype) 38.5 ms ± 3.01 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) In [9]: %timeit np.char.capitalize(arr_strdtype) 46.4 ms ± 1.32 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) The poor performance here is a reflection of the slow iterator-based implementation of operations in ``np.char``. If we were to rewrite these operations as ufuncs, we could unlock substantial performance improvements. Using the example of the ``add`` ufunc, which we have implemented for the ``StringDType`` prototype:: In [10]: %timeit arr_object + arr_object 10 ms ± 308 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) In [11]: %timeit arr_stringdtype + arr_stringdtype 5.91 ms ± 18.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each) In [12]: %timeit np.char.add(arr_strdtype, arr_strdtype) 65.9 ms ± 1.3 ms per loop (mean ± std. dev. of 7 runs, 10 loops each) As described below, we have already updated a fork of Pandas to use a prototype version of ``StringDType``. This demonstrates the performance improvements available when data are already loaded into a NumPy array and are passed to a third-party library. Currently Pandas attempts to coerce all ``str`` data to ``object`` DType by default, and has to check and sanitize existing ``object `` arrays that are passed in. This requires a copy or pass over the data made unnecessary by first-class support for variable-width strings in both NumPy and Pandas:: In [13]: import pandas as pd In [14]: %timeit pd.Series(arr_stringdtype) 20.9 µs ± 341 ns per loop (mean ± std. dev. of 7 runs, 10,000 loops each) In [15]: %timeit pd.Series(arr_object) 1.08 ms ± 23.4 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) We have also implemented a Pandas extension DType that uses ``StringDType`` under the hood, which is also substantially faster for creating Pandas data structures than the existing Pandas string DType that uses ``object`` arrays:: In [16]: %timeit pd.Series(arr_stringdtype, dtype='string[numpy]') 54.7 µs ± 1.38 µs per loop (mean ± std. dev. of 7 runs, 10,000 loops each) In [17]: %timeit pd.Series(arr_object, dtype='string[python]') 1.39 ms ± 1.16 µs per loop (mean ± std. dev. of 7 runs, 1,000 loops each) Backward compatibility ---------------------- We are not proposing a change to DType inference for python strings and do not expect to see any impacts on existing usages of NumPy, besides warnings or errors related to new deprecations or expiring deprecations in ``np.char``.

5 11

NumPy 1.26.0 released
by Charles R Harris 16 Sep '23

16 Sep '23

Hi All, On behalf of the NumPy team, I'm pleased to announce the release of NumPy 1.26.0. The NumPy 1.26.0 release is a continuation of the 1.25.x release cycle with the addition of Python 3.12.0 support. Python 3.12 dropped distutils, consequently supporting it required finding a replacement for the setup.py/distutils based build system NumPy was using. We have chosen to use the Meson build system instead, and this is the first NumPy release supporting it. This is also the first release that supports Cython 3.0 in addition to retaining 0.29.X compatibility. Supporting those two upgrades was a large project, over 100 files have been touched in this release. The changelog doesn't capture the full extent of the work, special thanks to Ralf Gommers, Sayed Adel, Stéfan van der Walt, and Matti Picus who did much of the work in the main development branch. The highlights of this release are: - Python 3.12.0 support. - Cython 3.0.0 compatibility. - Use of the Meson build system - Updated SIMD support - f2py fixes, meson and bind(x) support - Support for the updated Accelerate BLAS/LAPACK library The Python versions supported by this release are 3.9-3.12. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.26.0/>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.26.0> *Contributors* A total of 20 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - @DWesl - Albert Steppi + - Bas van Beek - Charles Harris - Developer-Ecosystem-Engineering - Filipe Laíns + - Jake Vanderplas - Liang Yan + - Marten van Kerkwijk - Matti Picus - Melissa Weber Mendonça - Namami Shanker - Nathan Goldbaum - Ralf Gommers - Rohit Goswami - Sayed Adel - Sebastian Berg - Stefan van der Walt - Tyler Reddy - Warren Weckesser *Build system changes* In this release, NumPy has switched to Meson as the build system and meson-python as the build backend. Installing NumPy or building a wheel can be done with standard tools like ``pip`` and ``pypa/build``. The following are supported: - Regular installs: ``pip install numpy`` or (in a cloned repo) ``pip install .`` - Building a wheel: ``python -m build`` (preferred), or ``pip wheel .`` - Editable installs: ``pip install -e . --no-build-isolation`` - Development builds through the custom CLI implemented with spin <https://github.com/scientific-python/spin>: ``spin build``. All the regular ``pip`` and ``pypa/build`` flags (e.g., ``--no-build-isolation``) should work as expected. *NumPy-specific build customization* Many of the NumPy-specific ways of customizing builds have changed. The ``NPY_*`` environment variables which control BLAS/LAPACK, SIMD, threading, and other such options are no longer supported, nor is a ``site.cfg`` file to select BLAS and LAPACK. Instead, there are command-line flags that can be passed to the build via ``pip``/``build``'s config-settings interface. These flags are all listed in the ``meson_options.txt`` file in the root of the repo. Detailed documented will be available before the final 1.26.0 release; for now, see the SciPy building from source <http://scipy.github.io/devdocs/building/index.html> docs since most build customization works in an almost identical way in SciPy as it does in NumPy. *Build dependencies* While the runtime dependencies of NumPy have not changed, the build dependencies have. Because we temporarily vendor Meson and meson-python, there are several new dependencies - please see the ``[build-system]`` section of ``pyproject.toml`` for details. *Troubleshooting* This build system change is quite large. In case of unexpected issues, it is still possible to use a ``setup.py``-based build as a temporary workaround (on Python 3.9-3.11, not 3.12), by copying ``pyproject.toml.setuppy`` to ``pyproject.toml``. However, please open an issue with details on the NumPy issue tracker. We aim to phase out ``setup.py`` builds as soon as possible, and therefore would like to see all potential blockers surfaced early on in the 1.26.0 release cycle. Cheers, Charles Harris

1 0

Proposal to accept NEP 52 (Python API cleanup for 2.0)
by Ralf Gommers 15 Sep '23

15 Sep '23

Hi all, A lot of work has been happening to implement NEP 52 ( https://numpy.org/neps/nep-0052-python-api-cleanup.html) over the past 1.5 months - mostly work by Mateusz Sokol, and review effort of Sebastian, Nathan and myself. The majority of API changes have been made. There's more to do of course and there are pending PRs for a good fraction of that. These two tracking issues cover a lot of ground and discussion around decision on individual APIs: - main namespace: https://github.com/numpy/numpy/issues/24306 - numpy.lib namespace: https://github.com/numpy/numpy/issues/24507 This PR with a migration guide will give a good sense of what has been removed or changed so far: https://github.com/numpy/numpy/pull/24693. In https://github.com/numpy/numpy/pull/24620 the NEP itself is being updated for changes that have been made. And it will mark the NEP as Accepted, which seems about time given that a lot of the work has already been merged. If there are no substantive objections within 7 days from this email, then the NEP will be accepted; see NEP 0 for more details. Cheers, Ralf

3 3

ANN: NumExpr 2.8.6 Released
by Robert McLeod 13 Sep '23

13 Sep '23

Hi everyone, NumExpr 2.8.6 is a release to deal with issues related to downstream `pandas` where the sanitization blacklist was hitting private variables used in their evaluate. In addition the sanitization was hitting on scientific notation. For those who do not wish to have sanitization on by default, it can be changed by setting an environment variable, `NUMEXPR_SANITIZE=0`. If you use `pandas` in your packages it is advisable you pin `numexpr >= 2.8.6` in your requirements. Project documentation is available at: http://numexpr.readthedocs.io/ Changes from 2.8.5 to 2.8.6 --------------------------- * The sanitization can be turned off by default by setting an environment variable, `set NUMEXPR_SANITIZE=0` * Improved behavior of the blacklist to avoid triggering on private variables and scientific notation numbers. What's Numexpr? --------------- Numexpr is a fast numerical expression evaluator for NumPy. With it, expressions that operate on arrays (like "3*a+4*b") are accelerated and use less memory than doing the same calculation in Python. It has multi-threaded capabilities, as well as support for Intel's MKL (Math Kernel Library), which allows an extremely fast evaluation of transcendental functions (sin, cos, tan, exp, log...) while squeezing the last drop of performance out of your multi-core processors. Look here for a some benchmarks of numexpr using MKL: https://github.com/pydata/numexpr/wiki/NumexprMKL Its only dependency is NumPy (MKL is optional), so it works well as an easy-to-deploy, easy-to-use, computational engine for projects that don't want to adopt other solutions requiring more heavy dependencies. Where I can find Numexpr? ------------------------- The project is hosted at GitHub in: https://github.com/pydata/numexpr You can get the packages from PyPI as well (but not for RC releases): http://pypi.python.org/pypi/numexpr Documentation is hosted at: http://numexpr.readthedocs.io/en/latest/ Share your experience --------------------- Let us know of any bugs, suggestions, gripes, kudos, etc. you may have. Enjoy data! -- Robert McLeod robbmcleod(a)gmail.com robert.mcleod(a)hitachi-hightech.com

2 1