Mailman 3 November 2023 - NumPy-Discussion

Meson - C extension - Finding numpy includes in virtual env
by Doug Turnbull 28 Nov '23

28 Nov '23

I am converting a Numpy C extension Python project from distutils to meson. I've been following along the meson-python instructions ( https://meson-python.readthedocs.io/en/latest/tutorials/introduction.html) I've installed meson / ninja (and everything else...) into a virtualenv (python -m venv venv). Running Python 3.11 Unfortunately the following command fails: incdir_numpy = run_command(py, ['-c', 'import numpy; print(numpy.get_include())'], capture: true, check: false, ).stdout().strip() Capturing stderr, gives the error: > ModuleNotFoundError: No module named 'numpy' does not exist. But If I hardcode incdir_numpy to my actual location, it works fine incdir_numpy = 'venv/lib/python3.11/site-packages/numpy/core/include' I've tried deactivating / activating the venv and rebuilding, no such luck. I feel like I'm missing something obvious about meson / numpy and virtual environments? My changes are here https://github.com/softwaredoug/np-sims/compare/meson?expand=1#diff-30d8f6b…

4 9

savetxt() has fmt='%.18e' as default, but fmt='%.16e' is always sufficient
by Jeppe Dakin 28 Nov '23

28 Nov '23

Double-precision numbers need at most 17 significant decimal digits to be serialised losslessly. Yet, savetxt() uses 19 by default, meaning that most files produced with savetxt() takes up about 9% more disk space than they need to, without any benefit. I have described the problem more detailed on Stackoverflow: https://stackoverflow.com/questions/77535380/minimum-number-of-digits-for-e… Is there any reason behind the default choice of savetxt(..., fmt='%.18e')? If not, why not reduce it to savetxt(..., fmt='%.16e')?

3 3

Enhancement: np.convolve(..., mode="normalized")
by filip.dominec＠gmail.com 23 Nov '23

23 Nov '23

Convolution is often used for smoothing noisy data; a typical use will keep the 'same' length of data and may look like this: > convol = 2**-np.linspace(-2,2,100)**2; > y2 = np.convolve(y,convol/np.sum(convol), mode='same') ## simple smoothing > ax.plot(x, y2, label="simple smoothing", color='g') However, when the smoothed curve has some nonzero background value at its edges, this convolution mode internally pads it with zeros, resulting in the curve looking like moustache of Frank Zappa. I made an example plot illustrating this here: https://www.fzu.cz/~dominecf/misc/numpy_smoothing_example.png. Such a result, i.e. the green curve, is not publication ready! 1) One way around is to np.pad(..., mode='edge'), then convolve & properly truncate the curve back to its original length. This is not a correct approach, however, as it makes the curve edges smooth, but their actual shape becomes very sensitive to the pointwise noise. Moreover, it artificially removes the curve's slope at its edges. 2) Another way around is to generate an auxiliary "Zappa's moustache" by applying the same convolution to a fresh array of np.ones_like(y). Then one can normalize the convolved curve by this auxiliary function. This has only one downside of keeping the curve more noisy at its edges, which however appears more scientifically honest to me - at the dataset edges one simply has less means to filter out noise. > convol = 2**-np.linspace(-2,2,100)**2; > norm = np.convolve(np.ones_like(y),convol/np.sum(convol), mode='same') > y2 = np.convolve(y,convol/np.sum(convol), mode='same')/norm ## simple smoothing > ax.plot(x, y2, label="approach 2", color='k') In my experimental work, I am missing this behaviour of np.convolve in a single function. I suggest this option should be accessible numpy under the mode="normalized" option. (Actually I believe this could have been set as default, but this would break compatibility.) Thanks for consideration, Filip

3 2

pickling dtype values
by Sebastien Binet 21 Nov '23

21 Nov '23

hi there, I have written a Go package[1] that can read/write simple arrays in the numpy file format [2]. when I wrote it, it was for simple interoperability use cases, but now people would like to be able to read back ragged-arrays[3]. unless I am mistaken, this means I need to interpret pieces of pickled data (`ndarray`, `multiarray` and `dtype`). so I am trying to understand how to unpickle `dtype` values that have been pickled: ```python import numpy as np import pickle import pickletools as pt pt.dis(pickle.dumps(np.dtype("int32"), protocol=4), annotate=True) ``` gives: ``` 0: \x80 PROTO 4 Protocol version indicator. 2: \x95 FRAME 55 Indicate the beginning of a new frame. 11: \x8c SHORT_BINUNICODE 'numpy' Push a Python Unicode string object. 18: \x94 MEMOIZE (as 0) Store the stack top into the memo. The stack is not popped. 19: \x8c SHORT_BINUNICODE 'dtype' Push a Python Unicode string object. 26: \x94 MEMOIZE (as 1) Store the stack top into the memo. The stack is not popped. 27: \x93 STACK_GLOBAL Push a global object (module.attr) on the stack. 28: \x94 MEMOIZE (as 2) Store the stack top into the memo. The stack is not popped. 29: \x8c SHORT_BINUNICODE 'i4' Push a Python Unicode string object. 33: \x94 MEMOIZE (as 3) Store the stack top into the memo. The stack is not popped. 34: \x89 NEWFALSE Push False onto the stack. 35: \x88 NEWTRUE Push True onto the stack. 36: \x87 TUPLE3 Build a three-tuple out of the top three items on the stack. 37: \x94 MEMOIZE (as 4) Store the stack top into the memo. The stack is not popped. 38: R REDUCE Push an object built from a callable and an argument tuple. 39: \x94 MEMOIZE (as 5) Store the stack top into the memo. The stack is not popped. 40: ( MARK Push markobject onto the stack. 41: K BININT1 3 Push a one-byte unsigned integer. 43: \x8c SHORT_BINUNICODE '<' Push a Python Unicode string object. 46: \x94 MEMOIZE (as 6) Store the stack top into the memo. The stack is not popped. 47: N NONE Push None on the stack. 48: N NONE Push None on the stack. 49: N NONE Push None on the stack. 50: J BININT -1 Push a four-byte signed integer. 55: J BININT -1 Push a four-byte signed integer. 60: K BININT1 0 Push a one-byte unsigned integer. 62: t TUPLE (MARK at 40) Build a tuple out of the topmost stack slice, after markobject. 63: \x94 MEMOIZE (as 7) Store the stack top into the memo. The stack is not popped. 64: b BUILD Finish building an object, via __setstate__ or dict update. 65: . STOP Stop the unpickling machine. highest protocol among opcodes = 4 ``` I have tried to find the usual `__reduce__` and `__setstate__` methods to understand what are the various arguments, to no avail. so, in : ```python >>> np.dtype("int32").__reduce__()[1] ('i4', False, True) >>> np.dtype("int32").__reduce__()[2] (3, '<', None, None, None, -1, -1, 0) ``` what are the meaning of the various arguments ? thanks in advance, sebastien. [1] https://github.com/sbinet/npyio [2] https://numpy.org/neps/nep-0001-npy-format.html [3] https://github.com/sbinet/npyio/issues/20

2 2

Making np.object_ generic for better object array type hinting
by christopher.ariza＠gmail.com 21 Nov '23

21 Nov '23

Greetings! Could np.object_ be made generic? For example, if I have an object array of two-element integer tuples, it would be ideal if I could type hint the array with np.dtype[np.object_[tuple[int, int]]]; this would allow us to distinguish between object arrays by their contained types. (I.e., np.dtype[np.object_[tuple[int, int]]] is not the same as np.dtype[np.object_[datetime.date]].) Context: I am the lead developer of StaticFrame (https://github.com/static-frame/static-frame), an alternative DataFrame library built on an immutable data model. StaticFrame has recently made DataFrames (and other containers) generic (https://towardsdatascience.com/type-hinting-dataframes-for-static-analysis-…). When specifying columnar types, we use NumPy generic types. While this provides a great deal of flexibility for most types, when an object array is involved, we cannot express anything about what is contained in the object array. Making np.object_ generic would solve this issue. I am happy to create an issue and explore a PR if this seems like a good enhancement.

1 0

*New Time* Next Documentation team meeting
by Mukulika Pahari 21 Nov '23

21 Nov '23

Hi all, sorry for the late notice. Our next Documentation Team meeting will happen on *Monday, October 9* at *11PM UTC*. If this time slot is inconvenient for you to join, please let me know in the replies or Slack and we will try to add another time slot. All are welcome - you don't need to already be a contributor to join. If you have questions or are curious about what we're doing, we'll be happy to meet you! If you wish to join on Zoom, use this (updated) link: https://numfocus-org.zoom.us/j/85016474448?pwd=TWEvaWJ1SklyVEpwNXUrcHV1YmFJQ... Here's the permanent hackmd document with the meeting notes (still being updated): https://hackmd.io/oB_boakvRqKR-_2jRV-Qjg Hope to see you around! Best wishes, Mukulika

2 3

ANN: SciPy 1.11.4
by Tyler Reddy 19 Nov '23

19 Nov '23

Hi all, On behalf of the SciPy development team, I'm pleased to announce the release of SciPy 1.11.4, which is a bug fix release. Sources and binary wheels can be found at: https://pypi.org/project/scipy/ and at: https://github.com/scipy/scipy/releases/tag/v1.11.4 One of a few ways to install this release with pip: pip install scipy==1.11.4 ====================== SciPy 1.11.4 Release Notes ====================== SciPy 1.11.4 is a bug-fix release with no new features compared to 1.11.3. Authors ====== * Name (commits) * Jake Bowhay (2) * Ralf Gommers (4) * Julien Jerphanion (2) * Nikolay Mayorov (2) * Melissa Weber Mendonça (1) * Tirth Patel (1) * Tyler Reddy (22) * Dan Schult (3) * Nicolas Vetsch (1) + A total of 9 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 1.11.4 -------------------------------- * `#19189 <https://github.com/scipy/scipy/issues/19189>`__: Contradiction in \`pyproject.toml\` requirements? * `#19228 <https://github.com/scipy/scipy/issues/19228>`__: Doc build fails with Python 3.11 * `#19245 <https://github.com/scipy/scipy/issues/19245>`__: BUG: upcasting of indices dtype from DIA to COO/CSR/BSR arrays * `#19351 <https://github.com/scipy/scipy/issues/19351>`__: BUG: Regression in 1.11.3 can still fail for \`optimize.least_squares\`... * `#19357 <https://github.com/scipy/scipy/issues/19357>`__: BUG: build failure with Xcode 15 linker * `#19359 <https://github.com/scipy/scipy/issues/19359>`__: BUG: DiscreteAliasUrn construction fails with UNURANError for... * `#19387 <https://github.com/scipy/scipy/issues/19387>`__: BUG: problem importing libgfortran.5.dylib on macOS Sonoma * `#19403 <https://github.com/scipy/scipy/issues/19403>`__: BUG: scipy.sparse.lil_matrix division by complex number leads... * `#19437 <https://github.com/scipy/scipy/issues/19437>`__: BUG: can't install scipy on mac m1 with poetry due to incompatible... * `#19500 <https://github.com/scipy/scipy/issues/19500>`__: DOC: doc build failing * `#19513 <https://github.com/scipy/scipy/issues/19513>`__: BUG: Python version constraints in releases causes issues for... Pull requests for 1.11.4 -------------------------------- * `#19230 <https://github.com/scipy/scipy/pull/19230>`__: DOC, MAINT: workaround for py311 docs * `#19307 <https://github.com/scipy/scipy/pull/19307>`__: set idx_dtype in sparse dia_array.tocoo * `#19316 <https://github.com/scipy/scipy/pull/19316>`__: MAINT: Prep 1.11.4 * `#19320 <https://github.com/scipy/scipy/pull/19320>`__: BLD: fix up version parsing issue in cythonize.py for setup.py... * `#19329 <https://github.com/scipy/scipy/pull/19329>`__: DOC: stats.chisquare: result object contains attribute 'statistic' * `#19335 <https://github.com/scipy/scipy/pull/19335>`__: BUG: fix pow method for sparrays with power zero * `#19364 <https://github.com/scipy/scipy/pull/19364>`__: MAINT, BUG: stats: update the UNU.RAN submodule with DAU fix * `#19379 <https://github.com/scipy/scipy/pull/19379>`__: BUG: Restore the original behavior of 'trf' from least_squares... * `#19400 <https://github.com/scipy/scipy/pull/19400>`__: BLD: use classic linker on macOS 14 (Sonoma), the new linker... * `#19408 <https://github.com/scipy/scipy/pull/19408>`__: BUG: Fix typecasting problem in scipy.sparse.lil_matrix truediv * `#19504 <https://github.com/scipy/scipy/pull/19504>`__: DOC, MAINT: Bump CircleCI Python version to 3.11 * `#19517 <https://github.com/scipy/scipy/pull/19517>`__: MAINT, REL: unpin Python 1.11.x branch * `#19550 <https://github.com/scipy/scipy/pull/19550>`__: MAINT, BLD: poetry loongarch shims Checksums ========= MD5 ~~~ 3324298ede271261e0b74df81ca2aa3a Changelog e8b456252ac318dd9f853bf8d8c45029 README.txt b99a5cf10c6a76d2d87ce59d4f87b811 scipy-1.11.4-cp310-cp310-macosx_10_9_x86_64.whl 67d95164f2366ff0efdff3dd25313e8e scipy-1.11.4-cp310-cp310-macosx_12_0_arm64.whl 198cdbc80b37bc4eb262b4478f8bab52 scipy-1.11.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl e61c5838aa069eddaa1e71d8d9d85509 scipy-1.11.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 517f82f0b581aa34630158591377b127 scipy-1.11.4-cp310-cp310-musllinux_1_1_x86_64.whl 9b9fa69812be52fc1da643d4dcab2abf scipy-1.11.4-cp310-cp310-win_amd64.whl 955a224e94e29e03d0c3cfda7147d756 scipy-1.11.4-cp311-cp311-macosx_10_9_x86_64.whl c6d0bfc7b2e57822de9249f468e234e6 scipy-1.11.4-cp311-cp311-macosx_12_0_arm64.whl 617c587f9442fa26fb36876efefdb311 scipy-1.11.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 00fe7507f91e11730ed6f48d5881fd74 scipy-1.11.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 340b520348315d183455e7b79fc0c1d3 scipy-1.11.4-cp311-cp311-musllinux_1_1_x86_64.whl fc3aade39fda56dbcd07106bd4b26c84 scipy-1.11.4-cp311-cp311-win_amd64.whl 6b2eeebed81a8aba66d703d785b562e1 scipy-1.11.4-cp312-cp312-macosx_10_9_x86_64.whl f61c07733fcd8747fa3212295bf639e7 scipy-1.11.4-cp312-cp312-macosx_12_0_arm64.whl b8526ce9d78ee7629fa1c00733233a01 scipy-1.11.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 13794337c432999fa37f8cf0280b08f8 scipy-1.11.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 30e84671d615659f0041a47f6aee5105 scipy-1.11.4-cp312-cp312-musllinux_1_1_x86_64.whl eae94466eb0c36b61db723b5cd835fec scipy-1.11.4-cp312-cp312-win_amd64.whl 616e80dff4d6590aba7096ad1dcccb31 scipy-1.11.4-cp39-cp39-macosx_10_9_x86_64.whl 750ff6a0490796885a159a0aa338072b scipy-1.11.4-cp39-cp39-macosx_12_0_arm64.whl 96c002ce95d2cc8e16c81bfe9746198b scipy-1.11.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 07695f6f859e93e736008d592cf2080e scipy-1.11.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 700b898a5a373cefdc76954282c66bab scipy-1.11.4-cp39-cp39-musllinux_1_1_x86_64.whl af9246a44e37b2ecfb7370eeb55bd84c scipy-1.11.4-cp39-cp39-win_amd64.whl SHA256 ~~~~~~ cd3df3413447fa60638bd98eb7b3066d93855197ce67210b52f8cf4093b6242b Changelog 753ed3b3ee750a8d52535c860aac29f8bcfb0183c6eafc925f7b41c68921a68b README.txt bc9a714581f561af0848e6b69947fda0614915f072dfd14142ed1bfe1b806710 scipy-1.11.4-cp310-cp310-macosx_10_9_x86_64.whl cf00bd2b1b0211888d4dc75656c0412213a8b25e80d73898083f402b50f47e41 scipy-1.11.4-cp310-cp310-macosx_12_0_arm64.whl b9999c008ccf00e8fbcce1236f85ade5c569d13144f77a1946bef8863e8f6eb4 scipy-1.11.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 933baf588daa8dc9a92c20a0be32f56d43faf3d1a60ab11b3f08c356430f6e56 scipy-1.11.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 8fce70f39076a5aa62e92e69a7f62349f9574d8405c0a5de6ed3ef72de07f446 scipy-1.11.4-cp310-cp310-musllinux_1_1_x86_64.whl 6550466fbeec7453d7465e74d4f4b19f905642c89a7525571ee91dd7adabb5a3 scipy-1.11.4-cp310-cp310-win_amd64.whl f313b39a7e94f296025e3cffc2c567618174c0b1dde173960cf23808f9fae4be scipy-1.11.4-cp311-cp311-macosx_10_9_x86_64.whl 1b7c3dca977f30a739e0409fb001056484661cb2541a01aba0bb0029f7b68db8 scipy-1.11.4-cp311-cp311-macosx_12_0_arm64.whl 00150c5eae7b610c32589dda259eacc7c4f1665aedf25d921907f4d08a951b1c scipy-1.11.4-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 530f9ad26440e85766509dbf78edcfe13ffd0ab7fec2560ee5c36ff74d6269ff scipy-1.11.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 5e347b14fe01003d3b78e196e84bd3f48ffe4c8a7b8a1afbcb8f5505cb710993 scipy-1.11.4-cp311-cp311-musllinux_1_1_x86_64.whl acf8ed278cc03f5aff035e69cb511741e0418681d25fbbb86ca65429c4f4d9cd scipy-1.11.4-cp311-cp311-win_amd64.whl 028eccd22e654b3ea01ee63705681ee79933652b2d8f873e7949898dda6d11b6 scipy-1.11.4-cp312-cp312-macosx_10_9_x86_64.whl 2c6ff6ef9cc27f9b3db93a6f8b38f97387e6e0591600369a297a50a8e96e835d scipy-1.11.4-cp312-cp312-macosx_12_0_arm64.whl b030c6674b9230d37c5c60ab456e2cf12f6784596d15ce8da9365e70896effc4 scipy-1.11.4-cp312-cp312-manylinux_2_17_aarch64.manylinux2014_aarch64.whl ad669df80528aeca5f557712102538f4f37e503f0c5b9541655016dd0932ca79 scipy-1.11.4-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl ce7fff2e23ab2cc81ff452a9444c215c28e6305f396b2ba88343a567feec9660 scipy-1.11.4-cp312-cp312-musllinux_1_1_x86_64.whl 36750b7733d960d7994888f0d148d31ea3017ac15eef664194b4ef68d36a4a97 scipy-1.11.4-cp312-cp312-win_amd64.whl 6e619aba2df228a9b34718efb023966da781e89dd3d21637b27f2e54db0410d7 scipy-1.11.4-cp39-cp39-macosx_10_9_x86_64.whl f3cd9e7b3c2c1ec26364856f9fbe78695fe631150f94cd1c22228456404cf1ec scipy-1.11.4-cp39-cp39-macosx_12_0_arm64.whl d10e45a6c50211fe256da61a11c34927c68f277e03138777bdebedd933712fea scipy-1.11.4-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 91af76a68eeae0064887a48e25c4e616fa519fa0d38602eda7e0f97d65d57937 scipy-1.11.4-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 6df1468153a31cf55ed5ed39647279beb9cfb5d3f84369453b49e4b8502394fd scipy-1.11.4-cp39-cp39-musllinux_1_1_x86_64.whl ee410e6de8f88fd5cf6eadd73c135020bfbbbdfcd0f6162c36a7638a1ea8cc65 scipy-1.11.4-cp39-cp39-win_amd64.whl

1 0

How to sample unique vectors
by Stefan van der Walt 18 Nov '23

18 Nov '23

Hi all, I am trying to sample k N-dimensional vectors from a uniform distribution without replacement. It seems like this should be straightforward, but I can't seem to pin it down. Specifically, I am trying to get random indices in an d0 x d1 x d2.. x dN-1 array. I thought about sneaking in a structured dtype into `rng.integers`, but of course that doesn't work. If we had a string sampler, I could sample k unique words (consisting of digits), and convert them to indices. I could over-sample and filter out the non-unique indices. Or iteratively draw blocks of samples until I've built up my k unique indices. The most straightforward solution would be to flatten indices, and to sample from those. The integers get large quickly, though. The rng.integers docstring suggests that it can handle object arrays for very large integers: > When using broadcasting with uint64 dtypes, the maximum value (2**64) > cannot be represented as a standard integer type. > The high array (or low if high is None) must have object dtype, e.g., array([2**64]). But, that doesn't work: In [35]: rng.integers(np.array([2**64], dtype=object)) ValueError: high is out of bounds for int64 Is there an elegant way to handle this problem? Best regards, Stéfan

4 10

Assessment of the difficulty in porting CPU architecture for numpy
by xuanbao 17 Nov '23

17 Nov '23

Hello everyone! I am working on implementing a tool to assess the complexity of CPU architecture porting. It primarily focuses on RISC-V architecture porting. In fact, the tool may have an average estimate of various architecture porting efforts.My focus is on the overall workload and difficulty of transplantation in the past and future,even if a project has already been ported.As part of my dataset, I have collected the **numpy** project. **I would like to gather community opinions to support my assessment. I appreciate your help and response!** Based on scanning tools, the porting complexity is determined to be moderate leaning towards simple, with a moderate amount of code related to the CPU architecture in the project. Is this assessment accurate?Do you often have any opinions on personnel allocation and consumption time？ I look forward to your help and response.

2 1

Removing `np.f2py.compile`
by Rohit Goswami 16 Nov '23

16 Nov '23

Hello. As part of the spring cleanup for NumPy 2.0 I'm soliciting opinions on the removal of the numpy.f2py.compile() function (documented here (https://link.getmailspring.com/link/937E4C5C-47D5-40D9-89BB-8FC04B19D105@ge…. The associated issue is gh-25122 (https://link.getmailspring.com/link/937E4C5C-47D5-40D9-89BB-8FC04B19D105@ge…). Here are the main points against it: Hasn't been updated (beyond compatibility) in 18 years Is a (not great) wrapper around subprocess.run Leaks memory (noted in this test case (https://link.getmailspring.com/link/937E4C5C-47D5-40D9-89BB-8FC04B19D105@ge… As for why now.. After the meson transition, the -c workflow is mostly to generate a skeleton Many of the older -c flags are not coming back (since parsing build arguments and forwarding them to meson is basically resurrecting distutils ) It still generates a .so file in the working directory which is weird at best .. and likely to be very broken very often in practice The expectation that -c based builds work exactly the same post meson is not correct, and removing this function would also make downstream users come check the new documentation (https://link.getmailspring.com/link/937E4C5C-47D5-40D9-89BB-8FC04B19D105@ge…) Projects should generate their own subprocess.run wrappers for the post meson f2py , so they can set the environment variables to interact with meson or provide native files... All said, it also seems to be mostly (https://link.getmailspring.com/link/937E4C5C-47D5-40D9-89BB-8FC04B19D105@ge…) (but not completely) unused. Happy to hear thoughts against / for this, though the default assumption is going to be remove it :) --- Rohit

1 0