Mailman 3 February 2023 - NumPy-Discussion

non normalised eigenvectors
by louis.petingi＠csi.cuny.edu 26 Feb '23

26 Feb '23

Dear all, I am not an expert in NumPy but my undergraduate student is having some issues with the way Numpy returns the normalized eigenvectors corresponding to the eigenvalues. We do understand that an eigenvector is divided by the norm to get the unit eigenvectors, however we do need the original vectors for the purpose of my research. This has been a really frustrated experience as NumPy returns the normalized vectors as a default. I appreciate any suggestions of how to go about this issue. This seems to be a outstanding issue from people using Numpy. Thanks LP

5 12

ANN: A new version (0.14.0) of sparse has been released.
by Hameer Abbasi 24 Feb '23

24 Feb '23

I’d like to announce the release of PyData/Sparse 0.14.0. PyData/Sparse is available on conda-forge and PyPI. PyData/Sparse is a package that provides sparse arrays conforming to the NumPy API, along with a subset of NumPy operations on sparse arrays. The highlights of this release include faster indexing, sampling and Sparse einsum support. The changelog from the last release is available at: https://sparse.pydata.org/en/0.14,0/changelog.html The code is available at: https://github.com/pydata/sparse/ And the latest documentation is available at https://sparse.pydata.org/

1 0

__array_function__ for assignment?
by Mark Harfouche 24 Feb '23

24 Feb '23

I've been working on creating a few lazy arrays for our applications. For large data volumes, it is often critical to avoid the use of bounce buffers. Essentially, I'm looking to see if there exists a function like `np.take` that would take: 1. A slice for the input array. 2. A slice for the output array. 3. Assign the slice of the input array to the output array. I could "slice" and assign, Pseudocode: s = MyFancyArray((3, 1024, 1024, 1024)) # A very big array a = np.zeros(s.shape, dtype=s.dtype) a[...] = s # I want this operation to be crafted carefully But that seems to: 1. Instantiate the full array `s` with `__array__`. 2. slice into s without passing me a reference to the array `a`. Is there something like `np.take` or `np.copyto` that is more general?

1 0

ANN: SciPy 1.10.1
by Tyler Reddy 20 Feb '23

20 Feb '23

Hi all, On behalf of the SciPy development team, I'm pleased to announce the release of SciPy 1.10.1. Sources and binary wheels can be found at: https://pypi.org/project/scipy/ and at: https://github.com/scipy/scipy/releases/tag/v1.10.1 One of a few ways to install this release with pip: pip install scipy==1.10.1 ====================== SciPy 1.10.1 Release Notes ====================== SciPy 1.10.1 is a bug-fix release with no new features compared to 1.10.0. Authors ======= * Name (commits) * alice (1) + * Matt Borland (2) + * Evgeni Burovski (2) * CJ Carey (1) * Ralf Gommers (9) * Brett Graham (1) + * Matt Haberland (5) * Alex Herbert (1) + * Ganesh Kathiresan (2) + * Rishi Kulkarni (1) + * Loïc Estève (1) * Michał Górny (1) + * Jarrod Millman (1) * Andrew Nelson (4) * Tyler Reddy (50) * Pamphile Roy (2) * Eli Schwartz (2) * Tomer Sery (1) + * Kai Striega (1) * Jacopo Tissino (1) + * windows-server-2003 (1) A total of 21 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 1.10.1 -------------------------------- * `#14980 <https://github.com/scipy/scipy/issues/14980>`__: BUG: Johnson's algorithm fails without negative cycles * `#17670 <https://github.com/scipy/scipy/issues/17670>`__: Failed to install on Raspberry Pi (ARM) 32bit in 3.11.1 * `#17715 <https://github.com/scipy/scipy/issues/17715>`__: scipy.stats.bootstrap broke with statistic returning multiple... * `#17716 <https://github.com/scipy/scipy/issues/17716>`__: BUG: interpolate.interpn fails with read only input * `#17718 <https://github.com/scipy/scipy/issues/17718>`__: BUG: RegularGridInterpolator 2D mixed precision crashes * `#17727 <https://github.com/scipy/scipy/issues/17727>`__: BUG: RegularGridInterpolator does not work on non-native byteorder... * `#17736 <https://github.com/scipy/scipy/issues/17736>`__: BUG: SciPy requires OpenBLAS even when building against a different... * `#17775 <https://github.com/scipy/scipy/issues/17775>`__: BUG: Asymptotic computation of ksone.sf has intermediate overflow * `#17782 <https://github.com/scipy/scipy/issues/17782>`__: BUG: Segfault in scipy.sparse.csgraph.shortest_path() with v1.10.0 * `#17795 <https://github.com/scipy/scipy/issues/17795>`__: BUG: stats.pearsonr one-sided hypothesis yields incorrect p-value... * `#17801 <https://github.com/scipy/scipy/issues/17801>`__: BUG: stats.powerlaw.fit: raises OverflowError * `#17808 <https://github.com/scipy/scipy/issues/17808>`__: BUG: name of cython executable is hardcoded in _build_utils/cythoner.py * `#17811 <https://github.com/scipy/scipy/issues/17811>`__: CI job with numpy nightly build failing on missing \`_ArrayFunctionDispatcher.__code__\` * `#17839 <https://github.com/scipy/scipy/issues/17839>`__: BUG: 1.10.0 tests fail on i386 and other less common arches * `#17896 <https://github.com/scipy/scipy/issues/17896>`__: DOC: publicly expose \`multivariate_normal\` attributes \`mean\`... * `#17934 <https://github.com/scipy/scipy/issues/17934>`__: BUG: meson \`__config__\` generation - truncated unicode characters * `#17938 <https://github.com/scipy/scipy/issues/17938>`__: BUG: \`scipy.stats.qmc.LatinHypercube\` with \`optimization="random-cd"\`... Pull requests for 1.10.1 ------------------------------- * `#17712 <https://github.com/scipy/scipy/pull/17712>`__: REL, MAINT: prepare for 1.10.1 * `#17717 <https://github.com/scipy/scipy/pull/17717>`__: BUG: allow readonly input to interpolate.interpn * `#17721 <https://github.com/scipy/scipy/pull/17721>`__: MAINT: update \`meson-python\` upper bound to <0.13.0 * `#17726 <https://github.com/scipy/scipy/pull/17726>`__: BUG: interpolate/RGI: upcast float32 to float64 * `#17735 <https://github.com/scipy/scipy/pull/17735>`__: MAINT: stats.bootstrap: fix BCa with vector-valued statistics * `#17743 <https://github.com/scipy/scipy/pull/17743>`__: DOC: improve the docs on using BLAS/LAPACK libraries with Meson * `#17777 <https://github.com/scipy/scipy/pull/17777>`__: BLD: link to libatomic if necessary * `#17783 <https://github.com/scipy/scipy/pull/17783>`__: BUG: Correct intermediate overflow in KS one asymptotic in SciPy.stats * `#17790 <https://github.com/scipy/scipy/pull/17790>`__: BUG: signal: fix check_malloc extern declaration type * `#17797 <https://github.com/scipy/scipy/pull/17797>`__: MAINT: stats.pearsonr: correct p-value with negative correlation... * `#17800 <https://github.com/scipy/scipy/pull/17800>`__: [sparse.csgraph] Fix a bug in dijkstra and johnson algorithm * `#17803 <https://github.com/scipy/scipy/pull/17803>`__: MAINT: add missing \`__init__.py\` in test folder * `#17806 <https://github.com/scipy/scipy/pull/17806>`__: MAINT: stats.powerlaw.fit: fix overflow when np.min(data)==0 * `#17810 <https://github.com/scipy/scipy/pull/17810>`__: BLD: use Meson's found cython instead of a wrapper script * `#17831 <https://github.com/scipy/scipy/pull/17831>`__: MAINT, CI: GHA MacOS setup.py update * `#17850 <https://github.com/scipy/scipy/pull/17850>`__: MAINT: remove use of \`__code__\` in \`scipy.integrate\` * `#17854 <https://github.com/scipy/scipy/pull/17854>`__: TST: mark test for \`stats.kde.marginal\` as xslow * `#17855 <https://github.com/scipy/scipy/pull/17855>`__: BUG: Fix handling of \`powm1\` overflow errors * `#17859 <https://github.com/scipy/scipy/pull/17859>`__: TST: fix test failures on i386, s390x, ppc64, riscv64 (Debian) * `#17862 <https://github.com/scipy/scipy/pull/17862>`__: BLD: Meson \`__config__\` generation * `#17863 <https://github.com/scipy/scipy/pull/17863>`__: BUG: fix Johnson's algorithm * `#17872 <https://github.com/scipy/scipy/pull/17872>`__: BUG: fix powm1 overflow handling * `#17904 <https://github.com/scipy/scipy/pull/17904>`__: ENH: \`multivariate_normal_frozen\`: restore \`cov\` attribute * `#17910 <https://github.com/scipy/scipy/pull/17910>`__: CI: use nightly numpy musllinux_x86_64 wheel * `#17931 <https://github.com/scipy/scipy/pull/17931>`__: TST: test_location_scale proper 32bit Linux skip * `#17932 <https://github.com/scipy/scipy/pull/17932>`__: TST: 32-bit tol for test_pdist_jensenshannon_iris * `#17936 <https://github.com/scipy/scipy/pull/17936>`__: BUG: Use raw strings for paths in \`__config__.py.in\` * `#17940 <https://github.com/scipy/scipy/pull/17940>`__: BUG: \`rng_integers\` in \`_random_cd\` now samples on a closed... * `#17942 <https://github.com/scipy/scipy/pull/17942>`__: BLD: update classifiers for Python 3.11 * `#17963 <https://github.com/scipy/scipy/pull/17963>`__: MAINT: backports/prep for SciPy 1.10.1 * `#17981 <https://github.com/scipy/scipy/pull/17981>`__: BLD: make sure macosx_x86_64 10.9 tags are being made on maintenance/1.10.x * `#17984 <https://github.com/scipy/scipy/pull/17984>`__: DOC: update link of the logo in the readme * `#17997 <https://github.com/scipy/scipy/pull/17997>`__: BUG: at least one entry from trial should be used in exponential... Checksums ========= MD5 ~~~ c9f5d246ec9ebbda9ed600d751cc427e Changelog b327ecc51bffee2c319687daaf87e127 README.txt 038b40489075536015cdfdd5b3448287 scipy-1.10.1-cp310-cp310-macosx_10_9_x86_64.whl 3666cb6db6e2205214f3b5f743f32162 scipy-1.10.1-cp310-cp310-macosx_12_0_arm64.whl e3f6a817ff0a1d4ddf8c29f017930808 scipy-1.10.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl c529f1d6ff9d39ac7b5027778ad6055c scipy-1.10.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl cfb29e468d60cf7ba815bb82f84fabbe scipy-1.10.1-cp310-cp310-win_amd64.whl 00a473cd2d539dc122c7b37951659464 scipy-1.10.1-cp311-cp311-macosx_10_9_x86_64.whl 414c3e59fd6c0bec7626ed781cc699cf scipy-1.10.1-cp311-cp311-macosx_12_0_arm64.whl 7ce38386a60faeff9945764950afd31b scipy-1.10.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl f18b34eb6c2d2fc6386448f79f983c5a scipy-1.10.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 0611e707fc9601a221fe45734ee4c4a6 scipy-1.10.1-cp311-cp311-win_amd64.whl 40eed028ec0849127dce3333c97c443a scipy-1.10.1-cp38-cp38-macosx_10_9_x86_64.whl 87c12a8bc1b238202d086462d34838a3 scipy-1.10.1-cp38-cp38-macosx_12_0_arm64.whl 1bb4ada4082c43e8312d11182f006d80 scipy-1.10.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 4e7919ee7b126817d566bb32d7f99e0e scipy-1.10.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl d1130585fa460fc8267ec3bff5e7c231 scipy-1.10.1-cp38-cp38-win_amd64.whl de8ee3fa36d67057ee1c0a58ede8072d scipy-1.10.1-cp39-cp39-macosx_10_9_x86_64.whl b767b83437990cadb8cebd5513fc17e0 scipy-1.10.1-cp39-cp39-macosx_12_0_arm64.whl c0547cc19e7f83fbb2cff56f1bc378de scipy-1.10.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl b3b49cb6c274ed58f79c60e9f98282e4 scipy-1.10.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl b503abd182b6f2047b5af060e2df4c91 scipy-1.10.1-cp39-cp39-win_amd64.whl SHA256 ~~~~~~ ad652dda7a636fe49c19ab22d6c41bbab969f8e75a2c202930dd6ff1628a2abf Changelog ef6df1bfa21a442a7fbb2015a63d6470f27d17faf3e09866ff3c5f3772abc92b README.txt e7354fd7527a4b0377ce55f286805b34e8c54b91be865bac273f527e1b839019 scipy-1.10.1-cp310-cp310-macosx_10_9_x86_64.whl 4b3f429188c66603a1a5c549fb414e4d3bdc2a24792e061ffbd607d3d75fd84e scipy-1.10.1-cp310-cp310-macosx_12_0_arm64.whl 1553b5dcddd64ba9a0d95355e63fe6c3fc303a8fd77c7bc91e77d61363f7433f scipy-1.10.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 4c0ff64b06b10e35215abce517252b375e580a6125fd5fdf6421b98efbefb2d2 scipy-1.10.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl fae8a7b898c42dffe3f7361c40d5952b6bf32d10c4569098d276b4c547905ee1 scipy-1.10.1-cp310-cp310-win_amd64.whl 0f1564ea217e82c1bbe75ddf7285ba0709ecd503f048cb1236ae9995f64217bd scipy-1.10.1-cp311-cp311-macosx_10_9_x86_64.whl d925fa1c81b772882aa55bcc10bf88324dadb66ff85d548c71515f6689c6dac5 scipy-1.10.1-cp311-cp311-macosx_12_0_arm64.whl aaea0a6be54462ec027de54fca511540980d1e9eea68b2d5c1dbfe084797be35 scipy-1.10.1-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 15a35c4242ec5f292c3dd364a7c71a61be87a3d4ddcc693372813c0b73c9af1d scipy-1.10.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 43b8e0bcb877faf0abfb613d51026cd5cc78918e9530e375727bf0625c82788f scipy-1.10.1-cp311-cp311-win_amd64.whl 5678f88c68ea866ed9ebe3a989091088553ba12c6090244fdae3e467b1139c35 scipy-1.10.1-cp38-cp38-macosx_10_9_x86_64.whl 39becb03541f9e58243f4197584286e339029e8908c46f7221abeea4b749fa88 scipy-1.10.1-cp38-cp38-macosx_12_0_arm64.whl bce5869c8d68cf383ce240e44c1d9ae7c06078a9396df68ce88a1230f93a30c1 scipy-1.10.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 07c3457ce0b3ad5124f98a86533106b643dd811dd61b548e78cf4c8786652f6f scipy-1.10.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 049a8bbf0ad95277ffba9b3b7d23e5369cc39e66406d60422c8cfef40ccc8415 scipy-1.10.1-cp38-cp38-win_amd64.whl cd9f1027ff30d90618914a64ca9b1a77a431159df0e2a195d8a9e8a04c78abf9 scipy-1.10.1-cp39-cp39-macosx_10_9_x86_64.whl 79c8e5a6c6ffaf3a2262ef1be1e108a035cf4f05c14df56057b64acc5bebffb6 scipy-1.10.1-cp39-cp39-macosx_12_0_arm64.whl 51af417a000d2dbe1ec6c372dfe688e041a7084da4fdd350aeb139bd3fb55353 scipy-1.10.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 1b4735d6c28aad3cdcf52117e0e91d6b39acd4272f3f5cd9907c24ee931ad601 scipy-1.10.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 7ff7f37b1bf4417baca958d254e8e2875d0cc23aaadbe65b3d5b3077b0eb23ea scipy-1.10.1-cp39-cp39-win_amd64.whl

1 0

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Introducing `np.types`
by Sebastian Berg 16 Feb '23

16 Feb '23

Hi all, I was wondering if we should introduce a new `np.types` namespace. The main reason is that we have the DType classes, that most users don't need to worry about. These mirror the scalar classes, but getting them is weird currently. I never wanted to put these in the top-level (because I feel they probably won't be used much day to day). That would be thing like: * np.types.IntDType, np.types.Int64DType (or maybe without dtype) * np.types.NumberDType (an abstract DType) * np.types.InexactDType * ... * np.types.DTypeMeta (the metaclass used, just to have it somewhere) Maybe there are some more types that we could use a public entrypoint for (e.g. the type used by array-function dispatched functions or `np.ufunc` could in principle move also). What do you think? I don't really have a good idea for an alternative but at some point not making these nicely public is not great... (I will note that the DType classes do get printed sometimes in error messages.) Cheers, Sebastian

5 10

NumPy Documentation Meeting - Monday 2/13 @ 8AM PDT
by Ross Barnowski 13 Feb '23

13 Feb '23

Hi all, We'll be meeting to discuss numpy documentation tomorrow at 8AM Pacific time. All are welcome! The zoom link and meeting agenda can be found here: https://hackmd.io/oB_boakvRqKR-_2jRV-Qjg ~Ross

1 0

dtype with no metadata
by ninou sf 09 Feb '23

09 Feb '23

To avoid problems with metadata in dtypes when copying hdf5 arrays to zarr or npy file or in other cases: - https://github.com/zarr-developers/zarr-python/issues/1152 - https://github.com/numpy/numpy/issues/15488 - https://github.com/numpy/numpy/issues/23169 i add a numpy tool function that creates a dtype from an original one but without any metada inside. https://github.com/ninousf/numpy/tree/dtype-without-metadata what do you think of it ? thanks

1 0

Standard for dtype string representation?
by Francesc Alted 08 Feb '23

08 Feb '23

Hi, We are in the process of using a standard representation of data types for the forthcoming version of N-dim arrays in C-Blosc2, and we want to use the NumPy string representation for that (see the end of https://github.com/Blosc/c-blosc2/blob/main/README_B2ND_METALAYER.rst). It might seem a bit strange to use the specification of a Python package for that, but provided its predominant role in data science, I don't think this should com as a surprise to anyone. There are some small gotchas though. For simple data types, the string representation is *apparently* fine. E.g.: In [16]: str(np.dtype("i8")) Out[16]: 'int64' However, as long as we try to represent the endianness of the type, we get: In [17]: str(np.dtype(">i8")) Out[17]: '>i8' So, it uses the short version of the representation. And the same happens with the structured types: In [22]: str(np.dtype("S1,i8")) Out[22]: "[('f0', 'S1'), ('f1', '<i8')]" Finally, the endianness seems to be represented arbitrarily. E.g. in: In [23]: str(np.dtype("S1")) Out[23]: '|S1' one can note the '|' char is prefixed to indicate endian independency, while it does not appear in the structured representation. While I know that there are some other representations for types in NumPy (e.g. numeric integers via dtype.num), I very much appreciate (and I suppose the same should go for other makers of numerical libraries) the expressiveness of str(dtype), specially when it comes to structured dtypes, if not were by the (relatively small) inconsistencies listed above. BTW, I have had a quick glance at the Python array API standard effort ( https://data-apis.org/array-api/latest/API_specification/data_types.html#da…), but it does not seem this is being addressed. For now, (and for the Python-Blosc2 wrapper) we are going in this direction: if dtype.kind == 'V': repr = str(dtype) else: repr = dtype.str Is there a way (or an ongoing effort) to express the variety of data types in NumPy that beats the above (which seems somewhat inconsistent to me)? Thanks! -- Francesc Alted

2 6

Introducing Arm Optimized Routines
by Chris Sidebottom 08 Feb '23

08 Feb '23

Hello, Here at Arm, we've been investigating how we can improve performance on AArch64. One way in which we can improve performance is by integrating some existing optimized routines (https://github.com/ARM-software/optimized-routines), similar to the SVML methods for AVX512 that are currently included as a git submodule. Our intent is to include the optimized routines repository as an additional submodule which we can then use to provide routines on AArch64 for ASIMD, SVE and beyond. Currently, we're targeting 4-ULP as this aligns with libmvec (https://sourceware.org/glibc/wiki/libmvec) and the SVML integration (https://github.com/numpy/numpy/pull/19478). This is alongside adding sufficient error handling to pass the Numpy test suite, meeting the test requirements highlighted in the SVML integration (https://github.com/numpy/numpy/pull/19478#issuecomment-893001722). We've already started curating the necessary functions, let us know if you have any feedback. Cheers, Chris IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.

3 4