Mailman 3 November 2021 - NumPy-Discussion

win32 wheels for Python 3.10
by Aivar Annamaa 14 Nov '21

14 Nov '21

Has it been decided to stop publishing win32 wheels for Python 3.10? Numpy 1.21.4 currently misses win32 wheel for Python 3.10 on PyPI. Is this temporary? I'm confused, because https://www.mail-archive.com/numpy-discussion@python.org/msg05512.html says: > We will continue to offer 32 wheels on Windows as it remains popular there.

3 3

Conversion from C-layout to Fortran-layout in Cython
by Ilhan Polat 12 Nov '21

12 Nov '21

I've asked this in Cython mailing list but probably I should also get some feedback here too. I have the following function defined in Cython and using flat memory pointers to hold n by n array data. cdef some_C_layout_func(double[:, :, ::1] Am) nogil: # ... cdef double * work1 = <double*>malloc(n*n*sizeof(double)) cdef double *work2 = <double*> malloc(n*n*sizeof(double)) # ... # Lots of C-layout operations here # ... dgetrs(<char*>'T', &n, &n, &work1[0], &n, &ipiv[0], &work2[0], &n, &info ) dcopy(&n2, &work2[0], &int1, &Am[0, 0, 0], &int1) free(...) Here, I have done everything in C layout with work1 and work2 but I have to convert work2 into Fortran layout to be able to solve AX = B. A can be transposed in Lapack internally via the flag 'T' so the only obstacle I have now is to shuffle work2 which holds B transpose in the eyes of Fortran since it is still in C layout. If I go naively and make loops to get one layout to the other that actually spoils all the speed benefits from this Cythonization due to cache misses. In fact 60% of the time is spent in that naive loop across the whole function. Same goes for the copy_fortran() of memoryviews. I have measured the regular NumPy np.asfortranarray() and the performance is quite good enough compared to the actual linear solve. Hence whatever it is doing underneath I would like to reach out and do the same possibly via the C-API. But my C knowledge basically failed me around this line https://github.com/numpy/numpy/blob/8dbd507fb6c854b362c26a0dd056cd04c9c10f2… I have found the SO post from https://stackoverflow.com/questions/45143381/making-a-memoryview-c-contiguo… but I am not sure if that is the canonical way to do it in newer Python versions. Can anyone show me how to go about it without interacting with Python objects? Best, ilhan

4 8

NumPy Community Meeting Wednesday (For those in the US: end of daylight saving means this is one hour earlier)
by Sebastian Berg 09 Nov '21

09 Nov '21

Hi all, There will be a NumPy Community meeting Wednesday November 10th at 16:30 UTC. Everyone is invited and encouraged to join in and edit the work-in-progress meeting topics and notes at: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian

1 0

branching NumPy 1.22.x
by Charles R Harris 08 Nov '21

08 Nov '21

Hi All, I am aiming to branch NumPy 1.22.x next weekend. If there are any PRs that you think need to be merged before the branch, please raise the issue. Chuck

2 1

Make the pickle default protocol 4.
by Charles R Harris 08 Nov '21

08 Nov '21

Hi All, I'd like to propose making the NumPy default pickle protocol 4, the same as the Python 3.8 default. That would have the advantage of supporting large pickles. The current default protocol is 2, last the default in Python 2.7. Thoughts? Chuck

3 2

Deprecate np.MachAr?
by bas van beek 08 Nov '21

08 Nov '21

Hi all, The subject of `MachAr` recently came up in https://github.com/numpy/numpy/pull/20132 and an earlier community meeting, more notably: it’s questionably role as public (and even private?) API. Ever since the introduction of hard-coded `finfo` parameters back in numpy/numpy#8504 there has been progressively less need for computing machine parameters during runtime, to the point where `MachAr` is effectively unused in the numpy codebase itself[1]. From a user-API point of view, the main difference between ` finfo` and `MachAr` these days is that the latter produces the same results roughly 4 orders of magnitude slower than the former… Are there any thoughts about deprecating it? Regards, Bas [1] It’s still used as fallback for `longdouble` types with unknown bytes-representations, rare cases that probably deserve their own pull request.

3 2

Percentile/Quantile "interpolation" refactor
by Sebastian Berg 08 Nov '21

08 Nov '21

Hi all, after a long time Abel has helped us and refactored the quantile and percentile functions' `interpolation` keyword. This was long overdue since NumPy implements three (the non-default) interpolation methods that appear to be very much non-standard. On the other hand, NumPy currently has no unbiased methods (i.e. population estimate). There are two main questions right now with respect to the API. First which names to use for the methods and second, how to deal with "outliers". The PR https://github.com/numpy/numpy/pull/19857#issuecomment-939852134 adds the methods and gives them (currently) the following names (sorted by the R methods) – the names will be used as string identifiers: 1. inverted cdf 2. averaged inverted cdf 3. closest observation 4. interpolated inverted cdf 5. hazen (name from wolfram) 6. weibull (name from wolfram) 7. linear (default! Better name deferred) 8. median unbiased 9. normal unbiased And additionally the four ones we currently have: * lower * higher * nearest * midpoint Number 5. and 6. are named "exclusive" and "inclusive" by Python in their `method` keyword argument. While I like the name `method=` and may want to move to it, I am not sure I like "inclusive" and "exclusive". The current plan was to defer the kwarg rename into a followup, although it should be discussed before the next release. The second main question is how to deal with outliers (this does not affect the default method 7, which finds the sample quantiles and not a population estimate). Wikipedia says this: Packages differ in how they estimate quantiles beyond the lowest and highest values in the sample, i.e. p < 1/N and p > (N − 1)/N. Choices include returning an error value, computing linear extrapolation, or assuming a constant value. The current choice is clipping (assuming a constant value), but this could be modified. Any feedback is appreciated! Otherwise, this will probably move forward in the current state for the next release. Cheers, Sebastian

8 8

ENH: add functionality NpyAppendArray to numpy.format
by Michael Siebert 07 Nov '21

07 Nov '21

Dear all, I'd like to add the NpyAppendArray functionality, compare https://github.com/xor2k/npy-append-array (15 Stars so far) and https://stackoverflow.com/a/64403144/2755796 (10 Upvotes so far) I have prepared a pull request and want to "test the waters" as suggested by the message I have received when creating the pull request. So what is NpyAppendArray about? I love the .npy file format. It is really great! I cannot appreciate the .npy capabilities mentioned in https://numpy.org/devdocs/reference/generated/numpy.lib.format.html enough, especially its simplicity. No comparison with the struggles we had with HDF5. However, there is one feature Numpy currently does not provide: a simple, efficient, easy-to-use and safe option to append to .npy (here the text I've used in the Github repository above): Appending to an array created by np.save might be possible under certain circumstances, since the .npy total header byte count is required to be evenly divisible by 64. Thus, there might be some spare space to grow the shape entry in the array descriptor. However, this is not guaranteed and might randomly fail. Initialize the array with NpyAppendArray(filename) directly so the header will be created with 64 byte of spare header space for growth. Will this be enough? It allows for up to 10^64 >= 2^212 array entries or data bits. Indeed, this is less than the number of atoms in the universe. However, fully populating such an array, due to limits imposed by quantum mechanics, would require more energy than would be needed to boil the oceans, compare https://hbfs.wordpress.com/2009/02/10/to-boil-the-oceans Therefore, a wide range of use cases should be coverable with this approach. Who could use that? I developed and use NpyAppendArray to efficiently create .npy arrays which are larger than the main memory and can be loaded by memory mapping later, e.g. for Deep Learning workflows. Another use case are binary log files, which could be created on low end embedded devices and later be processed without parsing, optionally again using memory maps. How does the code look like? Here some demo code of how this would look like in practice (taken from the test file): def test_NpyAppendArray(tmpdir): arr1 = np.array([[1,2],[3,4]]) arr2 = np.array([[1,2],[3,4],[5,6]]) fname = os.path.join(tmpdir, 'npaa.npy') with format.NpyAppendArray(fname) as npaa: npaa.append(arr1) npaa.append(arr2) npaa.append(arr2) arr = np.load(fname, mmap_mode="r") arr_ref = np.concatenate([arr1, arr2, arr2]) assert_array_equal(arr, arr_ref) Some more aspects: 1. appending efficiently only works along axis=0 at least for c order (probably different for Fortran order) 2. One could also add the 64 bytes of spare space right on np.save. However, I cannot really judge on how much of an issue that would be to the users of np.save and it is not really necessary since users who want to append to .npy files would create them with NpyAppendArray anyway. 3. Probably I have forgotten something here, some time has passed since the initial Github commit. So what do you think? Yes/No/Maybe? It would be really nice to get some feedback on the mailing list here! Although this might not be perfectly consistent with the protocol, I've created the pull request already, just to force myself to finish this up and I'm prepared to fail if there is no interest to get NpyAppendArray directly into numpy ;) Best from Berlin, Michael

1 1

ANN: SciPy 1.7.2
by Tyler Reddy 06 Nov '21

06 Nov '21

Hi all, On behalf of the SciPy development team I'm pleased to announce the release of SciPy 1.7.2, which is a bug fix release that includes wheels for Python 3.10 on many platforms. Many thanks to upstream developers in the ecosystem for their assistance in making this possible. Sources and binary wheels can be found at: https://pypi.org/project/scipy/ and at: https://github.com/scipy/scipy/releases/tag/v1.7.2 <https://github.com/scipy/scipy/releases/tag/v1.6.3> <https://github.com/scipy/scipy/releases/tag/v1.6.1> One of a few ways to install this release with pip: pip install scipy==1.7.2 ==================== SciPy 1.7.2 Release Notes ==================== SciPy 1.7.2 is a bug-fix release with no new features compared to 1.7.1. Notably, the release includes wheels for Python 3.10, and wheels are now built with a newer version of OpenBLAS, 0.3.17. Python 3.10 wheels are provided for MacOS x86_64 (thin, not universal2 or arm64 at this time), and Windows/Linux 64-bit. Many wheels are now built with newer versions of manylinux, which may require newer versions of pip. Authors ====== * Peter Bell * da-woods + * Isuru Fernando * Ralf Gommers * Matt Haberland * Nicholas McKibben * Ilhan Polat * Judah Rand + * Tyler Reddy * Pamphile Roy * Charles Harris * Matti Picus * Hugo van Kemenade * Jacob Vanderplas A total of 14 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 1.7.2 ------------------------------ * `#6019 <https://github.com/scipy/scipy/issues/6019>`__: minimize_scalar doesn't seem to honor "disp" option * `#14321 <https://github.com/scipy/scipy/issues/14321>`__: BUG: Indexing of CSR matrices with many rows is much slower than... * `#14465 <https://github.com/scipy/scipy/issues/14465>`__: BUG: n-d interpolation parameter provided to geometric_slerp * `#14599 <https://github.com/scipy/scipy/issues/14599>`__: SciPy 1.7 builds as zipped egg, ruining imports * `#14606 <https://github.com/scipy/scipy/issues/14606>`__: BUG: crash / core dump when calling scipy.stats.beta.ppf with... * `#14732 <https://github.com/scipy/scipy/issues/14732>`__: CI, TST: pre-release failures for scipy/interpolate/tests/test_rbfinterp.py * `#14802 <https://github.com/scipy/scipy/issues/14802>`__: CI: Azure Main coverage job failure * `#14829 <https://github.com/scipy/scipy/issues/14829>`__: macOS CI failing with \`ld: library not found for -lSystem\` * `#14887 <https://github.com/scipy/scipy/issues/14887>`__: BUG: scipy.stats.multivariate_normal.logpdf mutates some inputs Pull requests for 1.7.2 ------------------------------ * `#14207 <https://github.com/scipy/scipy/pull/14207>`__: DOC: stats: remove 'Methods' section from \`binomtest\` docstring... * `#14316 <https://github.com/scipy/scipy/pull/14316>`__: MAINT: Update \`openblas_support.py\` to support Apple Silicon * `#14323 <https://github.com/scipy/scipy/pull/14323>`__: BUG: Speed up sparse compressed indexing with very many rows * `#14333 <https://github.com/scipy/scipy/pull/14333>`__: MAINT: Use /usr/bin/linux32 so that sysconfig.get_platform()... * `#14478 <https://github.com/scipy/scipy/pull/14478>`__: BUG: geometric_slerp t ndim guard * `#14605 <https://github.com/scipy/scipy/pull/14605>`__: MAINT: Skip some interpolative decomposition tests * `#14616 <https://github.com/scipy/scipy/pull/14616>`__: REL: update build dependency versions in pyproject.toml for 1.7.2 * `#14618 <https://github.com/scipy/scipy/pull/14618>`__: FIX: raise RuntimeWarning when Boost evaluation_error is encountered * `#14672 <https://github.com/scipy/scipy/pull/14672>`__: BLD: add \`zip_safe=False\` to the \`setup()\` call * `#14791 <https://github.com/scipy/scipy/pull/14791>`__: MAINT: SciPy 1.7.2 prep/backports * `#14803 <https://github.com/scipy/scipy/pull/14803>`__: MAINT: disable include/source coverage warning. * `#14813 <https://github.com/scipy/scipy/pull/14813>`__: Added missing np.import_array() * `#14831 <https://github.com/scipy/scipy/pull/14831>`__: CI: Add stdlib to LD_LIBRARY_PATH * `#14893 <https://github.com/scipy/scipy/pull/14893>`__: BUG: Fix alignment errors due to relaxed stride checking * `#14897 <https://github.com/scipy/scipy/pull/14897>`__: BUG: avoid mutating inputs in multivariate distributions * `#14921 <https://github.com/scipy/scipy/pull/14921>`__: MAINT: "backport" 3.10 support * `#14937 <https://github.com/scipy/scipy/pull/14937>`__: MAINT: backports for 1.7.2, plus update Pythran min version to... * `#14938 <https://github.com/scipy/scipy/pull/14938>`__: TST: silence test failures on macOS for \`beta.ppf\` overflow Checksums ========= MD5 ~~~ 45913b1797379cc2e258b0d544276cc8 scipy-1.7.2-cp310-cp310-macosx_10_9_x86_64.whl 443a83b81b59b627aa63bdfaa66d8f62 scipy-1.7.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl cf5762eba17598002d2d310a5a6b6f9f scipy-1.7.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl c507b5599acb13b0b7d6e58619e8fb83 scipy-1.7.2-cp310-cp310-win_amd64.whl 4b1b6a83d39a3cb50d185a6808de7cb5 scipy-1.7.2-cp37-cp37m-macosx_10_9_x86_64.whl 8197788d9ed63a8c7859a1ec4419cf16 scipy-1.7.2-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl fa5e8f6f39feab419e2e86d4bb77980d scipy-1.7.2-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl da45607f5a6fdcb7fba852596f98e30b scipy-1.7.2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 3e98c25269ecf277dfd7614a11bef941 scipy-1.7.2-cp37-cp37m-win32.whl 19edbbc9da80a06ae20fcddca4e3ae53 scipy-1.7.2-cp37-cp37m-win_amd64.whl 49efccacf70e9c42d9e2e8b914753dba scipy-1.7.2-cp38-cp38-macosx_10_9_x86_64.whl 105b381e6eac333665a086012ed39d73 scipy-1.7.2-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl 9e217d4c7210170b9c29e11fb8ed7831 scipy-1.7.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl a424529ee29a85c295aac3e03753f6b0 scipy-1.7.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 59dd26cb1157fd2d70cfd260b1cae642 scipy-1.7.2-cp38-cp38-win32.whl 09090e0e2cd87cff88fec796c5dbf27a scipy-1.7.2-cp38-cp38-win_amd64.whl 98596dcc617bbbd4bd0532d1519d297f scipy-1.7.2-cp39-cp39-macosx_10_9_x86_64.whl e99fd9c32cf797b923add67878feae97 scipy-1.7.2-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl d4fa30d25f20b236d5d3e6c15d296c9b scipy-1.7.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 14d3114a82d556958480feebb971cd39 scipy-1.7.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 50ec716ebd1d31e195fe1658a986698e scipy-1.7.2-cp39-cp39-win32.whl bc4214e901ffa92c703b051f0c1d4f1b scipy-1.7.2-cp39-cp39-win_amd64.whl 9ba02f72fbdc922644ddb9ce5501c37e scipy-1.7.2.tar.gz d0fe9dfdada24195dd909d0f587140ec scipy-1.7.2.tar.xz c99d58b1069378ae4ccf9aecca57afb3 scipy-1.7.2.zip SHA256 ~~~~~~ 97eb573e361a73a553b915dc195c6f72a08249964b1a33f157f9659f3b6210d1 scipy-1.7.2-cp310-cp310-macosx_10_9_x86_64.whl 359b60a0cccd17723b9d5e329a5212a710e771a3ddde800e472fb93732756c46 scipy-1.7.2-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 82c5befebf54d799d77e5f0205c03030f57f69ba2541baa44d2e6ad138c28cd3 scipy-1.7.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 54951f51d731c832b1b8885e0a92e89f33d087de7e40d02078bf0d49c7cbdbb5 scipy-1.7.2-cp310-cp310-win_amd64.whl 30bdda199667e74b50208a793eb1ba47a04e5e3fa16f5ff06c6f7969ae78e4da scipy-1.7.2-cp37-cp37m-macosx_10_9_x86_64.whl e3efe7ef75dfe627b354ab0af0dbc918eadee97cc80ff1aabea6d3e01114ebdd scipy-1.7.2-cp37-cp37m-manylinux_2_12_i686.manylinux2010_i686.whl 17fd991a275e4283453f89d404209aa92059ac68d76d804b4bc1716a3742e1b5 scipy-1.7.2-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl 74f518ce542533054695f743e4271cb8986b63f95bb51d70fcee4f3929cbff7d scipy-1.7.2-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 1437073f1d4664990879aa8f9547524764372e0fef84a077be4b19e82bba7a8d scipy-1.7.2-cp37-cp37m-win32.whl 39f838ea5ce8da868785193d88d05cf5a6d5c390804ec99de29a28e1dcdd53e6 scipy-1.7.2-cp37-cp37m-win_amd64.whl e08b81fcd9bf98740b58dc6fdd7879e33a64dcb682201c1135f7d4a75216bb05 scipy-1.7.2-cp38-cp38-macosx_10_9_x86_64.whl 7b1d0f5f524518f1a86f288443528e4ff4a739c0966db663af4129b7ac7849f8 scipy-1.7.2-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl cac71d5476a6f56b50459da21f6221707e0051ebd428b2137db32ef4a43bb15e scipy-1.7.2-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 4d175ba93e00d8eef8f7cd70d4d88a9106a86800c82ea03cf2268c36d6545483 scipy-1.7.2-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 8b5726a0fedeaa6beb1095e4466998bdd1d1e960b28db9b5a16c89cbd7b2ebf1 scipy-1.7.2-cp38-cp38-win32.whl 8482c8e45857ab0a5446eb7460d2307a27cbbe659d6d2257820c6d6eb950fd0f scipy-1.7.2-cp38-cp38-win_amd64.whl 1ea6233f5a365cb7945b4304bd06323ece3ece85d6a3fa8598d2f53e513467c9 scipy-1.7.2-cp39-cp39-macosx_10_9_x86_64.whl d86abd1ddf421dea5e9cebfeb4de0d205b3dc04e78249afedba9c6c3b2227ff2 scipy-1.7.2-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl 2d25272c03ee3c0fe5e0dff1bb7889280bb6c9e1766fa9c7bde81ad8a5f78694 scipy-1.7.2-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 5273d832fb9cd5724ee0d335c16a903b923441107dd973d27fc4293075a9f4e3 scipy-1.7.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 87cf3964db0f1cce17aeed5bfc1b89a6b4b07dbfc48e50d21fa3549e00456803 scipy-1.7.2-cp39-cp39-win32.whl a80eb01c43fd98257ec7a49ff5cec0edba32031b5f86503f55399a48cb2c5379 scipy-1.7.2-cp39-cp39-win_amd64.whl fa2dbabaaecdb502641b0b3c00dec05fb475ae48655c66da16c9ed24eda1e711 scipy-1.7.2.tar.gz ee5d018ecad0364289efe3301f6445d7ef548637e0e14d0205bbf363f0dfe66a scipy-1.7.2.tar.xz 1a6f793b0d7d902b9b7717bf1ca4295efafb7927b1cce2f7b66cb65e87ebcd89 scipy-1.7.2.zip

1 0

ndenumerate for masked arrays
by Jouke Witteveen 05 Nov '21

05 Nov '21

Hi all, Using `np.ndenumerate` on masked arrays gives unexpected results. For instance, next(np.ndenumerate(np.ma.masked_all(()))) does *not* equal `((), np.ma.masked)`, but instead contains whatever value happens to be in the uninitialized data part of the masked array. Even better than yielding `((), np.ma.masked)` would be for `ndenumerate` to skip masked entries altogether. After all, we are also getting the index of the entries we are iterating over. I went ahead and implemented this specialization of `ndenumerate` for masked arrays: https://github.com/numpy/numpy/pull/20020 If there is anything I can do to advance this pull request, please let me know. Regards, - Jouke

1 0