Mailman 3 August 2022 - SciPy-Dev

RFC: decouple doctesting from refguide-check
by Evgeni Burovski Feb. 23, 2024

Feb. 23, 2024

Hi, Code examples in the SciPy and NumPy documentation are doctested, with a modified doctesting machinery which understands floating point, numpy formatting and some details of our documentation and API. Our modified doctesting machinery is buried in the refguide-check utility, which also does several unrelated things, all of which are tightly coupled to each other and to the libraries themselves. It has been sometimes described as hard to understand or work with or extend due to this tight coupling and a lack of a dedicated test suite. The numpy and scipy versions of the utility are both vendored, and the numpy version has diverged from the scipy version somewhat. Following a discussion in https://github.com/numpy/numpy/issues/21070 I did a small experiment to decouple the doctesting into a separate package, so that it's easier to consolidate the two versions. Plus, a separate repo is generally easier to maintain, configure, and possibly extend or adopt to other projects. The work-in-progress result is here: https://github.com/ev-br/scpdt It currently can run the full doctesting of the scipy API documentation (docstrings of objects) --- see https://github.com/ev-br/scpdt/pull/33 (the log of a test run with warnings turned to errors is in the GH actions: https://github.com/ev-br/scpdt/runs/6743881766?check_suite_focus=true). Note that it shows, among other things, a couple of deprecation warnings our docs have accumulated :-). The API of the tool closely follows that of the standard library doctest module and provides (nearly) drop-in replacements for doctest checking, parsing, finding and running. Various configuration options for our modifications are collected into a single bag object which is internally passed around. This way, it's user-configurable all the way from a plain standard doctest module behavior to what refguide-check does now. There are a couple of wrinkles to iron out; overall it does what refguide-check does already. One missing bit is doctesting rst or other text files, but it's coming soon. The current plan is to: - verify that the standalone version does not miss things checked by the refguide-check - plumb it through the SciPy dev interface and rip out the refguide-check bundled doctesting utilities. - Sync changes that NumPy version of refguide-check accumulated over time - Make sure it correctly tests the NumPy docs, too. - Better document the internals, there is currently only a readme file. If someone's interested to join me working on these, great, the more the merrier :-). I think it could make sense to move the tool's repository to the scipy github org (or maybe even numpy org?). I'm offering to maintain it regardless of the location. Thoughts? Cheers, Evgeni

5 8

Deprecate scipy.misc and Add scipy.datasets submodule
by Anirudh Dagar Jan. 7, 2024

Jan. 7, 2024

Hi all, I just filed https://github.com/scipy/scipy/pull/15607 after an offline discussion with Ralf earlier this month. The summary of that PR is: This is a good time to get rid of the `scipy.misc` submodule since it houses only a handful of functions. As of today, most of the methods have already been deprecated from the `misc` module. They have generally moved under some other submodule. In this case, it is best to create a `scipy.datasets` submodule that can be home to the dataset functions (ascent, face, ecg). Adding a datasets submodule is not a new idea, and was proposed a couple of years ago by Warren (see https://github.com/scipy/scipy/pull/8707), but unfortunately, it never went ahead. Adding this submodule is not only sensible from an organizational/structural point of view but will also help reduce the wheel size of SciPy package. This is possible if we decouple the dataset files from the SciPy repository making use of Pooch to download and cache datasets kept in separate repos. See the PR for more details. I also filed https://github.com/scipy/scipy/issues/15608 Out of the 5 methods in `scipy.misc`, three dataset functions can be moved to this datasets submodule and will slowly be deprecated from misc. The question about the other two methods i.e (derivative, central_diff_weights) is up for discussion and I'd like to understand what will be the best way to deprecate them. Should we move them to a submodule like integrate (keeping derivative makes sense to me)? On the other hand, considering these two methods are not used extensively so one possibility is to deprecate both of them completely and remove them from SciPy. Please share your opinion/concerns, and comment if you see a potential issue with this. Cheers, Anirudh

4 10

Next newcomers' meeting: Friday, July 22 at 8pm UTC
by Melissa Mendonça Aug. 17, 2023

Aug. 17, 2023

Hello, folks! This friday, *July 22 at 8pm UTC* we are having another newcomers' meeting. All are welcome to join and ask questions about contributing, setting up your development environment, PRs or issues you may want to work on, or just chat. Join us on Zoom <https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09>, and if you need more information check out some notes from our previous meeting here: https://hackmd.io/@melissawm/scipy-newcomers Hope to see you around! - Melissa

1 14

Basic spherical statistics in scipy
by Daniel Schmitz April 10, 2023

April 10, 2023

Hi everyone, recently questions came up with regards to statistics on the unit sphere in scipy. Issue 12041 <https://github.com/scipy/scipy/issues/12041> raises the question if spherical mean/variance should be added as descriptive statistics similar to circmean/circvar <https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.circvar.ht…> . The canonical reference for all things related to statistics on the unit sphere is "Directional Statistics" by Mardia & Jupp. Another important basic functionality is the generation of random samples on the unit sphere: discussion was started in this issue <https://github.com/scipy/scipy/issues/16205> . Any strong opinion against these? I think at least the random number generation is reimplemented every day around the world (done so myself at least 2 times) and would strongly benefit from being part of scipy. Best

4 9

SciPy Community Meeting on Wed, April 13 at 8pm UTC
by Melissa Mendonça Oct. 18, 2022

Oct. 18, 2022

Hello, all! The next SciPy Community Meeting will happen this week, on* Wednesday March 16th, at 8pm UTC*. Here's the hackmd we are using to take notes: https://hackmd.io/pyhudrC5TgSwdJtpPldHgQ If you wish to join on Zoom, use this (updated) link: https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09 <https://us06web.zoom.us/j/6345425936> Our community calendar is now also available here: https://scientific-python.org/calendars/ Cheers, Melissa

1 12

Add Symmetric LQ method for symmetric indefinite systems of linear equations
by 赵刚 Sept. 25, 2022

Sept. 25, 2022

Dear all, I submitted a PR #16202 to introduce the Symmetric LQ (SYMMLQ) method in SciPy and fix the user's issue #13537. The PR website is as follows: https://github.com/scipy/scipy/pull/16202 Just like Minimal Residual method (MINRES), the SYMMLQ is also a key and important Krylov subspace method for symmetric indefinite systems, which was originally proposed by C.C.Paige and M.A.Saunders (cited by 2283 in Google) and widely used in some well-known math libraries such as MATLAB, PETSc, Trilinos and PyKrylov mentioned in the issue #13537. SYMMLQ is a variant of Conjugate Gradient method (CG). For symmetric positive-definite problems, SYMMLQ and CG has same convergence, but for symmetric indefinite problems, SYMMLQ has a smoother convergence curve than CG (CG is easier to diverge), its test results are shown in `Additional Information` of #16202. This PR is a python translation of PETSc implementation (It has been cited in `References` of `symmlq.py`) and some changes have been made based on the operation features of NumPy/SciPy. If anyone is interested in this PR, please feel free to help review it. Hope you like it, thanks. Best Regards, Gang

2 1

WIP: Whittaker−Henderson smoothing to signal
by andythe_great Sept. 1, 2022

Sept. 1, 2022

Hello, I would like to add Whittaker−Henderson smoothing to /scipy/signal in addition to the existing savitzky_golay. This function smooths the data. Whittaker−Henderson smoothing provides fewer artifacts compared to Savitzky-Golay. https://pubs.acs.org/doi/10.1021/acsmeasuresciau.1c00054 I would like some comments if this is a good idea. Currently, the code is in progress and hosted here. https://github.com/kevinsmia1939/scipy/blob/main/scipy/signal/_whittaker_he… At the moment, I did not implement the way to handle inconsistent data interval. Andy

4 3

New feature: fitting distributions to censored data.
by Warren Weckesser Aug. 29, 2022

Aug. 29, 2022

Hey folks, A new enhancement for fitting probability distributions to censored data by maximum likelihood estimation is in progress in https://github.com/scipy/scipy/pull/13699 This is a feature in the statistics roadmap, and is part of the work for the CZI EOSS cycle 1 grant. Censored data is data where the true value of the measurement is unknown, but there is a known bound. The common terminology is: * left-censored: The true value is greater than some known bound. * right-censored: The true value is less than some known bound. * interval-censored: The true value is between known bounds. (By allowing the bounds to be +inf and -inf, we can think of all these cases as interval-censored.) In the PR, a new class, `CensoredData`, is defined to represent censored data. The `fit` method of the univariate continuous distributions is updated to accept an instance of `CensoredData`. The `CensoredData` class constructor accepts two arrays, `lower` and `upper`, to hold the known bounds of the data. (If `lower[i] == upper[i]`, it means that data value is not censored.) Because it is quite common for data to be encountered where the only censored values in the data set are either all right-censored or all left-censored, two convenience methods, `CensoredData.right_censored(x, censored)` and `CensoredData.left_censored(x, censored)`, are provide to create a `CensoredData` instance from an array of values and a corresponding boolean array that indicates if the value is censored. Here's a quick example, with data from https://www.itl.nist.gov/div898/handbook/apr/section4/apr413.htm. The data table shows 10 regular values and 10 right-censored values. The results reported there for fitting the two-parameter Weibull distribution (location fixed at 0) to that data are shape = 1.7208 scale = 606.5280 Here's the calculation using the proposed API in SciPy: In [55]: from scipy.stats import weibull_min, CensoredData Create the `CensoredData` instance: In [56]: x = np.array([54, 187, 216, 240, 244, 335, 361, 373, 375, 386] + [500]*10) In [57]: data = CensoredData.right_censored(x, x == 500) In [58]: print(data) CensoredData(20 values: 10 not censored, 10 right-censored) Fit `weibull_min` (with the location fixed at 0) to the censored data: In [59]: shape, loc, scale = weibull_min.fit(data, floc=0) In [60]: shape Out[60]: 1.720797180719942 In [61]: scale Out[61]: 606.527565269458 Matt Haberland has already suggested quite a few improvements to the PR. Additional comments would be appreciated. Warren

4 7

ANN: SciPy 1.9.1
by Tyler Reddy Aug. 27, 2022

Aug. 27, 2022

Hi all, On behalf of the SciPy development team, I'm pleased to announce the release of SciPy 1.9.1, which is a bug fix release that includes some important meson build fixes. Sources and binary wheels can be found at: https://pypi.org/project/scipy/ and at: https://github.com/scipy/scipy/releases/tag/v1.9.1 One of a few ways to install this release with pip: pip install scipy==1.9.1 ===================== SciPy 1.9.1 Release Notes ===================== SciPy 1.9.1 is a bug-fix release with no new features compared to 1.9.0. Notably, some important meson build fixes are included. Authors ====== * Anirudh Dagar (1) * Ralf Gommers (12) * Matt Haberland (2) * Andrew Nelson (1) * Tyler Reddy (14) * Atsushi Sakai (1) * Eli Schwartz (1) * Warren Weckesser (2) A total of 8 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 1.9.1 ------------------------------ * `#14517 <https://github.com/scipy/scipy/issues/14517>`__: scipy/linalg/tests/test_decomp.py::TestSchur::test_sort test... * `#16765 <https://github.com/scipy/scipy/issues/16765>`__: DOC: \`scipy.stats.skew\` no longer returns 0 on constant input * `#16787 <https://github.com/scipy/scipy/issues/16787>`__: BUG: Can't build 1.10 with mingw-w64 toolchain and numpy 1.21.6... * `#16813 <https://github.com/scipy/scipy/issues/16813>`__: BUG: scipy.interpolate interp1d extrapolate behaviour change... * `#16878 <https://github.com/scipy/scipy/issues/16878>`__: BUG: optimize.milp fails to execute when given exactly 3 constraints Pull requests for 1.9.1 ------------------------------ * `#16709 <https://github.com/scipy/scipy/pull/16709>`__: BLD: make the way we count commits for version numbering more... * `#16736 <https://github.com/scipy/scipy/pull/16736>`__: REL: prep for SciPy 1.9.1 * `#16749 <https://github.com/scipy/scipy/pull/16749>`__: BLD: install missing \`.pxd\` files, and update TODOs/FIXMEs... * `#16750 <https://github.com/scipy/scipy/pull/16750>`__: BLD: make OpenBLAS detection work with CMake * `#16755 <https://github.com/scipy/scipy/pull/16755>`__: TST: sparse.linalg: Loosen tolerance for the lobpcg test 'test_tolerance_float32' * `#16760 <https://github.com/scipy/scipy/pull/16760>`__: BLD: use a bit more idiomatic approach to constructing paths... * `#16768 <https://github.com/scipy/scipy/pull/16768>`__: DOC: stats.skew/kurtosis: returns NaN when input has only one... * `#16794 <https://github.com/scipy/scipy/pull/16794>`__: BLD/REL: on Windows use numpy 1.22.3 as the version to build... * `#16822 <https://github.com/scipy/scipy/pull/16822>`__: BUG/TST: linalg: Check the results of 'schur' more carefully. * `#16825 <https://github.com/scipy/scipy/pull/16825>`__: BUG: interpolate: fix "previous" and "next" extrapolate logic... * `#16862 <https://github.com/scipy/scipy/pull/16862>`__: BUG, DOC: Fix sphinx autosummary generation for \`odr\` and \`czt\` * `#16881 <https://github.com/scipy/scipy/pull/16881>`__: MAINT: optimize.milp: fix input validation when three constraints... * `#16901 <https://github.com/scipy/scipy/pull/16901>`__: MAINT: 1.9.1 backports * `#16904 <https://github.com/scipy/scipy/pull/16904>`__: BLD: update dependency ranges for meson-python and pybind11 for... Checksums ========= MD5 ~~~ 65d2b065c09d40b07272ae30332f6202 Changelog 69dd050c058c46e67d05956bf06e0566 README.txt 6108e7119fd6ae6d6338adede1904bb7 scipy-1.9.1-cp310-cp310-macosx_10_9_x86_64.whl 301053451a09b81268734133ce6d7bda scipy-1.9.1-cp310-cp310-macosx_12_0_arm64.whl 43a2fa03b951738d0deed6ddbc0a9f4d scipy-1.9.1-cp310-cp310-macosx_12_0_universal2.macosx_10_9_x86_64.whl 2339afc328d3c92cf85d1d86c5be4e19 scipy-1.9.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl d95584ddfe0703ba3b4624001a1f84f7 scipy-1.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 01937d372de92b3bcf9196b81bf8041d scipy-1.9.1-cp310-cp310-win_amd64.whl 3f85c3d508fced31f420ea31dccb1f83 scipy-1.9.1-cp38-cp38-macosx_10_9_x86_64.whl 3e07b61125901375f3ab99766465a428 scipy-1.9.1-cp38-cp38-macosx_12_0_arm64.whl 5cb2848a03030d3cec82a406b6d032de scipy-1.9.1-cp38-cp38-macosx_12_0_universal2.macosx_10_9_x86_64.whl cffdc85788b0d11e00f5c055e303538e scipy-1.9.1-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl ad0b411b353608b9fdd2fb3da25dd9ef scipy-1.9.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl fa1872fdcc63b57b95bffdcd2142a54e scipy-1.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 673905ce31524d16e5d44e600c48ff4d scipy-1.9.1-cp38-cp38-win32.whl bc421013525427036272cafa47c41f71 scipy-1.9.1-cp38-cp38-win_amd64.whl 19aa6604a3c39cbcc58f7a626e3fad89 scipy-1.9.1-cp39-cp39-macosx_10_9_x86_64.whl 0f6312acdf5edf5a6a5d2bf453e1c6cc scipy-1.9.1-cp39-cp39-macosx_12_0_arm64.whl 94ab8626a5d2aa5ed936f8571593567b scipy-1.9.1-cp39-cp39-macosx_12_0_universal2.macosx_10_9_x86_64.whl 4a9c22a4a059383cfb96cc2373156a30 scipy-1.9.1-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl 156f552987ff57cc347326b1df0a5916 scipy-1.9.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 8937a4cece2682585ceaf4115e526e21 scipy-1.9.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 9e1d2a5368f20c120da578a9f99b3986 scipy-1.9.1-cp39-cp39-win32.whl 3571be3d0e79cd46b0ab4c6418f6d44b scipy-1.9.1-cp39-cp39-win_amd64.whl SHA256 ~~~~~~ b6b1343d400fb775a7aaefe51654f56478a795318f11c29762a949a024cef9ca Changelog 1f18fbebfc36e5f6b3971b399fda124bd0c307308b407475d37059ba6a66be99 README.txt c61b4a91a702e8e04aeb0bfc40460e1f17a640977c04dda8757efb0199c75332 scipy-1.9.1-cp310-cp310-macosx_10_9_x86_64.whl d79da472015d0120ba9b357b28a99146cd6c17b9609403164b1a8ed149b4dfc8 scipy-1.9.1-cp310-cp310-macosx_12_0_arm64.whl 825951b88f56765aeb6e5e38ac9d7d47407cfaaeb008d40aa1b45a2d7ea2731e scipy-1.9.1-cp310-cp310-macosx_12_0_universal2.macosx_10_9_x86_64.whl f950a04b33e17b38ff561d5a0951caf3f5b47caa841edd772ffb7959f20a6af0 scipy-1.9.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 8cc81ac25659fec73599ccc52c989670e5ccd8974cf34bacd7b54a8d809aff1a scipy-1.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 8d3faa40ac16c6357aaf7ea50394ea6f1e8e99d75e927a51102b1943b311b4d9 scipy-1.9.1-cp310-cp310-win_amd64.whl 7a412c476a91b080e456229e413792bbb5d6202865dae963d1e6e28c2bb58691 scipy-1.9.1-cp38-cp38-macosx_10_9_x86_64.whl eb954f5aca4d26f468bbebcdc5448348eb287f7bea536c6306f62ea062f63d9a scipy-1.9.1-cp38-cp38-macosx_12_0_arm64.whl 3c6f5d1d4b9a5e4fe5e14f26ffc9444fc59473bbf8d45dc4a9a15283b7063a72 scipy-1.9.1-cp38-cp38-macosx_12_0_universal2.macosx_10_9_x86_64.whl bc4e2c77d4cd015d739e75e74ebbafed59ba8497a7ed0fd400231ed7683497c4 scipy-1.9.1-cp38-cp38-manylinux_2_12_i686.manylinux2010_i686.whl 0419485dbcd0ed78c0d5bf234c5dd63e86065b39b4d669e45810d42199d49521 scipy-1.9.1-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 34441dfbee5b002f9e15285014fd56e5e3372493c3e64ae297bae2c4b9659f5a scipy-1.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl b97b479f39c7e4aaf807efd0424dec74bbb379108f7d22cf09323086afcd312c scipy-1.9.1-cp38-cp38-win32.whl e8fe305d9d67a81255e06203454729405706907dccbdfcc330b7b3482a6c371d scipy-1.9.1-cp38-cp38-win_amd64.whl 39ab9240cd215a9349c85ab908dda6d732f7d3b4b192fa05780812495536acc4 scipy-1.9.1-cp39-cp39-macosx_10_9_x86_64.whl 71487c503e036740635f18324f62a11f283a632ace9d35933b2b0a04fd898c98 scipy-1.9.1-cp39-cp39-macosx_12_0_arm64.whl 3bc1ab68b9a096f368ba06c3a5e1d1d50957a86665fc929c4332d21355e7e8f4 scipy-1.9.1-cp39-cp39-macosx_12_0_universal2.macosx_10_9_x86_64.whl f7c39f7dbb57cce00c108d06d731f3b0e2a4d3a95c66d96bce697684876ce4d4 scipy-1.9.1-cp39-cp39-manylinux_2_12_i686.manylinux2010_i686.whl 47d1a95bd9d37302afcfe1b84c8011377c4f81e33649c5a5785db9ab827a6ade scipy-1.9.1-cp39-cp39-manylinux_2_17_aarch64.manylinux2014_aarch64.whl 96d7cf7b25c9f23c59a766385f6370dab0659741699ecc7a451f9b94604938ce scipy-1.9.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl 09412eb7fb60b8f00b328037fd814d25d261066ebc43a1e339cdce4f7502877e scipy-1.9.1-cp39-cp39-win32.whl 90c805f30c46cf60f1e76e947574f02954d25e3bb1e97aa8a07bc53aa31cf7d1 scipy-1.9.1-cp39-cp39-win_amd64.whl

1 0

Revisiting automatic Pull Requests labeling
by Julien Jerphanion Aug. 26, 2022

Aug. 26, 2022

Hello everyone, As indicated in SciPy Community Meeting on July 27, I am interested in helping SciPy on aspects regarding Cython / C++ / Fortran / performance. In starting those efforts, I am proposing in #16870 to label Pull Requests using labels for languages they use and modules they modified automatically and uniquely when they are created. Pamphile indicated that a similar setup I was unaware of was explored in the past, via #14677 #14709 and was then removed by #14967. As this setup is slightly different, he indicated that I send a message on this mailing list to follow-up with discussions regarding this CI setup. Personally, I think this setup help automating some repetitive triage work. What do you think of this proposal? If people are interested in discussing this further, I think this can be done in this PR itself for more openness. Kind Regards, Julien.

1 0