Mailman 3 July 2023 - SciPy-Dev

RFC: decouple doctesting from refguide-check
by Evgeni Burovski Feb. 23, 2024

Feb. 23, 2024

Hi, Code examples in the SciPy and NumPy documentation are doctested, with a modified doctesting machinery which understands floating point, numpy formatting and some details of our documentation and API. Our modified doctesting machinery is buried in the refguide-check utility, which also does several unrelated things, all of which are tightly coupled to each other and to the libraries themselves. It has been sometimes described as hard to understand or work with or extend due to this tight coupling and a lack of a dedicated test suite. The numpy and scipy versions of the utility are both vendored, and the numpy version has diverged from the scipy version somewhat. Following a discussion in https://github.com/numpy/numpy/issues/21070 I did a small experiment to decouple the doctesting into a separate package, so that it's easier to consolidate the two versions. Plus, a separate repo is generally easier to maintain, configure, and possibly extend or adopt to other projects. The work-in-progress result is here: https://github.com/ev-br/scpdt It currently can run the full doctesting of the scipy API documentation (docstrings of objects) --- see https://github.com/ev-br/scpdt/pull/33 (the log of a test run with warnings turned to errors is in the GH actions: https://github.com/ev-br/scpdt/runs/6743881766?check_suite_focus=true). Note that it shows, among other things, a couple of deprecation warnings our docs have accumulated :-). The API of the tool closely follows that of the standard library doctest module and provides (nearly) drop-in replacements for doctest checking, parsing, finding and running. Various configuration options for our modifications are collected into a single bag object which is internally passed around. This way, it's user-configurable all the way from a plain standard doctest module behavior to what refguide-check does now. There are a couple of wrinkles to iron out; overall it does what refguide-check does already. One missing bit is doctesting rst or other text files, but it's coming soon. The current plan is to: - verify that the standalone version does not miss things checked by the refguide-check - plumb it through the SciPy dev interface and rip out the refguide-check bundled doctesting utilities. - Sync changes that NumPy version of refguide-check accumulated over time - Make sure it correctly tests the NumPy docs, too. - Better document the internals, there is currently only a readme file. If someone's interested to join me working on these, great, the more the merrier :-). I think it could make sense to move the tool's repository to the scipy github org (or maybe even numpy org?). I'm offering to maintain it regardless of the location. Thoughts? Cheers, Evgeni

5 8

Deprecate scipy.misc and Add scipy.datasets submodule
by Anirudh Dagar Jan. 7, 2024

Jan. 7, 2024

Hi all, I just filed https://github.com/scipy/scipy/pull/15607 after an offline discussion with Ralf earlier this month. The summary of that PR is: This is a good time to get rid of the `scipy.misc` submodule since it houses only a handful of functions. As of today, most of the methods have already been deprecated from the `misc` module. They have generally moved under some other submodule. In this case, it is best to create a `scipy.datasets` submodule that can be home to the dataset functions (ascent, face, ecg). Adding a datasets submodule is not a new idea, and was proposed a couple of years ago by Warren (see https://github.com/scipy/scipy/pull/8707), but unfortunately, it never went ahead. Adding this submodule is not only sensible from an organizational/structural point of view but will also help reduce the wheel size of SciPy package. This is possible if we decouple the dataset files from the SciPy repository making use of Pooch to download and cache datasets kept in separate repos. See the PR for more details. I also filed https://github.com/scipy/scipy/issues/15608 Out of the 5 methods in `scipy.misc`, three dataset functions can be moved to this datasets submodule and will slowly be deprecated from misc. The question about the other two methods i.e (derivative, central_diff_weights) is up for discussion and I'd like to understand what will be the best way to deprecate them. Should we move them to a submodule like integrate (keeping derivative makes sense to me)? On the other hand, considering these two methods are not used extensively so one possibility is to deprecate both of them completely and remove them from SciPy. Please share your opinion/concerns, and comment if you see a potential issue with this. Cheers, Anirudh

4 10

Implementation of Landau distribution into scipy
by Matias Senger Aug. 28, 2023

Aug. 28, 2023

Hi, I am new here, though have been a SciPy user for several years. I am working on solid state radiation detectors and often need to use the [Landau distribution](https://es.wikipedia.org/wiki/Distribuci%C3%B3n_de_Landau) to model my data. Since I could not find it in Python I implemented it myself (see here https://github.com/SengerM/landaupy). It is based on the Root implementation, still it is a "pure Python" version only using scipy and numpy. Now I would like to add it into SciPy so it is there for other people as well. I would like to do this myself mainly for two reasons, one is to give back a grain of salt to SciPy and the other is to actually contribute to a large open source project, which I've never done before but always wanted to. I am following the steps listed in the [contributor quickstart guide](https://scipy.github.io/devdocs/dev/dev_quickstart.html). I run into [an issue](https://github.com/scipy/scipy/issues/18390) which will hopefully be solved but while asking for help about that I was suggested to propose this feature here before sitting to work on it. Please let me know your thoughts. Kind regards, Matias.

2 2

should a benchmark be meaningful and assert accuracy?
by Knyazev, Andrew Aug. 24, 2023

Aug. 24, 2023

https://github.com/scipy/scipy/issues/18568 https://github.com/scipy/scipy/pull/18954 Update/rewrite sparse_linalg_lobpcg.py to make it more reasonable raise a discussion if a benchmark needs to make a meaningful comparison and assert accuracy or just barely run the code for timing even if the code produces garbage and the comparison baseline is irrelevant. In my view, a benchmark is a tool for a user to choose a better performing solver. If a benchmark is meaningless computing garbage because the solver is called in a wrong way or the comparison baseline is wrong, it fails this goal - the issue with the present benchmark. It is also a tool for a developer to validate that changes in a code do not degrade performance. For iterative solvers, performance is determined by a total execution time, which is the time per iteration times the number of iterations needed to get the requested accuracy. The present benchmark does not check the accuracy, so fails this goal too, i.e., is quite useless and misleading. The property that for a known problem a code of an iterative solver gives a guaranteed accuracy in a known fixed number of iterations is desired but not given and must be checked and rechecked with every chance of the code.

4 4

SciPy Community Meeting on Wed, April 5 at 12pm UTC
by Pamphile Roy Aug. 22, 2023

Aug. 22, 2023

Hi everyone :) The next SciPy Community Meeting will happen tomorrow, on Wednesday April 5 at 12pm (noon) UTC. Here's the hackmd we are using to take notes: https://hackmd.io/pyhudrC5TgSwdJtpPldHgQ If you wish to join on Zoom, use this link: https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09 Our community calendar is also available here: https://scientific-python.org/calendars/ Cheers, Pamphile

2 12

Next newcomers' meeting: Friday, July 22 at 8pm UTC
by Melissa Mendonça Aug. 17, 2023

Aug. 17, 2023

Hello, folks! This friday, *July 22 at 8pm UTC* we are having another newcomers' meeting. All are welcome to join and ask questions about contributing, setting up your development environment, PRs or issues you may want to work on, or just chat. Join us on Zoom <https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09>, and if you need more information check out some notes from our previous meeting here: https://hackmd.io/@melissawm/scipy-newcomers Hope to see you around! - Melissa

1 14

Sparse Arrays working group
by CJ Carey Aug. 11, 2023

Aug. 11, 2023

Are you interested in helping to move the scipy.sparse API away from matrix-like semantics? Are you a fan of scipy.sparse in general? If so, you may be interested in attending our biweekly Sparse Array meetings <https://scientific-python.org/calendars/#:~:text=Sparse%20Arrays%20Group,-S…>. You can also follow along asynchronously via the meeting notes: https://hackmd.io/9UCjKdRUTEeoS8KdQNll-A?view We just had our second meeting after the 2023 Developer Summit <https://hackmd.io/1Q2832LDR_2Uv_-cV-wnYg>, and there's plenty of work to be done yet. The next meeting will be two weeks from today, on July 21st. -CJ, on behalf of the Sparse Arrays working group

3 3

NumPy 1.25.2 released
by Charles R Harris July 31, 2023

July 31, 2023

Hi All, On behalf of the NumPy team, I'm pleased to announce the release of NumPy 1.25.2. NumPy 1.25.2 is a maintenance release that fixes bugs and regressions discovered after the 1.25.1 release. This is the last planned release in the 1.25.x series, the next final release will be 1.26.0, which will use the meson build system and support Python 3.12. The Python versions supported by this release are 3.9-3.11 Note that 32 bit wheels are only provided for Windows, all other wheels are 64 bits on account of Ubuntu, Fedora, and other Linux distributions dropping 32 bit support. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.25.2/>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.25.2>. *Contributors* A total of 13 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Aaron Meurer - Andrew Nelson - Charles Harris - Kevin Sheppard - Matti Picus - Nathan Goldbaum - Peter Hawkins - Ralf Gommers - Randy Eckenrode + - Sam James + - Sebastian Berg - Tyler Reddy - dependabot[bot] *Pull requests merged* A total of 19 pull requests were merged for this release. - #24148: MAINT: prepare 1.25.x for further development - #24174: ENH: Improve clang-cl compliance - #24179: MAINT: Upgrade various build dependencies. - #24182: BLD: use ``-ftrapping-math`` with Clang on macOS - #24183: BUG: properly handle negative indexes in ufunc_at fast path - #24184: BUG: PyObject_IsTrue and PyObject_Not error handling in setflags - #24185: BUG: histogram small range robust - #24186: MAINT: Update meson.build files from main branch - #24234: MAINT: exclude min, max and round from ``np.__all__`` - #24241: MAINT: Dependabot updates - #24242: BUG: Fix the signature for np.array_api.take - #24243: BLD: update OpenBLAS to an intermediate commit - #24244: BUG: Fix reference count leak in str(scalar). - #24245: BUG: fix invalid function pointer conversion error - #24255: BUG: Factor out slow ``getenv`` call used for memory policy warning - #24292: CI: correct URL in cirrus.star [skip cirrus] - #24293: BUG: Fix C types in scalartypes - #24294: BUG: do not modify the input to ufunc_at - #24295: BUG: Further fixes to indexing loop and added tests Cheers, Charles Harris

1 0

Signal: Least-squares deconvolution. Autoregressive model, impulse responses, yule-walker, etc.
by Aleksander Kringstad July 30, 2023

July 30, 2023

Hi, I have written some efficient code for learning impulse responses using least-squares minimization. First time trying to contribute. Wanted to know if it seems interesting and if someone wants to review. *Problem* I.e. given input x and output y (both N samples long), we wish to learn an impulse response with M < N samples that minimizes the least squares error in y. Given yp[n] = x[n]h[0]+x[n-1]h[1]+x[n-2]h[2]+... + x[n-M]h[M], minimize sum( (y-yp)**2) wrt. h[0],h[1],h[2],... , Circular convolution is assumed. *Applications* - Learning impulse responses. - Learn parameters of autoregressive models. If you use x and y so that x[n] = y[n-1], t*he algorithm will be equal to learning the parameters of an autoregressive model using the Yule-Walker equation*. *Performance* It will be fast for long time-series, since the A matrix and the b vector in the normal equation are calculated using FFTs, instead of the regular dot(X.T,X) and dot(X.T,b). Also, the toeplitz structure of the normal equation matrix is exploited for performance. This allows for learning longer impulse responses. And for using less memory to store the normal equation matrix. *Notes* Circular convolution is often not what we want, but can be overcome by zero-padding. Alternatively, one wants to avoid that the first M samples of the output to contributes to the error at all.This can be achieved by first using the algorithm to learn the impulse response and then fine-tuning it using scipy.optimize Can add functionality for this fine-tuning + zero-padding. Should also add a few alternatives for linear solvers. Core functionality is only 10 lines of self-written code and uses only numpy and scipy.linalg.solve_toeplitz.

2 2

define Laplacian matrices with a public API
by Knyazev, Andrew July 26, 2023

July 26, 2023

I am asked to attract the list attention to https://github.com/scipy/scipy/issues/18940 https://github.com/scipy/scipy/pull/18963 Add a new special matrix, of the Laplacian, in multiple formats with a public API for testing and benchmarking.

1 0