Mailman 3 January 2023 - SciPy-Dev

RFC: decouple doctesting from refguide-check
by Evgeni Burovski Feb. 23, 2024

Feb. 23, 2024

Hi, Code examples in the SciPy and NumPy documentation are doctested, with a modified doctesting machinery which understands floating point, numpy formatting and some details of our documentation and API. Our modified doctesting machinery is buried in the refguide-check utility, which also does several unrelated things, all of which are tightly coupled to each other and to the libraries themselves. It has been sometimes described as hard to understand or work with or extend due to this tight coupling and a lack of a dedicated test suite. The numpy and scipy versions of the utility are both vendored, and the numpy version has diverged from the scipy version somewhat. Following a discussion in https://github.com/numpy/numpy/issues/21070 I did a small experiment to decouple the doctesting into a separate package, so that it's easier to consolidate the two versions. Plus, a separate repo is generally easier to maintain, configure, and possibly extend or adopt to other projects. The work-in-progress result is here: https://github.com/ev-br/scpdt It currently can run the full doctesting of the scipy API documentation (docstrings of objects) --- see https://github.com/ev-br/scpdt/pull/33 (the log of a test run with warnings turned to errors is in the GH actions: https://github.com/ev-br/scpdt/runs/6743881766?check_suite_focus=true). Note that it shows, among other things, a couple of deprecation warnings our docs have accumulated :-). The API of the tool closely follows that of the standard library doctest module and provides (nearly) drop-in replacements for doctest checking, parsing, finding and running. Various configuration options for our modifications are collected into a single bag object which is internally passed around. This way, it's user-configurable all the way from a plain standard doctest module behavior to what refguide-check does now. There are a couple of wrinkles to iron out; overall it does what refguide-check does already. One missing bit is doctesting rst or other text files, but it's coming soon. The current plan is to: - verify that the standalone version does not miss things checked by the refguide-check - plumb it through the SciPy dev interface and rip out the refguide-check bundled doctesting utilities. - Sync changes that NumPy version of refguide-check accumulated over time - Make sure it correctly tests the NumPy docs, too. - Better document the internals, there is currently only a readme file. If someone's interested to join me working on these, great, the more the merrier :-). I think it could make sense to move the tool's repository to the scipy github org (or maybe even numpy org?). I'm offering to maintain it regardless of the location. Thoughts? Cheers, Evgeni

5 8

Deprecate scipy.misc and Add scipy.datasets submodule
by Anirudh Dagar Jan. 7, 2024

Jan. 7, 2024

Hi all, I just filed https://github.com/scipy/scipy/pull/15607 after an offline discussion with Ralf earlier this month. The summary of that PR is: This is a good time to get rid of the `scipy.misc` submodule since it houses only a handful of functions. As of today, most of the methods have already been deprecated from the `misc` module. They have generally moved under some other submodule. In this case, it is best to create a `scipy.datasets` submodule that can be home to the dataset functions (ascent, face, ecg). Adding a datasets submodule is not a new idea, and was proposed a couple of years ago by Warren (see https://github.com/scipy/scipy/pull/8707), but unfortunately, it never went ahead. Adding this submodule is not only sensible from an organizational/structural point of view but will also help reduce the wheel size of SciPy package. This is possible if we decouple the dataset files from the SciPy repository making use of Pooch to download and cache datasets kept in separate repos. See the PR for more details. I also filed https://github.com/scipy/scipy/issues/15608 Out of the 5 methods in `scipy.misc`, three dataset functions can be moved to this datasets submodule and will slowly be deprecated from misc. The question about the other two methods i.e (derivative, central_diff_weights) is up for discussion and I'd like to understand what will be the best way to deprecate them. Should we move them to a submodule like integrate (keeping derivative makes sense to me)? On the other hand, considering these two methods are not used extensively so one possibility is to deprecate both of them completely and remove them from SciPy. Please share your opinion/concerns, and comment if you see a potential issue with this. Cheers, Anirudh

4 10

Next newcomers' meeting: Friday, July 22 at 8pm UTC
by Melissa Mendonça Aug. 17, 2023

Aug. 17, 2023

Hello, folks! This friday, *July 22 at 8pm UTC* we are having another newcomers' meeting. All are welcome to join and ask questions about contributing, setting up your development environment, PRs or issues you may want to work on, or just chat. Join us on Zoom <https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09>, and if you need more information check out some notes from our previous meeting here: https://hackmd.io/@melissawm/scipy-newcomers Hope to see you around! - Melissa

1 14

Basic spherical statistics in scipy
by Daniel Schmitz April 10, 2023

April 10, 2023

Hi everyone, recently questions came up with regards to statistics on the unit sphere in scipy. Issue 12041 <https://github.com/scipy/scipy/issues/12041> raises the question if spherical mean/variance should be added as descriptive statistics similar to circmean/circvar <https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.circvar.ht…> . The canonical reference for all things related to statistics on the unit sphere is "Directional Statistics" by Mardia & Jupp. Another important basic functionality is the generation of random samples on the unit sphere: discussion was started in this issue <https://github.com/scipy/scipy/issues/16205> . Any strong opinion against these? I think at least the random number generation is reimplemented every day around the world (done so myself at least 2 times) and would strongly benefit from being part of scipy. Best

4 9

SciPy Community Meeting on Wed, November 2 at 8pm UTC
by Melissa Mendonça March 21, 2023

March 21, 2023

Hello, all! The next SciPy Community Meeting will happen this week, on* Wednesday November 2 at 8pm UTC*. Here's the hackmd we are using to take notes: https://hackmd.io/pyhudrC5TgSwdJtpPldHgQ If you wish to join on Zoom, use this link: https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09 Our community calendar is also available here: https://scientific-python.org/calendars/ Cheers, Melissa

1 8

Add isotonic regression
by Christian Lorentzen Feb. 9, 2023

Feb. 9, 2023

Dear scipy community I would like to propose to add isotonic regression to scipy [1] and I am very interested in your opinion. What is isotonic regression? "In monotone (or isotone) regression we fit an increasing or decreasing function to a set of points in the plane." [2] Isotonic regression is a nonparametric curve fitting technique consisting of 2 steps: 1. Solve the isotonic regression problem, see Eq. (2) in [2] or problem statement in [3]. Given a set of points y, it results in a set of points x of same length that is monotonically increasing. This step can be formulated as a constrained convex optimization problem and is usually solved by the pool adjacent violators algorithm (PAVA). 2. The isotonic solution points x are linearly interpolated to get a monotonically increasing nonparametric function approximation or a fit curve. What is it used for? As an old method (Brunk 1955, Ayers et al 1955, van Eeden 1958), isotonic regression has applications in many fields like operations research, signal processing, and statistics. To be more specific, it is used in dose-response curve fitting and calibration of ML models, in particular for probabilistic classifiers, see also [4]. Situation of the ecosystem? While isotonic regression is available in base R (isoreg in the stats package), there are at least 2 further high quality implementations available as R packages [5] and many further packages that depend on them. The only Python implementation I am aware of is the one in scikit-learn [6]. As a univariate regression technique, I personally would have never searched in scikit-learn and I guess that scikit-learn would happily use an implementation from scipy if available. Place in scipy? This brings me back to the start and the proposal to include it in scipy. Given a positive answer, the next question is where in scipy? I guess the options are scipy.interpolate and scipy.optimize (or, very bold, a new module scipy.curve_fitting). - As PAVA, in the end, is a convex optimisation problem, "pava" or a base "isotonic_regression" function could have it's place in scipy.optimize. - The functionality for curve fitting, say class "IsotonicRegression", could be place in either scipy.optimize or scipy.interpolate as both have curve fitting functionalities. I am looking forward to your insights and thoughts. All the best Christian Refereces [1] https://github.com/scipy/scipy/issues/17706 [2] de Leeuw, J., Hornik, K., & Mair, P. (2009). Isotone Optimization in R: Pool-Adjacent-Violators Algorithm (PAVA) and Active Set Methods. Journal of Statistical Software, 32(5), 1–24. https://doi.org/10.18637/jss.v032.i05 [3] https://en.wikipedia.org/wiki/Isotonic_regression [4] Busing, F. M. T. A. (2022). Monotone Regression: A Simple and Fast O(n) PAVA Implementation. Journal of Statistical Software, Code Snippets, 102(1), 1–25. https://doi.org/10.18637/jss.v102.c01 [5] - https://cran.r-project.org/package=monotone - https://cran.r-project.org/package=isotone - https://cran.r-project.org/package=cir [6] https://scikit-learn.org/stable/modules/generated/sklearn.isotonic.isotonic… and https://scikit-learn.org/stable/modules/generated/sklearn.isotonic.Isotonic…

1 1

Doc side bar cleaning
by Pamphile Roy Feb. 9, 2023

Feb. 9, 2023

Hi everyone, I would like to propose to slim down the side bar on the API pages. At the moment, we have a long list which includes most/all submodules as well. I would propose to remove all submodules to only keep top level modules. From the analytics, we have around 10% of our viewer using small screens and for them the experience is very bad. (Even when we will be able to new versions of the theme which improve the situation, it would still be a long list to go through.) Still from analytics, most of the exit pages are not from submodules. Hence I think the impact on readers of the doc should be minimal (see image attached, and stats are similar for individual page count and entry pages). In the following PR, we removed the stats submodules to only keep stats in the side bar. We thing it’s cleaner. (We originally started to look into this as we wanted to remove things like result classes mentions.) https://github.com/scipy/scipy/pull/17876 If there is no objections, I would make a follow up PR to do the change. See the attached images for a preview of how it would look like. Cheers, Pamphile

5 8

Several enhancements to the Rotation module
by wsshambaugh＠gmail.com Feb. 5, 2023

Feb. 5, 2023

Hello! I've been working on a series of enhancements to the `scipy.spatial.transform.Rotation` module that are ready for review. I'd appreciate a look over them from anyone familiar with this area of the codebase or 3D rotations in general. * ENH: Distance between rotations and rotation equality checks https://github.com/scipy/scipy/pull/17460 * Implements `Rotation.dist()`, which calculates the smallest angular distance between two rotations. * Implements `Rotation.approx_equal()`, which calculates the distance between rotations and checks if the angle in radians is under an absolute tolerance. * ENH: Add the `__pow__` operator for Rotations https://github.com/scipy/scipy/pull/17452 * This adds the `__pow__` operator for Rotation objects. This allows them to be composed with themselves, for example: `R * R == R ** 2`. Fractional powers are also allowed. * ENH: Map the rotation quaternion double cover of rotation space to a unique single cover https://github.com/scipy/scipy/pull/17334 * Rotation quaternions form a dual cover of the rotation space, such that `q = -q` describe the same rotation. Using a `Rotation.as_quat(single_cover=True)` flag (default True), this chooses the representation with a positive `w` term. * Changes the implementation of `Rotation.inv()` so that the signs on the quaternion x, y, z terms are flipped instead of the w term. * ENH: Implement `Rotation.align_constrain()` classmethod https://github.com/scipy/scipy/pull/17542 * This implements the "align-constrain" algorithm for defining a rotation. The user passes in primary and secondary vectors in reference frames A and B. The algorithm calculates a rotation from B to A that exactly aligns the primary vectors, and then rotates about that axis to most closely align the constrained secondary vectors. * Feedback wanted on defining behavior when primary and secondary vectors are parallel. Cheers - Scott Shambaugh

2 2

Google Summer of Code
by Hameer Abbasi Feb. 3, 2023

Feb. 3, 2023

Hello everyone, Are we willing to participate in GSoC this year? Last year I had a good student and we were able to make good progress. I’d be thrilled to apply again. Is there anyone else who would be benefited by this? Best regards, Hameer Abbasi

5 6

ENH: overlap parameter in scipy.linalg.block_diag
by Nicolas Vetsch Jan. 31, 2023

Jan. 31, 2023

Hi everyone, While working on a particular problem with finite-element matrices for periodic mesh sub-domains, I found it very useful to have a piece of code that allows me to create block-diagonal matrices with some defined overlap, such that consecutive matrices on the diagonal overlap additively. I opened a PR over at https://github.com/scipy/scipy/pull/17884, where I implemented an `overlap` parameter for `scipy.linalg.block_diag`. I guess for my use-case it makes more sense to have this feature in `scipy.sparse.block_diag` (will add this as well if you are happy with this API change) but I think others may find this useful in linalg.block_diag as well and the default behavior would naturally not be affected in any way with `overlap=0`. From a quick google of "block diagonal matrix with overlap" there seems to be at least some interest in block diagonal matrices with overlap. What do you think? Cheers, Nicolas

1 0