Mailman 3 October 2022 - SciPy-Dev

RFC: decouple doctesting from refguide-check
by Evgeni Burovski Feb. 23, 2024

Feb. 23, 2024

Hi, Code examples in the SciPy and NumPy documentation are doctested, with a modified doctesting machinery which understands floating point, numpy formatting and some details of our documentation and API. Our modified doctesting machinery is buried in the refguide-check utility, which also does several unrelated things, all of which are tightly coupled to each other and to the libraries themselves. It has been sometimes described as hard to understand or work with or extend due to this tight coupling and a lack of a dedicated test suite. The numpy and scipy versions of the utility are both vendored, and the numpy version has diverged from the scipy version somewhat. Following a discussion in https://github.com/numpy/numpy/issues/21070 I did a small experiment to decouple the doctesting into a separate package, so that it's easier to consolidate the two versions. Plus, a separate repo is generally easier to maintain, configure, and possibly extend or adopt to other projects. The work-in-progress result is here: https://github.com/ev-br/scpdt It currently can run the full doctesting of the scipy API documentation (docstrings of objects) --- see https://github.com/ev-br/scpdt/pull/33 (the log of a test run with warnings turned to errors is in the GH actions: https://github.com/ev-br/scpdt/runs/6743881766?check_suite_focus=true). Note that it shows, among other things, a couple of deprecation warnings our docs have accumulated :-). The API of the tool closely follows that of the standard library doctest module and provides (nearly) drop-in replacements for doctest checking, parsing, finding and running. Various configuration options for our modifications are collected into a single bag object which is internally passed around. This way, it's user-configurable all the way from a plain standard doctest module behavior to what refguide-check does now. There are a couple of wrinkles to iron out; overall it does what refguide-check does already. One missing bit is doctesting rst or other text files, but it's coming soon. The current plan is to: - verify that the standalone version does not miss things checked by the refguide-check - plumb it through the SciPy dev interface and rip out the refguide-check bundled doctesting utilities. - Sync changes that NumPy version of refguide-check accumulated over time - Make sure it correctly tests the NumPy docs, too. - Better document the internals, there is currently only a readme file. If someone's interested to join me working on these, great, the more the merrier :-). I think it could make sense to move the tool's repository to the scipy github org (or maybe even numpy org?). I'm offering to maintain it regardless of the location. Thoughts? Cheers, Evgeni

5 8

Deprecate scipy.misc and Add scipy.datasets submodule
by Anirudh Dagar Jan. 7, 2024

Jan. 7, 2024

Hi all, I just filed https://github.com/scipy/scipy/pull/15607 after an offline discussion with Ralf earlier this month. The summary of that PR is: This is a good time to get rid of the `scipy.misc` submodule since it houses only a handful of functions. As of today, most of the methods have already been deprecated from the `misc` module. They have generally moved under some other submodule. In this case, it is best to create a `scipy.datasets` submodule that can be home to the dataset functions (ascent, face, ecg). Adding a datasets submodule is not a new idea, and was proposed a couple of years ago by Warren (see https://github.com/scipy/scipy/pull/8707), but unfortunately, it never went ahead. Adding this submodule is not only sensible from an organizational/structural point of view but will also help reduce the wheel size of SciPy package. This is possible if we decouple the dataset files from the SciPy repository making use of Pooch to download and cache datasets kept in separate repos. See the PR for more details. I also filed https://github.com/scipy/scipy/issues/15608 Out of the 5 methods in `scipy.misc`, three dataset functions can be moved to this datasets submodule and will slowly be deprecated from misc. The question about the other two methods i.e (derivative, central_diff_weights) is up for discussion and I'd like to understand what will be the best way to deprecate them. Should we move them to a submodule like integrate (keeping derivative makes sense to me)? On the other hand, considering these two methods are not used extensively so one possibility is to deprecate both of them completely and remove them from SciPy. Please share your opinion/concerns, and comment if you see a potential issue with this. Cheers, Anirudh

4 10

Next newcomers' meeting: Friday, July 22 at 8pm UTC
by Melissa Mendonça Aug. 17, 2023

Aug. 17, 2023

Hello, folks! This friday, *July 22 at 8pm UTC* we are having another newcomers' meeting. All are welcome to join and ask questions about contributing, setting up your development environment, PRs or issues you may want to work on, or just chat. Join us on Zoom <https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09>, and if you need more information check out some notes from our previous meeting here: https://hackmd.io/@melissawm/scipy-newcomers Hope to see you around! - Melissa

1 14

Basic spherical statistics in scipy
by Daniel Schmitz April 10, 2023

April 10, 2023

Hi everyone, recently questions came up with regards to statistics on the unit sphere in scipy. Issue 12041 <https://github.com/scipy/scipy/issues/12041> raises the question if spherical mean/variance should be added as descriptive statistics similar to circmean/circvar <https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.circvar.ht…> . The canonical reference for all things related to statistics on the unit sphere is "Directional Statistics" by Mardia & Jupp. Another important basic functionality is the generation of random samples on the unit sphere: discussion was started in this issue <https://github.com/scipy/scipy/issues/16205> . Any strong opinion against these? I think at least the random number generation is reimplemented every day around the world (done so myself at least 2 times) and would strongly benefit from being part of scipy. Best

4 9

RFC: a proposal to deprecate interpolate.interp2d
by Evgeni Burovski Nov. 8, 2022

Nov. 8, 2022

Hi, TL;DR : I propose to deprecate and eventually remove the `interp2d` routine from `scipy.interpolate` I believe it has both interface and structural problems which cannot be fixed in a satisfactory backwards compatible way; all its functionality is available otherwise via equivalent and more predictable routines (*BivariateSpline and bisplrep); for interpolation specifically we now have better behaved routines (interpn and griddata). Cons for deprecation: - The routine is a direct analog of the MATLAB routine of the same name. - It has been there "forever". - It is probably widely used. This list of reasons make it sound like there is no way we can deprecate it; please read on however. Pros for deprecation: 1. The internal implementation of `interp2d` is not really meant for interpolation, and is known to produce wrong answers in some cases. 2. If it's gone, no functionality is missing from scipy.interpolate: it is a third equivalent way of wrapping FITPACK routines, along with bisplrep/bisplev and *BivariateSpline classes. 3. We now have better, more controllable, more predictable, and easier-to-understand routines for almost all interp2d functionality. 4. `interp2d` interface is confusing to users and error-prone and cannot be fixed while keeping backwards compatibility 5. Being a MATLAB name clone, it naturally attracts new users, who would otherwise use better, more suitable or more predictable routines. Interp2d has at least five open issues: see the FITPACK column of if it's gone, no functionality is missing from scipy.interpolate: it is a third equivalent way of wrapping FITPACK routines, along with bisplrep and *BivariateSpline classes. `interp2d` has at least five open issues, see e.g. the FITPACK column of https://github.com/orgs/scipy/projects/1. I'll now expand on the "pros" points above and try to summarize the open issues. 1. `interp2d` is a wrapper around the FITPACK spline-fitting routines. These are mainly targeted for spline *smoothing*, i.e. fitting a smoothing spline to noisy data. They do so by automagically selecting the spline knots to minimize a certain quality metric which tells how close the fit is to the data. There is a single control knob, the "smoothing parameter s", which can be set to zero to require the interpolation conditions, which influences the knot selection. When this knot construction procedure fails --- and it's most prone to failing for linear interpolation, which is the default for interp2d --- it returns a wrong result, either silently or with an extremely obscure warning. For details on this, see the discussion https://github.com/scipy/scipy/issues/2167. While that issue is about `BivariateSpline`s, interp2d has exactly the same set of problems and no way for a user to control the result. Also note https://github.com/scipy/scipy/issues/11309 , which shows essentially the same problem, only compounded by peculiarities of the interp2d interface. In passing, note a helpful code comment, "# TODO: surfit is really not meant for interpolation!", https://github.com/scipy/scipy/blob/main/scipy/interpolate/_interpolate.py#…, which git blame attributes to commit 7adfbbda57e from 2012 2. We ship at least three ways of wrapping 2D FITPACK routines: the functional interface, `bisplrep / bisplev`, the OOP interface `*BivariateSpline`s and interp2d. The main difference is the set of defaults--- which adds to user confusion, and that interp2d switches between essentially SmoothBivariateSpline and RectBivariateSpline, depending on the shapes of the input arrays. All in all, exactly zero functionality is missing from scipy.interpolate if interp2d is gone. 3. One confusing aspect of the FITPACK procedure is that there is no guarantee that the spline knots coincide with the data points. This makes sense for fitting and smoothing; for interpolation however, most users (I'm speculating here) expect that these coincide. Elsewhere in scipy.interpolate, the currently recommended way is - for regular grids in N-D, use RegularGridInterpolator. This does guarantee that knots=data points. - for scattered data in N-D, use `griddata`. Internally it is triangulation-bases via QHull. In 2D specifically, it allows constructing C1-continuous splines in addition to linear and nearest-neighbor interpolators. 4. First, as mentioned above, `interp2d` switches between different algorithms depending on the input shapes, https://github.com/scipy/scipy/issues/11309. For 1D input arrays it does scattered data fitting, equivalent to the SmoothBivariateSpline; for meshgrid inputs, it's equivalent to RectBivariateSpline. Second, also as mentioned, there is no guarantee that the spline knots coincide with the data. This breaks e.g. a natural expectation that a linear interpolator is bilinear in data. Third, it has an unusual behavior with respect to scalar inputs, see Warren's detailed comment in https://github.com/scipy/scipy/issues/16396#issuecomment-1154175268, which contains the note "I suspect this breaks anyone's mental model for what should happen" Fourth, it does not handle data well with increasing/decreasing ordering, see https://github.com/scipy/scipy/issues/16492. There is also an ample opportunity to silently get wrong results. Concluding, The routine's name seems to hint that it is _the_ way for 2D interpolation, even more so for users with prior Matlab experience. It is however not the recommended routine, it has too many problems, and scipy.interpolate would, I believe, be better with it gone. So the proposal is to deprecate it and add ample documentation for users to migrate to either equivalent or better functionality. I'm ready to do the legwork, and I'm planning to be around to help with user queries. Thoughts? Evgeni

6 18

DOC: missing-bits: document recommendations on return object type
by Matt Haberland Nov. 1, 2022

Nov. 1, 2022

Hi team, gh-17310 proposes to add a note to the "Missing Bits" section of the developer documentation about the following recommendation: when a new function needs to return more than one conceptually distinct piece of information, a non-iterable class is preferable to a tuple, namedtuple, or bunch. This forces callers to be more explicit about the element of the returned object that they wish to access, and it makes it easier to extend the function or method in a backward-compatible way. Feedback is welcome at: https://github.com/scipy/scipy/pull/17310 Thanks! -- Matt Haberland Assistant Professor BioResource and Agricultural Engineering 08A-3K, Cal Poly

6 8

GCV Smoothing Spline reimplementation
by Егор З Oct. 31, 2022

Oct. 31, 2022

Hi everyone! I’ve implemented an algorithm that makes a Smoothing Spline using the Generalized Cross-Validation criteria. It is a clean room reimplementation of gcvspl.f ( https://github.com/srkuberski/gcvspl ). The pull-request https://github.com/scipy/scipy/pull/16653 proposes to add this implementation in the SciPy source code. I’m inviting you to join the discussion directly in the pull-request or by mail. By the way, the discussion about the reimplementation itself can be seen here ( https://github.com/scipy/scipy/issues/6730#issuecomment-299073565 ). Sincerely, Egor Zemlyanoy egorz734(a)mail.ru

1 0

musllinux_x86_64 run
by Andrew Nelson Oct. 26, 2022

Oct. 26, 2022

I've been experimenting (on my fork) this afternoon with a CI run on cirrus-ci for musllinux_x86_64. What do people think of adding such a run to the scipy CI suite? You can see the PR at: https://github.com/andyfaff/scipy/pull/37 Example run at: https://cirrus-ci.com/task/4724609919483904 Part of the run is building numpy (and caching it). Having a test run would be part of creating a musllinux wheel for release, and would imply that we support the platform. See https://github.com/scipy/scipy/issues/15628 for more context. When I run the tests locally there are some failures. Andrew.

4 5

Alternative to the deprecated extradoc parameter
by Kevin Sheppard Oct. 26, 2022

Oct. 26, 2022

extradoc in rv_continuus has been visibly deprecated in 1.9. It is used extensively in some older code in statsmodels. I would like to ensure that statsmodels doesn't fall over when 1.11 is released. Is there a clear alternative way to replace extradoc? Thanks, Kevin

2 1

Not releasing sdists on PyPI
by Andrew Nelson Oct. 25, 2022

Oct. 25, 2022

My position in the past has been that it made sense to release sdists on PyPI, for completeness of historical record. However, recently I've become a lot more aware of people trying to pip install scipy and it failing to work for various reasons. For example on 32 bit windows pip will try to download the sdist (as there's no wheel available) and start building the wheel. The wheel build fails because they don't have the setup to build scipy. We then see those failed build attempts as issues on GH or stackoverflow. Should we stop releasing sdists to stop this issue cropping up? We could always make the sdist available on Github releases. THe main argument against this would be the various *NIX platforms that do have build chains (musllinux, ppc64, etc) capable of building scipy, would no longer be able to pip install from source via PyPI, but would manually have to download the source from elsewhere. -- _____________________________________ Dr. Andrew Nelson _____________________________________

3 2