Mailman 3 February 2023 - SciPy-Dev

RFC: decouple doctesting from refguide-check
by Evgeni Burovski 23 Feb '24

23 Feb '24

Hi, Code examples in the SciPy and NumPy documentation are doctested, with a modified doctesting machinery which understands floating point, numpy formatting and some details of our documentation and API. Our modified doctesting machinery is buried in the refguide-check utility, which also does several unrelated things, all of which are tightly coupled to each other and to the libraries themselves. It has been sometimes described as hard to understand or work with or extend due to this tight coupling and a lack of a dedicated test suite. The numpy and scipy versions of the utility are both vendored, and the numpy version has diverged from the scipy version somewhat. Following a discussion in https://github.com/numpy/numpy/issues/21070 I did a small experiment to decouple the doctesting into a separate package, so that it's easier to consolidate the two versions. Plus, a separate repo is generally easier to maintain, configure, and possibly extend or adopt to other projects. The work-in-progress result is here: https://github.com/ev-br/scpdt It currently can run the full doctesting of the scipy API documentation (docstrings of objects) --- see https://github.com/ev-br/scpdt/pull/33 (the log of a test run with warnings turned to errors is in the GH actions: https://github.com/ev-br/scpdt/runs/6743881766?check_suite_focus=true). Note that it shows, among other things, a couple of deprecation warnings our docs have accumulated :-). The API of the tool closely follows that of the standard library doctest module and provides (nearly) drop-in replacements for doctest checking, parsing, finding and running. Various configuration options for our modifications are collected into a single bag object which is internally passed around. This way, it's user-configurable all the way from a plain standard doctest module behavior to what refguide-check does now. There are a couple of wrinkles to iron out; overall it does what refguide-check does already. One missing bit is doctesting rst or other text files, but it's coming soon. The current plan is to: - verify that the standalone version does not miss things checked by the refguide-check - plumb it through the SciPy dev interface and rip out the refguide-check bundled doctesting utilities. - Sync changes that NumPy version of refguide-check accumulated over time - Make sure it correctly tests the NumPy docs, too. - Better document the internals, there is currently only a readme file. If someone's interested to join me working on these, great, the more the merrier :-). I think it could make sense to move the tool's repository to the scipy github org (or maybe even numpy org?). I'm offering to maintain it regardless of the location. Thoughts? Cheers, Evgeni

5 8

Deprecate scipy.misc and Add scipy.datasets submodule
by Anirudh Dagar 07 Jan '24

07 Jan '24

Hi all, I just filed https://github.com/scipy/scipy/pull/15607 after an offline discussion with Ralf earlier this month. The summary of that PR is: This is a good time to get rid of the `scipy.misc` submodule since it houses only a handful of functions. As of today, most of the methods have already been deprecated from the `misc` module. They have generally moved under some other submodule. In this case, it is best to create a `scipy.datasets` submodule that can be home to the dataset functions (ascent, face, ecg). Adding a datasets submodule is not a new idea, and was proposed a couple of years ago by Warren (see https://github.com/scipy/scipy/pull/8707), but unfortunately, it never went ahead. Adding this submodule is not only sensible from an organizational/structural point of view but will also help reduce the wheel size of SciPy package. This is possible if we decouple the dataset files from the SciPy repository making use of Pooch to download and cache datasets kept in separate repos. See the PR for more details. I also filed https://github.com/scipy/scipy/issues/15608 Out of the 5 methods in `scipy.misc`, three dataset functions can be moved to this datasets submodule and will slowly be deprecated from misc. The question about the other two methods i.e (derivative, central_diff_weights) is up for discussion and I'd like to understand what will be the best way to deprecate them. Should we move them to a submodule like integrate (keeping derivative makes sense to me)? On the other hand, considering these two methods are not used extensively so one possibility is to deprecate both of them completely and remove them from SciPy. Please share your opinion/concerns, and comment if you see a potential issue with this. Cheers, Anirudh

4 10

Next newcomers' meeting: Friday, July 22 at 8pm UTC
by Melissa Mendonça 17 Aug '23

17 Aug '23

Hello, folks! This friday, *July 22 at 8pm UTC* we are having another newcomers' meeting. All are welcome to join and ask questions about contributing, setting up your development environment, PRs or issues you may want to work on, or just chat. Join us on Zoom <https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09>, and if you need more information check out some notes from our previous meeting here: https://hackmd.io/@melissawm/scipy-newcomers Hope to see you around! - Melissa

1 14

Possibly reduce duplication between `signal` and `ndimage` filtering routines
by Evgeni Burovski 22 Apr '23

22 Apr '23

Hi, In `scipy.signal` and `scipy.ndimage`, we ship a set of almost-if-not-completely overlapping functionality. For instance, - `signal.order_filter` has a counterpart, `ndimage.rank_filter`; - `signal.median_filter` can be implemented in terms of the rank filter In fact, applying the patch below (the patch is based on the CuPy implementation of `cupyx.signal`, see https://github.com/cupy/cupy/blob/v11.5.0/cupyx/scipy/signal/_signaltools.p…), and running `$ python dev.py test -s signal`, I only get three failures in `medfilt`, related to - object arrays of `Decimal` instances - `longdouble` arrays. So, assuming we can deprecate longdoubles and decimals in these filters, we could potentially remove some 700 lines of obscure C code in https://github.com/scipy/scipy/blob/main/scipy/signal/_lfilter.c.in Thoughts? Evgeni P.S. The patch to check the mapping between signal.{order,median}_filter and ndimage.rank_filter: ```diff diff --git a/scipy/signal/_signaltools.py b/scipy/signal/_signaltools.py index 31baba2421..2f00ead3a2 100644 --- a/scipy/signal/_signaltools.py +++ b/scipy/signal/_signaltools.py @@ -1492,7 +1492,15 @@ def order_filter(a, domain, rank): if (dimsize % 2) != 1: raise ValueError("Each dimension of domain argument " "should have an odd number of elements.") - return _sigtools._order_filterND(a, domain, rank) + + result = _sigtools._order_filterND(a, domain, rank) + + import scipy.ndimage as ndimage + import numpy.testing as testing + res_ndi = ndimage.rank_filter(a, rank, footprint=domain, mode='constant') + testing.assert_equal(result, res_ndi) + + return result def medfilt(volume, kernel_size=None): @@ -1555,7 +1563,17 @@ def medfilt(volume, kernel_size=None): numels = np.prod(kernel_size, axis=0) order = numels // 2 - return _sigtools._order_filterND(volume, domain, order) + result = _sigtools._order_filterND(volume, domain, order) + + + import scipy.ndimage as ndimage + import numpy.testing as testing + size = np.prod(kernel_size) + res_ndi = ndimage.rank_filter(volume, size // 2, size=kernel_size, + mode='constant') + testing.assert_equal(result, res_ndi) + + return result def wiener(im, mysize=None, noise=None): ```

2 2

Basic spherical statistics in scipy
by Daniel Schmitz 10 Apr '23

10 Apr '23

Hi everyone, recently questions came up with regards to statistics on the unit sphere in scipy. Issue 12041 <https://github.com/scipy/scipy/issues/12041> raises the question if spherical mean/variance should be added as descriptive statistics similar to circmean/circvar <https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.circvar.ht…> . The canonical reference for all things related to statistics on the unit sphere is "Directional Statistics" by Mardia & Jupp. Another important basic functionality is the generation of random samples on the unit sphere: discussion was started in this issue <https://github.com/scipy/scipy/issues/16205> . Any strong opinion against these? I think at least the random number generation is reimplemented every day around the world (done so myself at least 2 times) and would strongly benefit from being part of scipy. Best

4 9

ENH: stats: add ecdf function
by Matt Haberland 31 Mar '23

31 Mar '23

Hi team, gh-17431 requested the addition of an ECDF (empirical cumulative distribution function) in SciPy. gh-18035 would add one. Please join the discussion at: https://github.com/scipy/scipy/pull/18035 if you're interested. Thanks! Matt -- Matt Haberland Assistant Professor BioResource and Agricultural Engineering 08A-3K, Cal Poly

1 1

SciPy Community Meeting on Wed, November 2 at 8pm UTC
by Melissa Mendonça 22 Mar '23

22 Mar '23

Hello, all! The next SciPy Community Meeting will happen this week, on* Wednesday November 2 at 8pm UTC*. Here's the hackmd we are using to take notes: https://hackmd.io/pyhudrC5TgSwdJtpPldHgQ If you wish to join on Zoom, use this link: https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09 Our community calendar is also available here: https://scientific-python.org/calendars/ Cheers, Melissa

1 8

Support for complex data in stats.pearsonr
by Christopher Cowden 28 Feb '23

28 Feb '23

Hi, A recent issue [1] brought up the idea of adding support for complex data in stats.pearsonr. It seems like this would be a useful feature, especially for folks working in signal processing. It wouldn't be too difficult to implement, see [2]. The current estimation of p-values, however, seems to underestimate the significance. That is, the error rate tends to be lower than the p-value. It would be possible to introduce a permutation test to estimate the p-value of the correlation between complex numbers. Is this something worth pursuing? What do you think of introducing an alternative p-value estimation method? Cheers, Chris [1] https://github.com/scipy/scipy/issues/17518 [2] https://en.wikipedia.org/wiki/Complex_random_variable

3 3

Boost Submodule Replaced with Boost.Math Standalone
by Matt Haberland 28 Feb '23

28 Feb '23

Hi Team, gh-17432 (https://github.com/scipy/scipy/pull/17432) just merged, creating a new `boost_math` submodule and removing the old `boost` submodule. After pulling the latest commits from main, please run: git submodule update --init before building from source. `git status` may reveal some new untracked files in the repo; you're welcome to delete those manually. That should be all that's needed, but if you have trouble, please let us know in gh-17432. Thanks! Matt -- Matt Haberland Assistant Professor BioResource and Agricultural Engineering 08A-3K, Cal Poly

1 0

Seed in examples
by Pamphile Roy 22 Feb '23

22 Feb '23

Hi everyone, The current guidelines around seeding in examples is to not use plain >>> rng = np.random.default_rng() This is great to present a clean HTML without seed that users would copy past. In the background we add a seed when running doctests. This part is painful for contributors as we either need to fail the CI first or go get the seed to have the correct output. In the following PR, I propose some numpydoc magic that would ease our job. We would be able to put a seed, but the HTML would still be free of it. Best of both worlds. https://github.com/scipy/scipy/pull/17945 Feel free to join the discussion. I will send a follow up if this gets merge to give updated guidelines. The current behaviour stays in place regardless if you would not want to care. Cheers, Pamphile

1 1