Mailman 3 March 2023 - SciPy-Dev

RFC: decouple doctesting from refguide-check
by Evgeni Burovski Feb. 23, 2024

Feb. 23, 2024

Hi, Code examples in the SciPy and NumPy documentation are doctested, with a modified doctesting machinery which understands floating point, numpy formatting and some details of our documentation and API. Our modified doctesting machinery is buried in the refguide-check utility, which also does several unrelated things, all of which are tightly coupled to each other and to the libraries themselves. It has been sometimes described as hard to understand or work with or extend due to this tight coupling and a lack of a dedicated test suite. The numpy and scipy versions of the utility are both vendored, and the numpy version has diverged from the scipy version somewhat. Following a discussion in https://github.com/numpy/numpy/issues/21070 I did a small experiment to decouple the doctesting into a separate package, so that it's easier to consolidate the two versions. Plus, a separate repo is generally easier to maintain, configure, and possibly extend or adopt to other projects. The work-in-progress result is here: https://github.com/ev-br/scpdt It currently can run the full doctesting of the scipy API documentation (docstrings of objects) --- see https://github.com/ev-br/scpdt/pull/33 (the log of a test run with warnings turned to errors is in the GH actions: https://github.com/ev-br/scpdt/runs/6743881766?check_suite_focus=true). Note that it shows, among other things, a couple of deprecation warnings our docs have accumulated :-). The API of the tool closely follows that of the standard library doctest module and provides (nearly) drop-in replacements for doctest checking, parsing, finding and running. Various configuration options for our modifications are collected into a single bag object which is internally passed around. This way, it's user-configurable all the way from a plain standard doctest module behavior to what refguide-check does now. There are a couple of wrinkles to iron out; overall it does what refguide-check does already. One missing bit is doctesting rst or other text files, but it's coming soon. The current plan is to: - verify that the standalone version does not miss things checked by the refguide-check - plumb it through the SciPy dev interface and rip out the refguide-check bundled doctesting utilities. - Sync changes that NumPy version of refguide-check accumulated over time - Make sure it correctly tests the NumPy docs, too. - Better document the internals, there is currently only a readme file. If someone's interested to join me working on these, great, the more the merrier :-). I think it could make sense to move the tool's repository to the scipy github org (or maybe even numpy org?). I'm offering to maintain it regardless of the location. Thoughts? Cheers, Evgeni

5 8

Deprecate scipy.misc and Add scipy.datasets submodule
by Anirudh Dagar Jan. 7, 2024

Jan. 7, 2024

Hi all, I just filed https://github.com/scipy/scipy/pull/15607 after an offline discussion with Ralf earlier this month. The summary of that PR is: This is a good time to get rid of the `scipy.misc` submodule since it houses only a handful of functions. As of today, most of the methods have already been deprecated from the `misc` module. They have generally moved under some other submodule. In this case, it is best to create a `scipy.datasets` submodule that can be home to the dataset functions (ascent, face, ecg). Adding a datasets submodule is not a new idea, and was proposed a couple of years ago by Warren (see https://github.com/scipy/scipy/pull/8707), but unfortunately, it never went ahead. Adding this submodule is not only sensible from an organizational/structural point of view but will also help reduce the wheel size of SciPy package. This is possible if we decouple the dataset files from the SciPy repository making use of Pooch to download and cache datasets kept in separate repos. See the PR for more details. I also filed https://github.com/scipy/scipy/issues/15608 Out of the 5 methods in `scipy.misc`, three dataset functions can be moved to this datasets submodule and will slowly be deprecated from misc. The question about the other two methods i.e (derivative, central_diff_weights) is up for discussion and I'd like to understand what will be the best way to deprecate them. Should we move them to a submodule like integrate (keeping derivative makes sense to me)? On the other hand, considering these two methods are not used extensively so one possibility is to deprecate both of them completely and remove them from SciPy. Please share your opinion/concerns, and comment if you see a potential issue with this. Cheers, Anirudh

4 10

Next newcomers' meeting: Friday, July 22 at 8pm UTC
by Melissa Mendonça Aug. 17, 2023

Aug. 17, 2023

Hello, folks! This friday, *July 22 at 8pm UTC* we are having another newcomers' meeting. All are welcome to join and ask questions about contributing, setting up your development environment, PRs or issues you may want to work on, or just chat. Join us on Zoom <https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09>, and if you need more information check out some notes from our previous meeting here: https://hackmd.io/@melissawm/scipy-newcomers Hope to see you around! - Melissa

1 14

Possibly reduce duplication between `signal` and `ndimage` filtering routines
by Evgeni Burovski April 21, 2023

April 21, 2023

Hi, In `scipy.signal` and `scipy.ndimage`, we ship a set of almost-if-not-completely overlapping functionality. For instance, - `signal.order_filter` has a counterpart, `ndimage.rank_filter`; - `signal.median_filter` can be implemented in terms of the rank filter In fact, applying the patch below (the patch is based on the CuPy implementation of `cupyx.signal`, see https://github.com/cupy/cupy/blob/v11.5.0/cupyx/scipy/signal/_signaltools.p…), and running `$ python dev.py test -s signal`, I only get three failures in `medfilt`, related to - object arrays of `Decimal` instances - `longdouble` arrays. So, assuming we can deprecate longdoubles and decimals in these filters, we could potentially remove some 700 lines of obscure C code in https://github.com/scipy/scipy/blob/main/scipy/signal/_lfilter.c.in Thoughts? Evgeni P.S. The patch to check the mapping between signal.{order,median}_filter and ndimage.rank_filter: ```diff diff --git a/scipy/signal/_signaltools.py b/scipy/signal/_signaltools.py index 31baba2421..2f00ead3a2 100644 --- a/scipy/signal/_signaltools.py +++ b/scipy/signal/_signaltools.py @@ -1492,7 +1492,15 @@ def order_filter(a, domain, rank): if (dimsize % 2) != 1: raise ValueError("Each dimension of domain argument " "should have an odd number of elements.") - return _sigtools._order_filterND(a, domain, rank) + + result = _sigtools._order_filterND(a, domain, rank) + + import scipy.ndimage as ndimage + import numpy.testing as testing + res_ndi = ndimage.rank_filter(a, rank, footprint=domain, mode='constant') + testing.assert_equal(result, res_ndi) + + return result def medfilt(volume, kernel_size=None): @@ -1555,7 +1563,17 @@ def medfilt(volume, kernel_size=None): numels = np.prod(kernel_size, axis=0) order = numels // 2 - return _sigtools._order_filterND(volume, domain, order) + result = _sigtools._order_filterND(volume, domain, order) + + + import scipy.ndimage as ndimage + import numpy.testing as testing + size = np.prod(kernel_size) + res_ndi = ndimage.rank_filter(volume, size // 2, size=kernel_size, + mode='constant') + testing.assert_equal(result, res_ndi) + + return result def wiener(im, mysize=None, noise=None): ```

2 2

Basic spherical statistics in scipy
by Daniel Schmitz April 10, 2023

April 10, 2023

Hi everyone, recently questions came up with regards to statistics on the unit sphere in scipy. Issue 12041 <https://github.com/scipy/scipy/issues/12041> raises the question if spherical mean/variance should be added as descriptive statistics similar to circmean/circvar <https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.circvar.ht…> . The canonical reference for all things related to statistics on the unit sphere is "Directional Statistics" by Mardia & Jupp. Another important basic functionality is the generation of random samples on the unit sphere: discussion was started in this issue <https://github.com/scipy/scipy/issues/16205> . Any strong opinion against these? I think at least the random number generation is reimplemented every day around the world (done so myself at least 2 times) and would strongly benefit from being part of scipy. Best

4 9

Dropping support for Gitpod and our Docker image builds
by Pamphile Roy March 31, 2023

March 31, 2023

Hi everyone, You might have seen that Docker made a few announcements related to the free team plans. These decisions affect us as we are relying on DockerHub to host images for GitPod. Now, Docker reverted their decision to remove the free tier. Besides the uncertain future of this offering, Gitpod has been failing for a while on main and has proven to be challenging to maintain. https://github.com/scipy/scipy/actions/workflows/gitpod.yml NumPy already decided to drop support for Gitpod and as a replacement suggest users to use GitHub’s CodeSpaces. https://github.com/codespaces/ We had a brief discussion during our last community meeting and also on Slack and we seem to agree that we should follow the same route. I will keep this communication short and refer to NumPy’s communication and actions if you want more details: The discussion on NumPy’s mailing list: https://mail.python.org/archives/list/numpy-discussion@python.org/thread/SS… And the PR which followed: https://github.com/numpy/numpy/pull/23444 If there are no strong objections, I would propose a similar PR. Thanks in advance and have a good WE! Cheers, Pamphile

1 0

Initial kw behaviour for cumulative_trapezoid
by Praveer Nidamaluri March 31, 2023

March 31, 2023

Hello everyone! `integrate.cumulative_trapezoid` performs a cumulative integration of fixed samples. The function provides an `initial` kw that can be used to add the starting value for the result. The initial value is just prepended to the result array and not summed to the result. I think this is confusing and should be updated so that the initial value is prepended to the array AND summed to the array. This was previously discussed in #125, where the concluding thought was: "initial is just a mechanism to keep the output and the input the same length. This has more to do with the "initial" region (i.e. the area under the curve between the non-existent previous value of x and x[0])." However, cumulative_trapezoid is a cumulative sum of the areas. If we prepend the area from the "initial" region, we would necessarily have to add it to the sum of the subsequent areas. By prepending the value it is now now an initial condition for the integration. Consider the following practical example: - Let y be sampled velocity data of an oscillating object. We can cumulatively integrate to get the displacement of the object. We know the starting position (initial value) is 10m. Without the initial value input, the displacement data will show oscillations around 0m. With the initial value, the displacement should ideally show oscillations around 10m: eg: [10, 8, 5, 7, 10, 12, 13, 11, 5, 10] - But, with the cumulative_trapezoid implementation, the first value would be 10m, and the object would then be oscillating around 0m, which is incorrect, eg: [10, -2, -5, -3, 0, 2, 3, 1, -5, 0]. - This behaviour confused me as a user. It can be fixed by prepending and adding the initial value. Regarding backward compatibility, `initial` would likely be used as "initial=None" or "initial=0". The results for cumulative_trapezoid for these values of `initial` would be identical even if the proposed update is made. For context, this discussion came up again in #18151 for implementation of `initial` in the proposed `cumulative_simpson` function. What are your thoughts on this? Thank you, Praveer

2 4

ENH: stats: add ecdf function
by Matt Haberland March 30, 2023

March 30, 2023

Hi team, gh-17431 requested the addition of an ECDF (empirical cumulative distribution function) in SciPy. gh-18035 would add one. Please join the discussion at: https://github.com/scipy/scipy/pull/18035 if you're interested. Thanks! Matt -- Matt Haberland Assistant Professor BioResource and Agricultural Engineering 08A-3K, Cal Poly

1 1

integrate.simpson behaviour for even number of samples
by Andrew Nelson March 28, 2023

March 28, 2023

At the moment `integrate.simpson` has three different modes for how to deal with an even number of samples. This is specified using the `even` keyword. Each of those modes use the trapezoid rule. `first` and `last` use the trapzeoid rule on the last/first interval respectively, and simpson on the remaining intervals. `avg` takes the mean of those two. It was pointed out in #18151 that there's a different way of dealing with the last/first interval for an even number of points that retains the quadratic nature of the rule. See https://en.wikipedia.org/wiki/Simpson%27s_rule#Composite_Simpson's_rule_for_irregularly_spaced_data . The article references Cartwright, Kenneth V. (September 2017). "Simpson's Rule Cumulative Integration with MS Excel and Irregularly-spaced Data". Journal of Mathematical Sciences and Mathematics Education. 12 (2): 1–9. Should we implement this rule in `integrate.simpson`? It should be strictly better than what we have already and may be faster? -- _____________________________________ Dr. Andrew Nelson _____________________________________

2 2

Next SciPy newcomers meeting - Friday, March 31st at 12pm UTC
by Pamphile Roy March 27, 2023

March 27, 2023

Hello all, Our next newcomers meeting happens this friday, March 31st, at 12pm UTC (noon). This is the Europe/Asia-friendly meeting. Here's the hackmd we use to take notes, which includes the meeting details: https://hackmd.io/@melissawm/scipy-newcomers And here's the link if you wish to join on Zoom: https://us06web.zoom.us/j/89085469494?pwd=dVFwK3dVczRyRUowZWZkckFtTmloUT09 Hope to see you around! Cheers, Pamphile

1 0