Mailman 3 November 2022 - SciPy-Dev

RFC: decouple doctesting from refguide-check
by Evgeni Burovski 23 Feb '24

23 Feb '24

Hi, Code examples in the SciPy and NumPy documentation are doctested, with a modified doctesting machinery which understands floating point, numpy formatting and some details of our documentation and API. Our modified doctesting machinery is buried in the refguide-check utility, which also does several unrelated things, all of which are tightly coupled to each other and to the libraries themselves. It has been sometimes described as hard to understand or work with or extend due to this tight coupling and a lack of a dedicated test suite. The numpy and scipy versions of the utility are both vendored, and the numpy version has diverged from the scipy version somewhat. Following a discussion in https://github.com/numpy/numpy/issues/21070 I did a small experiment to decouple the doctesting into a separate package, so that it's easier to consolidate the two versions. Plus, a separate repo is generally easier to maintain, configure, and possibly extend or adopt to other projects. The work-in-progress result is here: https://github.com/ev-br/scpdt It currently can run the full doctesting of the scipy API documentation (docstrings of objects) --- see https://github.com/ev-br/scpdt/pull/33 (the log of a test run with warnings turned to errors is in the GH actions: https://github.com/ev-br/scpdt/runs/6743881766?check_suite_focus=true). Note that it shows, among other things, a couple of deprecation warnings our docs have accumulated :-). The API of the tool closely follows that of the standard library doctest module and provides (nearly) drop-in replacements for doctest checking, parsing, finding and running. Various configuration options for our modifications are collected into a single bag object which is internally passed around. This way, it's user-configurable all the way from a plain standard doctest module behavior to what refguide-check does now. There are a couple of wrinkles to iron out; overall it does what refguide-check does already. One missing bit is doctesting rst or other text files, but it's coming soon. The current plan is to: - verify that the standalone version does not miss things checked by the refguide-check - plumb it through the SciPy dev interface and rip out the refguide-check bundled doctesting utilities. - Sync changes that NumPy version of refguide-check accumulated over time - Make sure it correctly tests the NumPy docs, too. - Better document the internals, there is currently only a readme file. If someone's interested to join me working on these, great, the more the merrier :-). I think it could make sense to move the tool's repository to the scipy github org (or maybe even numpy org?). I'm offering to maintain it regardless of the location. Thoughts? Cheers, Evgeni

5 8

Deprecate scipy.misc and Add scipy.datasets submodule
by Anirudh Dagar 07 Jan '24

07 Jan '24

Hi all, I just filed https://github.com/scipy/scipy/pull/15607 after an offline discussion with Ralf earlier this month. The summary of that PR is: This is a good time to get rid of the `scipy.misc` submodule since it houses only a handful of functions. As of today, most of the methods have already been deprecated from the `misc` module. They have generally moved under some other submodule. In this case, it is best to create a `scipy.datasets` submodule that can be home to the dataset functions (ascent, face, ecg). Adding a datasets submodule is not a new idea, and was proposed a couple of years ago by Warren (see https://github.com/scipy/scipy/pull/8707), but unfortunately, it never went ahead. Adding this submodule is not only sensible from an organizational/structural point of view but will also help reduce the wheel size of SciPy package. This is possible if we decouple the dataset files from the SciPy repository making use of Pooch to download and cache datasets kept in separate repos. See the PR for more details. I also filed https://github.com/scipy/scipy/issues/15608 Out of the 5 methods in `scipy.misc`, three dataset functions can be moved to this datasets submodule and will slowly be deprecated from misc. The question about the other two methods i.e (derivative, central_diff_weights) is up for discussion and I'd like to understand what will be the best way to deprecate them. Should we move them to a submodule like integrate (keeping derivative makes sense to me)? On the other hand, considering these two methods are not used extensively so one possibility is to deprecate both of them completely and remove them from SciPy. Please share your opinion/concerns, and comment if you see a potential issue with this. Cheers, Anirudh

4 10

Next newcomers' meeting: Friday, July 22 at 8pm UTC
by Melissa Mendonça 17 Aug '23

17 Aug '23

Hello, folks! This friday, *July 22 at 8pm UTC* we are having another newcomers' meeting. All are welcome to join and ask questions about contributing, setting up your development environment, PRs or issues you may want to work on, or just chat. Join us on Zoom <https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09>, and if you need more information check out some notes from our previous meeting here: https://hackmd.io/@melissawm/scipy-newcomers Hope to see you around! - Melissa

1 14

Basic spherical statistics in scipy
by Daniel Schmitz 10 Apr '23

10 Apr '23

Hi everyone, recently questions came up with regards to statistics on the unit sphere in scipy. Issue 12041 <https://github.com/scipy/scipy/issues/12041> raises the question if spherical mean/variance should be added as descriptive statistics similar to circmean/circvar <https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.circvar.ht…> . The canonical reference for all things related to statistics on the unit sphere is "Directional Statistics" by Mardia & Jupp. Another important basic functionality is the generation of random samples on the unit sphere: discussion was started in this issue <https://github.com/scipy/scipy/issues/16205> . Any strong opinion against these? I think at least the random number generation is reimplemented every day around the world (done so myself at least 2 times) and would strongly benefit from being part of scipy. Best

4 9

SciPy Community Meeting on Wed, November 2 at 8pm UTC
by Melissa Mendonça 22 Mar '23

22 Mar '23

Hello, all! The next SciPy Community Meeting will happen this week, on* Wednesday November 2 at 8pm UTC*. Here's the hackmd we are using to take notes: https://hackmd.io/pyhudrC5TgSwdJtpPldHgQ If you wish to join on Zoom, use this link: https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09 Our community calendar is also available here: https://scientific-python.org/calendars/ Cheers, Melissa

1 8

stats.quad and integrate.quad changes to support expectation computations via integration of complex valued functions
by Chris Cowden 01 Dec '22

01 Dec '22

Hi Team, gh-16907 (https://github.com/scipy/scipy/pull/16907) changes not only the limits of integration for the computation of expectation values in the vonmises distribution class (vonmises::expect), it also introduces a complex exponential (exp(i x)) as a default function to estimate the mean value. These changes close gh-16655 (https://github.com/scipy/scipy/issues/16655). Associated with this proposed change to the vonmises class, is a change to the integrate.quad method to support the integration of complex valued functions, see gh-17051 (https://github.com/scipy/scipy/pull/17051). The change there required the addition of a new argument, complex_func (which defaults to False), to allow users to opt into complex integration. If you are interested, please take a look at these PRs and let me know if you have feedback that I can incorporate. Thanks and Cheers, Chris

1 0

releasing the Weave package name on PyPI
by Ralf Gommers 01 Dec '22

01 Dec '22

Hi all, We got approached about releasing the "weave" name on PyPI, by a company that has developed a new open source project with the Weave name. They're attached to that name, and want to explore what can be done to name it as such and have it installable from PyPI as `weave`. After a chat with them on how to make this a win-win situation, we came to the conclusion that that would be possible - by donating to SciPy, so it's able to execute more projects like the two ~$10,000 ones that we have done in the past 1.5 years. Those were (1) Meson build support and (2) scipy.interpolate cleanup, both quite impactful. After discussion in the core team, the outcome was that this would be very welcome, and we'd like to go ahead with releasing the `weave` name. For some context, we deprecated `scipy.weave` a very long time ago, and then split it off as a separate `weave` package 9 years ago. There are three releases at https://pypi.org/project/weave/#history, matching the SciPy 0.15.0 - 0.17.0 releases. We haven't made significant changes in 8 years (just a couple of smaller tweaks) - and `scipy.weave` was already in really bad shape then. It also never made the transition to Python 3, and Python 2.7 is quite far beyond end-of-life now. So it really will be at most a few handfuls of users I believe. There's still ~10 downloads/day ( https://pypistats.org/packages/weave), but those are highly likely to be folks trying to install `weaveio` or something else like that. Indication: most downloads are Python 3, and `weave` doesn't support that. Unless there is a blocking issue that we overlooked during this discussion, we'd like to go ahead with this plan by December 1st. The plan for that is: - Rename to `scipy-weave`, updating https://github.com/scipy/weave/ for that change (README, metadata). - Create a new `scipy-weave` package on PyPI - Migrate old releases on PyPI - Do a new release (0.17.1, no code changes) - Transfer control of the `weave` package on PyPI; ensuring there's at least a 10 day window in which no breaking changes are made to the `weave` package yet, in case there are still any users who see this announcement and would like to switch over. If there's an issue here or if something seems missing from the above plan, please speak up. Cheers, Ralf

4 5

Adding 2-D and multi-D (two-sided) Kolmogorov-Smirnov (KS) Tests
by mail.emir.k＠gmail.com 30 Nov '22

30 Nov '22

Hi! There are several widely cited papers with hundreds of citations which use multivariate KS tests to test whether 2 samples are drawn from the same distribution. There are many results returned when searching for a python implementation (even asking for it in scipy), but no installable or maintained package. I was going to contribute a 2-D and N-D two-sided KS test to astropy.stats, but was told to check here on scipy, since it might fit better. There is a lot of use for it outside of astronomy. Please see my proposal in gh_14065 for details, https://github.com/astropy/astropy/issues/14065. The basic idea is to implement the seminal paper algorithm from Peacock 1983 with the computational improvement in Fasano & Franceshini 1987, both with 350+ citations. However, while I was writing the issue, I found a newly published code in R (with Rcpp) with an MIT license, that has just implemented the same in algorithms in 2021. Their paper only has 1 citation but it is new. https://github.com/nesscoder/fasano.franceschini.test With the MIT license and their benchmarked paper, does it make sense to use their implementation as a baseline, despite the fact that it wouldn't count as a significant paper currently? Also, since this is my first contribution to scipy, I am not sure of the workflow (how widely is an issue discussed before PR is looked at). I will open a PR if the suggestion makes sense and my proposal looks good, but would appreciate any thoughts on it. Kind regards, Emir Karamehmetoglu Postdoctoral Researcher Department of Physics and Astronomy Aarhus University, Denmark.

1 0

ENH: Add relativistic Breit-Wigner Distribution
by Albert Steppi 30 Nov '22

30 Nov '22

Hi team, gh-17505 proposes to add a new continuous distribution to stats, the Relativistic Breit-Wigner distribution, which is used in high energy physics for modeling unstable particles. The addition of this distribution was first suggested in issue gh-6414 by contributor Andrew Fowlie (andrewfowlie), who is a physicist working in the area. Discussion faltered for several years but has picked up again recently. It has been determined that this is not a reparametrization or transformation of an existing distribution. Particle physicist and developer of relevant statistical software Hans Dembinski (HDembinski) recently provided context and voiced support for its inclusion in gh-6414. Hdembinski also pointed out that there is precedent for including distributions from high energy physics due to scipy.stats.crystalball. This seemed like a fun project to work on and a good excuse to try to refresh some long-faded physics knowledge so I pieced this together. Feedback is welcome at: https://github.com/scipy/scipy/pull/17505 Thanks, Albert

2 2

PyData Global sprint - Dec 3
by Melissa Mendonça 29 Nov '22

29 Nov '22

Hi folks! The PyData global conference is happening this week, Dec 1-3, and the organizers approached us about a sprint on *Saturday, December 3* - if you are available and interested in mentoring, you can choose a time slot that works for you. We are aiming for a sprint that goes from 2pm-8pm UTC and you don't need to be available for the entire duration - I will only be available after 5pm UTC, for example. If you are interested, please reach out (you can write to me in private at melissawm(a)gmail.com) or on slack. We have a sprint guide here ( https://hackmd.io/@melissawm/scipy-sprint) that can help you during the event, and you don't need to be a maintainer to join. Most questions involve triaging and clarifying issues or setting up the participants' dev environment and the PR workflow. Cheers! - Melissa

1 0