Mailman 3 April 2022 - SciPy-Dev

Deprecate scipy.misc and Add scipy.datasets submodule
by Anirudh Dagar Jan. 7, 2024

Jan. 7, 2024

Hi all, I just filed https://github.com/scipy/scipy/pull/15607 after an offline discussion with Ralf earlier this month. The summary of that PR is: This is a good time to get rid of the `scipy.misc` submodule since it houses only a handful of functions. As of today, most of the methods have already been deprecated from the `misc` module. They have generally moved under some other submodule. In this case, it is best to create a `scipy.datasets` submodule that can be home to the dataset functions (ascent, face, ecg). Adding a datasets submodule is not a new idea, and was proposed a couple of years ago by Warren (see https://github.com/scipy/scipy/pull/8707), but unfortunately, it never went ahead. Adding this submodule is not only sensible from an organizational/structural point of view but will also help reduce the wheel size of SciPy package. This is possible if we decouple the dataset files from the SciPy repository making use of Pooch to download and cache datasets kept in separate repos. See the PR for more details. I also filed https://github.com/scipy/scipy/issues/15608 Out of the 5 methods in `scipy.misc`, three dataset functions can be moved to this datasets submodule and will slowly be deprecated from misc. The question about the other two methods i.e (derivative, central_diff_weights) is up for discussion and I'd like to understand what will be the best way to deprecate them. Should we move them to a submodule like integrate (keeping derivative makes sense to me)? On the other hand, considering these two methods are not used extensively so one possibility is to deprecate both of them completely and remove them from SciPy. Please share your opinion/concerns, and comment if you see a potential issue with this. Cheers, Anirudh

4 10

SciPy Community Meeting on Wed, April 13 at 8pm UTC
by Melissa Mendonça Oct. 18, 2022

Oct. 18, 2022

Hello, all! The next SciPy Community Meeting will happen this week, on* Wednesday March 16th, at 8pm UTC*. Here's the hackmd we are using to take notes: https://hackmd.io/pyhudrC5TgSwdJtpPldHgQ If you wish to join on Zoom, use this (updated) link: https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09 <https://us06web.zoom.us/j/6345425936> Our community calendar is now also available here: https://scientific-python.org/calendars/ Cheers, Melissa

1 12

New feature: fitting distributions to censored data.
by Warren Weckesser Aug. 29, 2022

Aug. 29, 2022

Hey folks, A new enhancement for fitting probability distributions to censored data by maximum likelihood estimation is in progress in https://github.com/scipy/scipy/pull/13699 This is a feature in the statistics roadmap, and is part of the work for the CZI EOSS cycle 1 grant. Censored data is data where the true value of the measurement is unknown, but there is a known bound. The common terminology is: * left-censored: The true value is greater than some known bound. * right-censored: The true value is less than some known bound. * interval-censored: The true value is between known bounds. (By allowing the bounds to be +inf and -inf, we can think of all these cases as interval-censored.) In the PR, a new class, `CensoredData`, is defined to represent censored data. The `fit` method of the univariate continuous distributions is updated to accept an instance of `CensoredData`. The `CensoredData` class constructor accepts two arrays, `lower` and `upper`, to hold the known bounds of the data. (If `lower[i] == upper[i]`, it means that data value is not censored.) Because it is quite common for data to be encountered where the only censored values in the data set are either all right-censored or all left-censored, two convenience methods, `CensoredData.right_censored(x, censored)` and `CensoredData.left_censored(x, censored)`, are provide to create a `CensoredData` instance from an array of values and a corresponding boolean array that indicates if the value is censored. Here's a quick example, with data from https://www.itl.nist.gov/div898/handbook/apr/section4/apr413.htm. The data table shows 10 regular values and 10 right-censored values. The results reported there for fitting the two-parameter Weibull distribution (location fixed at 0) to that data are shape = 1.7208 scale = 606.5280 Here's the calculation using the proposed API in SciPy: In [55]: from scipy.stats import weibull_min, CensoredData Create the `CensoredData` instance: In [56]: x = np.array([54, 187, 216, 240, 244, 335, 361, 373, 375, 386] + [500]*10) In [57]: data = CensoredData.right_censored(x, x == 500) In [58]: print(data) CensoredData(20 values: 10 not censored, 10 right-censored) Fit `weibull_min` (with the location fixed at 0) to the censored data: In [59]: shape, loc, scale = weibull_min.fit(data, floc=0) In [60]: shape Out[60]: 1.720797180719942 In [61]: scale Out[61]: 606.527565269458 Matt Haberland has already suggested quite a few improvements to the PR. Additional comments would be appreciated. Warren

4 7

Consolidation of Stats Warnings/Errors
by Matt Haberland May 26, 2022

May 26, 2022

Hi team, gh-14362 <https://github.com/scipy/scipy/issues/13462> reported that many hypothesis test-specific warnings had been added to the public API of `scipy.stats` without discussion on the mailing list. It recommended keeping things cleaner by creating a few superclass warnings to be used in the public API, and any existing test-specific warnings would inherit from these. gh-15923 <https://github.com/scipy/scipy/pull/15923> performs this consolidation, removing class-specific warnings like `F_onewayConstantInputWarning`, `PearsonRConstantInputWarning`, and `SpearmanRConstantInputWarning` from the documentation and adding a public `ConstantInputWarning` in their place. It also adds `NearConstantInputWarning` (in place of, e..g., `PearsonRNearConstantInputWarning`), `DegenerateDataWarning` (a superclass of the other new warnings), and `FitError` (to serve as a superclass for several errors raised when fitting distributions to data). If you're interested, please join the discussion in gh-15923. Thanks! Matt Haberland -- Matt Haberland Assistant Professor BioResource and Agricultural Engineering 08A-3K, Cal Poly

1 1

Proposed Schedule for 1.9.0 Release
by Tyler Reddy May 26, 2022

May 26, 2022

Hi, Feedback is welcome on the draft schedule for the 1.9.0 release process below. Meson build system seems like a big/welcome change to have nice and stable for that release I think? Some of the dates may be +/- 1-2 days because of the US Holiday weekend in late May. - May 26/2022: branch 1.9.x (5 full weekends from now) - May 29/2022: rc1 - June 11/2022: rc2 (if needed) - June 20/2022: final release Best wishes, Tyler

3 4

under-maintained SciPy submodules and what to do about them
by Ralf Gommers April 25, 2022

April 25, 2022

Hi all, This is an email I've wanted to write for a while. A few things came together recently to finally actually write it: - The contributors not getting a review on their scipy.integrate PRs and asking about those: https://mail.python.org/archives/list/scipy-dev@python.org/thread/QDYCECIJD… - Same for Gang Zhao's emails on MINRES and other sparse.linalg improvements - NumFOCUS finally completing the move to Open Collective for all its sponsored projects (yay!): https://opencollective.com/scipy - The SciPy-project-funds-funded Meson work nearing its end: https://mail.python.org/archives/list/scipy-dev@python.org/message/4S43BYHD…. (I need to write a separate update; tl;dr is that it's going well and we should be able to do the 1.9.0 release with Meson). - A new proposal for use of SciPy funds - related to the topic of this PR - that I'd like to see / co-write. We have certain submodules and types of PRs that don't get the attention they need, because we're short of maintainers with the required bandwidth or skills. If I'd rank the pain points, high on that list are: scipy.integrate scipy.sparse.linalg scipy.interpolate scipy.signal scipy.special anything that requires touching Fortran code There are other things that I'd have put on the list if we hadn't gotten funding for them. As most of you are probably aware of, the overall funding situation for projects at the core of the PyData ecosystem has been improving, and part of that has been coming to SciPy. Past grants include: - the CZI EOSS cycle 1 grant focused on scipy.stats, (Warren, Matt, Nicholas) - a number of Small Development Grants from NumFOCUS, (see https://numfocus.org/programs/small-development-grants) - support from AMD for work on multiple-array-type support (to Quansight Labs - Greg, Peter, Ralf & broader team) Recently started or about-to-start grants are: - CZI EOSS cycle 4 grant (2 year duration) which funded the Contributor Experience Lead positions across SciPy, Pandas, Matplotlib and NumPy. (Melissa) - A NASA grant (3 year duration) which just was finalized earlier this week (details forthcoming) across NumPy, Pandas, scikit-learn and SciPy (time for Matt, Tyler, Pamphile, and Ralf for SciPy). This includes dedicated time for CI improvements & releases. Finally, I've also been able to create a significant amount of (not externally funded) day job time for SciPy work for multiple people at Quansight Labs over the past couple of years. Not quite a grant, but from the perspective of the SciPy project I'd say that that is also external support from industry. I could write a long essay on the above, but: back to the topic of under-maintained submodules & Fortran code. To me, this is a bottleneck for SciPy development and it's not a good situation that enthusiastic new contributors are submitting PRs for new functionality or rewrites that would improve existing functionality and they are sitting in the PR queue without review. This is not a problem that's very easy to resolve. Sometimes those contributors are just interested in those PRs, and they need a maintainer to look at them. Perhaps those contributors would like to become maintainers, which would be great - but it also means that if we don't review their PRs and interact with them, they are unlikely to get there and the cost of not reviewing is even higher. The best way of solving these bottlenecks is more time to work on SciPy for maintainers with the skills and motivation to review those PR, onboard new maintainers, and solve the most hairy/high-prio open issues. Which gets us to: how to get that bandwidth. In many cases that is difficult, because maintainers may have a job, or a degree to finish, or a family, or other hobbies or life that take priority - or all of those. That said, there may be maintainers (or ex-maintainers who we can onboard quickly again) that do have potential availability, and where money can help. We are in a different place from a few years ago, when there were no grants, no industry support, and the sum total of donations to SciPy was a few thousand dollars. Right now the unrestricted funds available to the project are on the order of $70,000 - see https://opencollective.com/scipy - mostly thanks to support from Tidelift (~$2,500/month). So I wanted to open the conversation about what our priorities are. I put my list of submodules (+ Fortran) higher up in this email. I'd love to see what others think is most important. If it's not reflected on our roadmap ( http://scipy.github.io/devdocs/dev/roadmap.html, http://scipy.github.io/devdocs/dev/roadmap-detailed.html) we may want to add those items there. In summary, I'd love to hear: 1. feedback on what type of work is most urgent, and submodules, code or anything else that needs attention 2. whether "more bandwidth from maintainers with the right skills" is the right way to spend project funds 3. any other thoughts regarding pain points for SciPy development or anything else I wrote in this email Cheers, Ralf

5 4

Can anyone help with 10 minute interview on your favorite editor?
by Matthew Brett April 21, 2022

April 21, 2022

Hi, Summary: are there any happy users of Visual Studio Code, or PyCharm, or Sublime Text, or Spyder, who would consider doing a 10 minute informal video interview for a free online course I'm running? If so, I'd be hugely grateful if you could email me! Detail I'm teaching a free open course on Python and brain imaging, in fact, funded by a CZI grant: https://nipraxis.org/ The students are currently at the stage where they have to choose a text editor. I'm trying to persuade them to take some time to really investigate their options and think about which editor to invest in: https://textbook.nipraxis.org/choosing_editor.html I think these decisions are best made when you've had a chance to see someone who really likes the relevant editor talk about why they like it. We (the tutors) nearly all use Vim, so we have that covered, and Emacs, but we don't have anyone who uses the other editors we've suggested as options: Visual Studio Code PyCharm Sublime text Spyder Notepad++ Is there anyone out there who would consider doing a 10 minute very informal chatty interview about their favorite editor? Here are the questions I had as a start, but you can talk about anything you like: https://docs.google.com/document/d/1_-c4ZV8Rkt6jIQByYnZoEVzJiARuY20OGOusAXU… Please do email me if you can do it... Cheers, Matthew

3 4

GSoC 2022 proposal - XSparse (Rewrite of PyData/Sparse)
by Bharath K K April 21, 2022

April 21, 2022

Hi, I'm interested in working on the XSparse library, which was mentioned in the SciPy GSoC ideas page. I was able to work on the proposal for the same under the guidance of my mentor. Any suggestion to improve the proposal is greatly appreciated. Thanks in advance. Proposal: https://docs.google.com/document/d/1EBxOPJ4QlbyL2CZdkcpdotSKZ7WsIbl2DlX40bw… Regards, Bharath

6 7

Change of behaviour on MultinomialQMC.random
by Pamphile Roy April 19, 2022

April 19, 2022

Hi everyone, We have been working on MultinomialQMC.random and noticed a bug and fixing it creates a backward incompatibility. https://github.com/scipy/scipy/pull/15945 Currently, MultinomialQMC.random has one parameter which is supposed to be the number of sample and is described as is. But it is in fact the number of trials. Hence in the PR, we propose to move the number of trials to the class initialization and have n actually return n samples. Right now to sample n you would need to call n times the function. We believe that the impact on this incompatibility will be very low and limited as the class is new and from a quick search did not return many occurrences. The API will be cleaner and less prone to errors. We also think that for a current user, the update path would be easier versus having a multistage deprecation strategy to remove the keyword from the method and move it over the initialization. Please join the discussion on the PR. Thank you for your thoughts. Cheers, Pamphile

1 0

Contribution proposal: Wrapper for fortran tridiagonal solvers similar to scipy.linalg.solve_triangular
by J Roth April 19, 2022

April 19, 2022

Hi, I'm a PhD student at UC Berkeley and in my research I often use tridiagonal solvers similar to those found in LAPACK (dgtsv for example). I would like to contribute a function similar to scipy.linalg.solve_triangular that accepts a tridiagonal matrix from the user and calls the appropriate LAPACK function. If this feature seems appropriate for scipy please let me know and I will open a pull request on github. Cheers! Jack

3 3