Mailman 3 February 2022 - SciPy-Dev

Deprecate scipy.misc and Add scipy.datasets submodule
by Anirudh Dagar 07 Jan '24

07 Jan '24

Hi all, I just filed https://github.com/scipy/scipy/pull/15607 after an offline discussion with Ralf earlier this month. The summary of that PR is: This is a good time to get rid of the `scipy.misc` submodule since it houses only a handful of functions. As of today, most of the methods have already been deprecated from the `misc` module. They have generally moved under some other submodule. In this case, it is best to create a `scipy.datasets` submodule that can be home to the dataset functions (ascent, face, ecg). Adding a datasets submodule is not a new idea, and was proposed a couple of years ago by Warren (see https://github.com/scipy/scipy/pull/8707), but unfortunately, it never went ahead. Adding this submodule is not only sensible from an organizational/structural point of view but will also help reduce the wheel size of SciPy package. This is possible if we decouple the dataset files from the SciPy repository making use of Pooch to download and cache datasets kept in separate repos. See the PR for more details. I also filed https://github.com/scipy/scipy/issues/15608 Out of the 5 methods in `scipy.misc`, three dataset functions can be moved to this datasets submodule and will slowly be deprecated from misc. The question about the other two methods i.e (derivative, central_diff_weights) is up for discussion and I'd like to understand what will be the best way to deprecate them. Should we move them to a submodule like integrate (keeping derivative makes sense to me)? On the other hand, considering these two methods are not used extensively so one possibility is to deprecate both of them completely and remove them from SciPy. Please share your opinion/concerns, and comment if you see a potential issue with this. Cheers, Anirudh

4 10

New feature: fitting distributions to censored data.
by Warren Weckesser 29 Aug '22

29 Aug '22

Hey folks, A new enhancement for fitting probability distributions to censored data by maximum likelihood estimation is in progress in https://github.com/scipy/scipy/pull/13699 This is a feature in the statistics roadmap, and is part of the work for the CZI EOSS cycle 1 grant. Censored data is data where the true value of the measurement is unknown, but there is a known bound. The common terminology is: * left-censored: The true value is greater than some known bound. * right-censored: The true value is less than some known bound. * interval-censored: The true value is between known bounds. (By allowing the bounds to be +inf and -inf, we can think of all these cases as interval-censored.) In the PR, a new class, `CensoredData`, is defined to represent censored data. The `fit` method of the univariate continuous distributions is updated to accept an instance of `CensoredData`. The `CensoredData` class constructor accepts two arrays, `lower` and `upper`, to hold the known bounds of the data. (If `lower[i] == upper[i]`, it means that data value is not censored.) Because it is quite common for data to be encountered where the only censored values in the data set are either all right-censored or all left-censored, two convenience methods, `CensoredData.right_censored(x, censored)` and `CensoredData.left_censored(x, censored)`, are provide to create a `CensoredData` instance from an array of values and a corresponding boolean array that indicates if the value is censored. Here's a quick example, with data from https://www.itl.nist.gov/div898/handbook/apr/section4/apr413.htm. The data table shows 10 regular values and 10 right-censored values. The results reported there for fitting the two-parameter Weibull distribution (location fixed at 0) to that data are shape = 1.7208 scale = 606.5280 Here's the calculation using the proposed API in SciPy: In [55]: from scipy.stats import weibull_min, CensoredData Create the `CensoredData` instance: In [56]: x = np.array([54, 187, 216, 240, 244, 335, 361, 373, 375, 386] + [500]*10) In [57]: data = CensoredData.right_censored(x, x == 500) In [58]: print(data) CensoredData(20 values: 10 not censored, 10 right-censored) Fit `weibull_min` (with the location fixed at 0) to the censored data: In [59]: shape, loc, scale = weibull_min.fit(data, floc=0) In [60]: shape Out[60]: 1.720797180719942 In [61]: scale Out[61]: 606.527565269458 Matt Haberland has already suggested quite a few improvements to the PR. Additional comments would be appreciated. Warren

4 7

proposal to use SciPy project funds for build and CI improvements
by Ralf Gommers 13 Apr '22

13 Apr '22

Hi all, I'd like to make a proposal to use some project funds to make faster progress on moving to a new build system and improving our CI timeout issues. Because this is the first such proposal, let me first deal with the meta issue of _how_ to deal with this. At the start of the year I co-wrote NEP 48 (https://numpy.org/neps/nep-0048-spending-project-funds.html) to have a framework for NumPy for spending project funds. I'd like to follow that NEP also for SciPy. So I'll make the proposal here, in accordance with that NEP, and then the other thing is to figure out where to put it up as a PR (or wiki page, or ...). For that my proposal is: create a new repo called something like `project-mgmt`, where we can put such proposals as well as meeting minutes, presentations, and other content. And have a single summary page in Markdown format with a list of all proposals made, linking out to the full proposals. On to the actual proposal, keeping in mind https://numpy.org/neps/nep-0048-spending-project-funds.html#defining-fundab… . Title: Accelerating the move to Meson as the SciPy build system Cost: $12,000 Time duration: 3 months Team members: Matthew Brett (funded), Ralf Gommers (unfunded), Gagandeep Singh (unfunded), Smit Lunagariya (funded) Context: 1. Multiple people recently told me that our current build system and CI issues are a pain point for working on SciPy. We have CI timeouts on our main repo, scipy-wheels is a pain, conda-forge is also timing out on aarch64 and ppc64le build. And I've found myself spending a whole day or weekend several times in the past months to fix broken CI jobs due to build/packaging issues. 2. About 4 months ago, I wrote an RFC to move to Meson as a build system, see https://github.com/scipy/scipy/issues/13615. Since then I have spent a lot of time on this, and also used Quansight Labs funding to get one of my colleagues, Gagandeep Singh, to help me move this forward faster (and he has been super helpful). Getting a Linux + OpenBLAS dev build is close to done (16 of 20 modules complete, all that's left is `misc`, `io`, `integrate` and `signal` - probably those can be completed in the coming week). 3. Doing everything else, like adding support for Windows/aarch64/MKL/ILP64/etc. is still a lot of work. There's a tracking issue at https://github.com/rgommers/scipy/issues/22. Some of this stuff is pretty specialized, e.g., building wheels, and Fortran-on-Windows. It will take a long time to complete if it's just me spending time on the weekends. 4. Matthew Brett is available to help soon. He has written multibuild and dealt with build/packaging topics for a long time, so I think he would be able to make progress quickly. He wants to work on this topic as an independent contractor (I won't add rates here, that's for the core team to look at - but they're very reasonable). The overall goals of this effort are: 1. Get a (mostly) feature-complete Meson based build system in place by the end of October, so we can ship it in the 1.8.0 release. If all goes well, get rid of the `numpy.distutils` based build 1 release later. 2. Resolve all our CI timeout issues. If the build system alone isn't enough (it should be though), analyze the remaining issues and figure out how to tackle whatever is still problematic. The deliverables are, in order of priority: 1. Add capability to build release artifacts (wheels, sdist). I did some initial tests at the start with mesonpep517, but I have no idea if it works for SciPy and what bugs are lurking there. 2. macOS support 3. Add support for things we need to a dev.py interface (basically, like runtests.py in that it can do everything, like building docs, running benchmarks, etc.) 4. Contribute a few things we need, or that would make the build setup better, to Meson. For example, I think we'd want to be able to use `py3.install_sources` on generated targets. Of course I'm still new-ish to Meson, so it could well be that after making a proposal on the Meson issue tracker, another solution will be preferred. 5. Windows support 6. Improve build dependency detection and configuration - in particular BLAS/LAPACK flavors. 7. Clean up as many build warnings as possible, and silence the rest. The goal is to have a CI job that is completely silent (passes with `-Werror`). 8. Optimize build time further, including on CI (e.g., reuse build cache between jobs). I'd like to use 75% of the funds to pay Matthew, and 25% of the funds to take on a talented intern, Smit Lunagariya. He is comfortable working in Python, C and C++, and will be able to help with deliverables 7 and 8, as well as with testing and other tasks. Overall I'm not certain that this amount of funds is enough to complete every last deliverable on this list; it's hard to estimate. But we should certainly get close, and be able to ship a Meson build in 1.8.0. For a quick sketch of the benefits: - On my machine the build time is currently 13 minutes with numpy.distutils and 90 seconds with Meson - The build log goes from >10,000 lines (mostly noise) to ~100 lines (mostly configuration info). Why should this be a funded activity? For two reasons: (1) build and CI issues are probably the number one maintenance issue we have, and (2) most people find working on build/CI boring and painful, and hence it doesn't happen with volunteer-only activities. Even regular maintenance of our CI is hard - recently jobs have been broken for 1-2 weeks at a time a couple of times. And the scipy-wheels repo is in even worse shape. My proposal is to designate this topic as high priority, let the core team approve the compensation levels for the funded people, and get the work started. For reporting on progress and to have some accountability about how we spent the funds, I propose a short report (say 1 page, giving a status update and linking the main PRs made or issues closed) when half the funds are spent, and another such report at the end. In addition I'll plan to write a small development grant for this when the next call for that opens up, because I think we want Matthew to continue working on this after this $12,000 has been spent. I'd like to get everything as good as it can be, so other projects can easily adopt Meson as well - to also achieve my not-so-secret goal of not having to integrate numpy.distutils capabilities into setuptools once distutils is removed from Python. Thoughts? Cheers, Ralf

5 6

SciPy Community Meeting on Wed, Feb 16 at 8PM UTC
by Melissa Mendonça 30 Mar '22

30 Mar '22

Hello, all! The next SciPy Community Meeting will happen this week, on* Wednesday February 16, at 8 pm UTC*. Here's the hackmd we are using to take notes: https://hackmd.io/pyhudrC5TgSwdJtpPldHgQ If you wish to join on Zoom, use this link: https://us06web.zoom.us/j/6345425936 Cheers, Melissa p.s. Here's a handy link where you can get the meeting time in your timezone: https://www.timeanddate.com/worldclock/fixedtime.html?msg=SciPy+Community+M… <https://www.timeanddate.com/worldclock/fixedtime.html?msg=SciPy+Community+M…>

3 5

Clarifications of current developer setup
by Juan Luis Cano Rodríguez 28 Feb '22

28 Feb '22

Hi all, I have been coaching a new contributor to SciPy and we are being routinely bitten by weird problems on import when trying to follow the development installation instructions. At this point, I'm not so concerned about getting things to work: I know that, with enough trial- and-error, we will manage at some point. Instead, I'd like to try to have a more solid mental model of why things fail (if that's even possible). I am undecided and hesitant on how to proceed here since I feel I'm trying to solve too many problems at the same time. In no particular order: 1) I could try to get help on the specific errors we're getting. However, these are of the difficult-to-debug kind (undefined symbols and such) and I anticipate that we would end up in something like "you have a borked SciPy installation, please start from scratch" or "use conda instead". 2) I'd love to get a better understanding of what's the role of `conda develop .` in the development instructions, and how it's different from `pip install -e .`. I tried to look around and the only references I see about `conda develop` are the manual pages, and an old issue pointing to lack of documentation https://github.com/conda/conda-build/issues/1992 3) I've been asking myself whether SciPy would consider moving to an src-layout. I did a quick search on the issue tracker and the mailing list archives, and don't see any references. I know this is a contentious topic though, so it would be a pity to start a flame war about it, and that's definitely not my intention. I'm just wondering if at least it would help us avoid a family of problems: the ones appearing when folks do `import scipy` from the root directory. 4) We could continue the conversation we started in https://github.com/scipy/scipy/issues/12633 and try to see if there's a "happy path" that folks can follow given some assumptions (modern Linux distro/WSL2, compilers coming from either apt or conda-forge, no dangling Python installations) to reduce the installation instructions to the minimum (which could be `pip install -e . && pytest`). Apologies if this rambling is not particularly helpful, hope I can get some help on these various points. Best, Juan Luis

2 5

Plan for adding Conjugate Residual (CR) type methods for linear systems
by 赵刚 28 Feb '22

28 Feb '22

Dear all, I resubmitted a PR to support Conjugate Residual (CR) method in gh-14675: https://github.com/scipy/scipy/pull/14675 which is a light-weight solver and can be used in symmetric indefinite or Hermitian systems. The algorithm is formally similar to Conjugate Gradient (CG) but mathematically equivalent to Minimal Residual (MINRES). For symmetric positive-definite problems, CR has the same convergence as CG. But for symmetric indefinite problems, CR has almost the same iterative steps and less computation in each iteration than MINRES, which means higher computational efficiency. The arrival of CR algorithm will further enrich the reentrant Krylov subspace methods implemented in SciPy and give users more/better choices. The PR is only a part of CR type methods which will include CR, Bi-CR, Generalized CR and GCRO-DR etc. This large class of algorithms are widely used in some specific problems and play an important role. So we have implemented CR in this PR firstly, and then we will continue to implement Bi-CR and GCRO-DR methods respectively. The PR used the functional features of NumPy and SciPy to implement the CR method. Hope you like it and look forward to your review. Thank you all. Best Regards, Gang

1 0

Deprecate casting to integers behavior in special.comb when exact=True
by Jake Bowhay 27 Feb '22

27 Feb '22

Hi, I currently have gh-15359 (https://github.com/scipy/scipy/pull/15359) open to deprecate casting to integers for exact calculation in the special.comb function. Due to this behaviour it is possible to get a less exact answer by setting `exact=True` which is misleading. See the example bellow: ``` >>> x, y = 3.9, 2.8 >>> (comb(x, y), comb(x, y, exact=True)) (4.2071983565457955, 3) ``` Any comments and discussion welcome. Thanks, Jake

2 1

Segmentation Fault (Core Dumped) when testing SciPy build
by Bharath K K 25 Feb '22

25 Feb '22

Hello SciPy Developers, I'm a SciPy newbie so please forgive if my question is silly. I built SciPy on my Linux machine and tried to test the build on my conda environment called scipydev. So, when I run, python3 runtests.py -v, I get a Segmentation Fault (core dumped)... Here's the related log: scipy/sparse/linalg/_isolve/tests/test_gcrotmk.py ....... [ 54%] scipy/sparse/linalg/_isolve/tests/test_iterative.py ...........................xxxxX............................ [ 55%] scipy/sparse/linalg/_isolve/tests/test_lgmres.py ........ [ 55%] scipy/sparse/linalg/_isolve/tests/test_lsmr.py ............... [ 55%] scipy/sparse/linalg/_isolve/tests/test_lsqr.py ..... [ 55%] scipy/sparse/linalg/_isolve/tests/test_minres.py ....... [ 55%] scipy/sparse/linalg/_isolve/tests/test_utils.py . [ 55%] scipy/sparse/linalg/tests/test_expm_multiply.py ................. [ 55%] scipy/sparse/linalg/tests/test_interface.py .............. [ 55%] scipy/sparse/linalg/tests/test_matfuncs.py ............................Fatal Python error: Segmentation fault Current thread 0x00007f9bed60f740 (most recent call first): File "/home/bharath/scipy/scipy/build/testenv/lib/python3.8/site-packages/scipy/linalg/_basic.py", line 223 in solve File "/home/bharath/scipy/scipy/build/testenv/lib/python3.8/site-packages/scipy/sparse/linalg/_matfuncs.py", line 708 in _solve_P_Q File "/home/bharath/scipy/scipy/build/testenv/lib/python3.8/site-packages/scipy/sparse/linalg/_matfuncs.py", line 672 in _expm File "/home/bharath/scipy/scipy/build/testenv/lib/python3.8/site-packages/scipy/sparse/linalg/_matfuncs.py", line 590 in expm File "/home/bharath/scipy/scipy/build/testenv/lib/python3.8/site-packages/scipy/sparse/linalg/tests/test_matfuncs.py", line 516 in test_pascal File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/_pytest/python.py", line 192 in pytest_pyfunc_call File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__ File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/_pytest/python.py", line 1718 in runtest File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/_pytest/runner.py", line 168 in pytest_runtest_call File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__ File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/_pytest/runner.py", line 261 in <lambda> File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/_pytest/runner.py", line 340 in from_call File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/_pytest/runner.py", line 260 in call_runtest_hook File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/_pytest/runner.py", line 221 in call_and_report File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/_pytest/runner.py", line 132 in runtestprotocol File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/_pytest/runner.py", line 113 in pytest_runtest_protocol File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__ File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/_pytest/main.py", line 347 in pytest_runtestloop File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__ File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/_pytest/main.py", line 322 in _main File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/_pytest/main.py", line 268 in wrap_session File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/_pytest/main.py", line 315 in pytest_cmdline_main File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/pluggy/_callers.py", line 39 in _multicall File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/pluggy/_manager.py", line 80 in _hookexec File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/pluggy/_hooks.py", line 265 in __call__ File "/home/bharath/anaconda3/envs/scipydev/lib/python3.8/site-packages/_pytest/config/__init__.py", line 165 in main File "/home/bharath/scipy/scipy/build/testenv/lib/python3.8/site-packages/scipy/_lib/_testutils.py", line 69 in __call__ File "runtests.py", line 334 in main File "runtests.py", line 612 in <module> Segmentation fault (core dumped) What might be causing this? OS: Ubuntu 20.04.3 LTS Python: 3.8 Numpy: 1.22.2

2 4

Proposal to remove PDF version of SciPy docs
by Ilhan Polat 24 Feb '22

24 Feb '22

Hi everyone, In https://github.com/scipy/scipy/issues/15635 we are preparing to kill the generated PDF version of the documentation that, in our opinion serves no purpose to anyone. And if there is any we missed the offline HTML is still way better than the generated PDF. Also it is draining a lot of CI resources for every commit of every PR. Hence please let us know if we are forgetting any use case for it and we'll try to accommodate it. Otherwise we want to get rid of it asap to reduce the CI load. Best, ilhan

4 3

GSoC'22 participation SciPy
by Ralf Gommers 23 Feb '22

23 Feb '22

Hi all, It's GSoC time soon! Several people have expressed interest in mentoring already. I'd like to participate again under the PSF umbrella this year. The GSoC project durations can now be either 175 hours (2021 style) or 350 hours (-2020 style) now. For more details, see https://summerofcode.withgoogle.com/. We're already 2 days past the first deadline (feels like it's a much tighter deadline this year), so it'd be great to get some ideas and potential mentors onto this ideas page: https://github.com/scipy/scipy/wiki/GSoC-2022-project-ideas. One project we added there already is an f2py project - given there's pretty tight coupling between f2py and SciPy I think this makes sense; NumPy never participates in GSoC and the overhead of doing that this year just for this one f2py idea is too large. Here is also a link to last years' discussion of ideas: https://mail.python.org/archives/list/scipy-dev@python.org/thread/4AO4WWXZL…. If anyone wants to reuse one of these ideas, please do. From the idea page of last year I retained only Nikolay's idea on digital filtering, the rest was either executed or no longer a prio. Not sure if the digital filtering is still okay or not, Nikolay maybe you want to remove it? Any thoughts, good project ideas, or volunteers to mentor? Cheers, Ralf

3 4