Mailman 3 August 2021 - SciPy-Dev

New feature: fitting distributions to censored data.
by Warren Weckesser Aug. 29, 2022

Aug. 29, 2022

Hey folks, A new enhancement for fitting probability distributions to censored data by maximum likelihood estimation is in progress in https://github.com/scipy/scipy/pull/13699 This is a feature in the statistics roadmap, and is part of the work for the CZI EOSS cycle 1 grant. Censored data is data where the true value of the measurement is unknown, but there is a known bound. The common terminology is: * left-censored: The true value is greater than some known bound. * right-censored: The true value is less than some known bound. * interval-censored: The true value is between known bounds. (By allowing the bounds to be +inf and -inf, we can think of all these cases as interval-censored.) In the PR, a new class, `CensoredData`, is defined to represent censored data. The `fit` method of the univariate continuous distributions is updated to accept an instance of `CensoredData`. The `CensoredData` class constructor accepts two arrays, `lower` and `upper`, to hold the known bounds of the data. (If `lower[i] == upper[i]`, it means that data value is not censored.) Because it is quite common for data to be encountered where the only censored values in the data set are either all right-censored or all left-censored, two convenience methods, `CensoredData.right_censored(x, censored)` and `CensoredData.left_censored(x, censored)`, are provide to create a `CensoredData` instance from an array of values and a corresponding boolean array that indicates if the value is censored. Here's a quick example, with data from https://www.itl.nist.gov/div898/handbook/apr/section4/apr413.htm. The data table shows 10 regular values and 10 right-censored values. The results reported there for fitting the two-parameter Weibull distribution (location fixed at 0) to that data are shape = 1.7208 scale = 606.5280 Here's the calculation using the proposed API in SciPy: In [55]: from scipy.stats import weibull_min, CensoredData Create the `CensoredData` instance: In [56]: x = np.array([54, 187, 216, 240, 244, 335, 361, 373, 375, 386] + [500]*10) In [57]: data = CensoredData.right_censored(x, x == 500) In [58]: print(data) CensoredData(20 values: 10 not censored, 10 right-censored) Fit `weibull_min` (with the location fixed at 0) to the censored data: In [59]: shape, loc, scale = weibull_min.fit(data, floc=0) In [60]: shape Out[60]: 1.720797180719942 In [61]: scale Out[61]: 606.527565269458 Matt Haberland has already suggested quite a few improvements to the PR. Additional comments would be appreciated. Warren

4 7

proposal to use SciPy project funds for build and CI improvements
by Ralf Gommers April 13, 2022

April 13, 2022

Hi all, I'd like to make a proposal to use some project funds to make faster progress on moving to a new build system and improving our CI timeout issues. Because this is the first such proposal, let me first deal with the meta issue of _how_ to deal with this. At the start of the year I co-wrote NEP 48 (https://numpy.org/neps/nep-0048-spending-project-funds.html) to have a framework for NumPy for spending project funds. I'd like to follow that NEP also for SciPy. So I'll make the proposal here, in accordance with that NEP, and then the other thing is to figure out where to put it up as a PR (or wiki page, or ...). For that my proposal is: create a new repo called something like `project-mgmt`, where we can put such proposals as well as meeting minutes, presentations, and other content. And have a single summary page in Markdown format with a list of all proposals made, linking out to the full proposals. On to the actual proposal, keeping in mind https://numpy.org/neps/nep-0048-spending-project-funds.html#defining-fundab… . Title: Accelerating the move to Meson as the SciPy build system Cost: $12,000 Time duration: 3 months Team members: Matthew Brett (funded), Ralf Gommers (unfunded), Gagandeep Singh (unfunded), Smit Lunagariya (funded) Context: 1. Multiple people recently told me that our current build system and CI issues are a pain point for working on SciPy. We have CI timeouts on our main repo, scipy-wheels is a pain, conda-forge is also timing out on aarch64 and ppc64le build. And I've found myself spending a whole day or weekend several times in the past months to fix broken CI jobs due to build/packaging issues. 2. About 4 months ago, I wrote an RFC to move to Meson as a build system, see https://github.com/scipy/scipy/issues/13615. Since then I have spent a lot of time on this, and also used Quansight Labs funding to get one of my colleagues, Gagandeep Singh, to help me move this forward faster (and he has been super helpful). Getting a Linux + OpenBLAS dev build is close to done (16 of 20 modules complete, all that's left is `misc`, `io`, `integrate` and `signal` - probably those can be completed in the coming week). 3. Doing everything else, like adding support for Windows/aarch64/MKL/ILP64/etc. is still a lot of work. There's a tracking issue at https://github.com/rgommers/scipy/issues/22. Some of this stuff is pretty specialized, e.g., building wheels, and Fortran-on-Windows. It will take a long time to complete if it's just me spending time on the weekends. 4. Matthew Brett is available to help soon. He has written multibuild and dealt with build/packaging topics for a long time, so I think he would be able to make progress quickly. He wants to work on this topic as an independent contractor (I won't add rates here, that's for the core team to look at - but they're very reasonable). The overall goals of this effort are: 1. Get a (mostly) feature-complete Meson based build system in place by the end of October, so we can ship it in the 1.8.0 release. If all goes well, get rid of the `numpy.distutils` based build 1 release later. 2. Resolve all our CI timeout issues. If the build system alone isn't enough (it should be though), analyze the remaining issues and figure out how to tackle whatever is still problematic. The deliverables are, in order of priority: 1. Add capability to build release artifacts (wheels, sdist). I did some initial tests at the start with mesonpep517, but I have no idea if it works for SciPy and what bugs are lurking there. 2. macOS support 3. Add support for things we need to a dev.py interface (basically, like runtests.py in that it can do everything, like building docs, running benchmarks, etc.) 4. Contribute a few things we need, or that would make the build setup better, to Meson. For example, I think we'd want to be able to use `py3.install_sources` on generated targets. Of course I'm still new-ish to Meson, so it could well be that after making a proposal on the Meson issue tracker, another solution will be preferred. 5. Windows support 6. Improve build dependency detection and configuration - in particular BLAS/LAPACK flavors. 7. Clean up as many build warnings as possible, and silence the rest. The goal is to have a CI job that is completely silent (passes with `-Werror`). 8. Optimize build time further, including on CI (e.g., reuse build cache between jobs). I'd like to use 75% of the funds to pay Matthew, and 25% of the funds to take on a talented intern, Smit Lunagariya. He is comfortable working in Python, C and C++, and will be able to help with deliverables 7 and 8, as well as with testing and other tasks. Overall I'm not certain that this amount of funds is enough to complete every last deliverable on this list; it's hard to estimate. But we should certainly get close, and be able to ship a Meson build in 1.8.0. For a quick sketch of the benefits: - On my machine the build time is currently 13 minutes with numpy.distutils and 90 seconds with Meson - The build log goes from >10,000 lines (mostly noise) to ~100 lines (mostly configuration info). Why should this be a funded activity? For two reasons: (1) build and CI issues are probably the number one maintenance issue we have, and (2) most people find working on build/CI boring and painful, and hence it doesn't happen with volunteer-only activities. Even regular maintenance of our CI is hard - recently jobs have been broken for 1-2 weeks at a time a couple of times. And the scipy-wheels repo is in even worse shape. My proposal is to designate this topic as high priority, let the core team approve the compensation levels for the funded people, and get the work started. For reporting on progress and to have some accountability about how we spent the funds, I propose a short report (say 1 page, giving a status update and linking the main PRs made or issues closed) when half the funds are spent, and another such report at the end. In addition I'll plan to write a small development grant for this when the next call for that opens up, because I think we want Matthew to continue working on this after this $12,000 has been spent. I'd like to get everything as good as it can be, so other projects can easily adopt Meson as well - to also achieve my not-so-secret goal of not having to integrate numpy.distutils capabilities into setuptools once distutils is removed from Python. Thoughts? Cheers, Ralf

5 6

directional derivatives in Wolfe line search
by Sara Fridovich-Keil Oct. 29, 2021

Oct. 29, 2021

Hi SciPy-Dev, I have been using scipy.optimize.fmin_bfgs on some derivative-free benchmark problems, and noticed that whenever the Wolfe line search requires a directional derivative, the current implementation estimates the entire gradient via finite differencing, and then computes the directional derivative by taking the inner product of the gradient and the search direction. In my experiments, replacing this full gradient estimation with a single extra function evaluation to estimate the directional derivative directly, is faster. The key places this happens in the code are: https://github.com/scipy/scipy/blob/701ffcc8a6f04509d115aac5e5681c538b5265a… <https://github.com/scipy/scipy/blob/701ffcc8a6f04509d115aac5e5681c538b5265a…> https://github.com/scipy/scipy/blob/701ffcc8a6f04509d115aac5e5681c538b5265a… <https://github.com/scipy/scipy/blob/701ffcc8a6f04509d115aac5e5681c538b5265a…> Both of the Wolfe line search implementations take an input function fprime that computes/estimates the full gradient, even though only derphi, the directional derivative, is used. What I’d like to do is have an option for fprime to be either provided or not provided to the Wolfe line search. If the objective has a nice/cheap gradient then the current behavior is fine (passing the gradient function as fprime, and computing directional derivatives with an inner product), but if the objective is derivative-free then derphi should be computed with finite differencing along the search direction (just one extra function evaluation) instead of using fprime. Do people agree this would be a good change? If so I can make a pull request. Best, Sara

4 10

Understanding scipy.stats.boxcox
by Keith Sloan Oct. 16, 2021

Oct. 16, 2021

I have the following questions about scipy.stats.boxcox and looking for pointers guidance. I would like to implement something similar to boxcox but with the following differences. 1) I would like to use a different transformation function from w=(y^x - 1) / x 2) I would separate from 1) like to have the target be other than a Normal/Gaussian distribution, I am thinking of a Plank function for a Blackbody I had a quick look at https://github.com/scipy/scipy/blob/master/scipy/stats/_continuous_distns.py but was still lacking enlightenment, so would welcome some pointers Thanks

5 4

GSoC'21 Blog: Improve performance through use of Pythran
by Xingyu Liu Aug. 31, 2021

Aug. 31, 2021

Hi everyone: I’m Xingyu Liu, this year’s SciPy GSoC student :) This summer, I’m very fortunate to work on improving SciPy algorithms’ performance via Pythran, under the supervision of my awesome mentors Ralf Gommers and Serge Guelton! I’m so happy to share the blog of our work: http://serge-sans-paille.github.io/pythran-stories/gsoc21-improve-performan… It has been a great experience working on this project in GSoC'21, and much thanks to the SciPy community for this opportunity and help on my project! Special thanks to my mentors, Ralf and Serge, who provided immense support for me to get through the difficulties! I have learnt a lot this summer and I’ll continue to contribute to SciPy and Pythran in the future!!! Cheers, Xingyu

3 4

ENH: add Lloyd's algorithm
by Pamphile Roy Aug. 27, 2021

Aug. 27, 2021

Hi everyone, I opened a PR which proposes to add Lloyd’s algorithm. https://github.com/scipy/scipy/pull/14642 <https://github.com/scipy/scipy/pull/14642> The TL;DR is: Lloyd's algorithm, or Central Voronoi Tesselation (CVT), post-process a sample to uniformize the inter-sample distances. It's an iterative process. From a sample, the Voronoi cells are calculated, then each sample is moved toward the centroid of its cell. Initially I though about adding this to scipy.stats.qmc as I intended to use this to improve a sample. But Matt rightly pointed out that this could just be put in scipy.spatial. What do you think? Thanks in advance for your inputs! Cheers, Pamphile

3 3

On the topic of separate merge commits
by Tyler Reddy Aug. 25, 2021

Aug. 25, 2021

Hi all, I recently received some criticism for merging a PR that had only 1 commit and producing a single merge commit on top of it (this happens to be the default option I have/what I've been doing for years). See here: https://github.com/scipy/scipy/pull/14630 I'd like to mention a few things here: - are we still "ok" with folks doing this? frankly, I find taking heat for precise merging philosophy a bit unwelcoming even as someone who has been contributing for a while; sometimes it is important to squash 100 messy commits, but otherwise do a good job reviewing PRs and relaxing on the purist "git history" attacks seem a bit more welcoming to me - I find those merge commits somewhat useful when navigating back to GitHub anyway, maybe because I'm so used to them - why does GitHub allow us to do this if it really isn't useful at all? Tyler

9 10

In-place developer installation instructions
by Stefan van der Walt Aug. 25, 2021

Aug. 25, 2021

Hi all, I noticed that the pip developer installation instructions are missing from the documentation. The instructions are in quickstart_pip.rst, but I don't see that included in contributor/index.rst. Besides that, the instructions do not show how to build in-place and then install that SciPy so it can be used locally. I would expect something like: pip install --no-build-isolation -e . Or: python setup.py build_ext -i Then add SciPy to the PYTHONPATH. If I can get a sense of what the instructions should look like, I'd be happy to file a PR. Best regards, Stéfan

2 3

PR to update scipy.stats.binned_statistic
by Teddy Ort Aug. 22, 2021

Aug. 22, 2021

Hi Scipy! https://github.com/scipy/scipy/pull/14629 is a PR that optimizes the *std* statistic for *scipy.stats.binned_statistic*. I recently submitted another PR <https://github.com/scipy/scipy/pull/14625> that does the same for *min*, *max*, and *median*, and have since realized how to do something similar for *std.* Thanks for looking! Teddy

1 0

New PR for scipy.stats.binned_statistic
by Teddy Ort Aug. 21, 2021

Aug. 21, 2021

Hi Scipy! I've made a small change to scipy.stats.binned_statistic that improves performance on the min, max, and median statistics. https://github.com/scipy/scipy/pull/14625 This is my first PR. I've tried to follow the checklist, but please let me know if I've missed something. Best, Teddy

1 0