Mailman 3 July 2021 - SciPy-Dev

New feature: fitting distributions to censored data.
by Warren Weckesser Aug. 29, 2022

Aug. 29, 2022

Hey folks, A new enhancement for fitting probability distributions to censored data by maximum likelihood estimation is in progress in https://github.com/scipy/scipy/pull/13699 This is a feature in the statistics roadmap, and is part of the work for the CZI EOSS cycle 1 grant. Censored data is data where the true value of the measurement is unknown, but there is a known bound. The common terminology is: * left-censored: The true value is greater than some known bound. * right-censored: The true value is less than some known bound. * interval-censored: The true value is between known bounds. (By allowing the bounds to be +inf and -inf, we can think of all these cases as interval-censored.) In the PR, a new class, `CensoredData`, is defined to represent censored data. The `fit` method of the univariate continuous distributions is updated to accept an instance of `CensoredData`. The `CensoredData` class constructor accepts two arrays, `lower` and `upper`, to hold the known bounds of the data. (If `lower[i] == upper[i]`, it means that data value is not censored.) Because it is quite common for data to be encountered where the only censored values in the data set are either all right-censored or all left-censored, two convenience methods, `CensoredData.right_censored(x, censored)` and `CensoredData.left_censored(x, censored)`, are provide to create a `CensoredData` instance from an array of values and a corresponding boolean array that indicates if the value is censored. Here's a quick example, with data from https://www.itl.nist.gov/div898/handbook/apr/section4/apr413.htm. The data table shows 10 regular values and 10 right-censored values. The results reported there for fitting the two-parameter Weibull distribution (location fixed at 0) to that data are shape = 1.7208 scale = 606.5280 Here's the calculation using the proposed API in SciPy: In [55]: from scipy.stats import weibull_min, CensoredData Create the `CensoredData` instance: In [56]: x = np.array([54, 187, 216, 240, 244, 335, 361, 373, 375, 386] + [500]*10) In [57]: data = CensoredData.right_censored(x, x == 500) In [58]: print(data) CensoredData(20 values: 10 not censored, 10 right-censored) Fit `weibull_min` (with the location fixed at 0) to the censored data: In [59]: shape, loc, scale = weibull_min.fit(data, floc=0) In [60]: shape Out[60]: 1.720797180719942 In [61]: scale Out[61]: 606.527565269458 Matt Haberland has already suggested quite a few improvements to the PR. Additional comments would be appreciated. Warren

4 7

proposal to use SciPy project funds for build and CI improvements
by Ralf Gommers April 13, 2022

April 13, 2022

Hi all, I'd like to make a proposal to use some project funds to make faster progress on moving to a new build system and improving our CI timeout issues. Because this is the first such proposal, let me first deal with the meta issue of _how_ to deal with this. At the start of the year I co-wrote NEP 48 (https://numpy.org/neps/nep-0048-spending-project-funds.html) to have a framework for NumPy for spending project funds. I'd like to follow that NEP also for SciPy. So I'll make the proposal here, in accordance with that NEP, and then the other thing is to figure out where to put it up as a PR (or wiki page, or ...). For that my proposal is: create a new repo called something like `project-mgmt`, where we can put such proposals as well as meeting minutes, presentations, and other content. And have a single summary page in Markdown format with a list of all proposals made, linking out to the full proposals. On to the actual proposal, keeping in mind https://numpy.org/neps/nep-0048-spending-project-funds.html#defining-fundab… . Title: Accelerating the move to Meson as the SciPy build system Cost: $12,000 Time duration: 3 months Team members: Matthew Brett (funded), Ralf Gommers (unfunded), Gagandeep Singh (unfunded), Smit Lunagariya (funded) Context: 1. Multiple people recently told me that our current build system and CI issues are a pain point for working on SciPy. We have CI timeouts on our main repo, scipy-wheels is a pain, conda-forge is also timing out on aarch64 and ppc64le build. And I've found myself spending a whole day or weekend several times in the past months to fix broken CI jobs due to build/packaging issues. 2. About 4 months ago, I wrote an RFC to move to Meson as a build system, see https://github.com/scipy/scipy/issues/13615. Since then I have spent a lot of time on this, and also used Quansight Labs funding to get one of my colleagues, Gagandeep Singh, to help me move this forward faster (and he has been super helpful). Getting a Linux + OpenBLAS dev build is close to done (16 of 20 modules complete, all that's left is `misc`, `io`, `integrate` and `signal` - probably those can be completed in the coming week). 3. Doing everything else, like adding support for Windows/aarch64/MKL/ILP64/etc. is still a lot of work. There's a tracking issue at https://github.com/rgommers/scipy/issues/22. Some of this stuff is pretty specialized, e.g., building wheels, and Fortran-on-Windows. It will take a long time to complete if it's just me spending time on the weekends. 4. Matthew Brett is available to help soon. He has written multibuild and dealt with build/packaging topics for a long time, so I think he would be able to make progress quickly. He wants to work on this topic as an independent contractor (I won't add rates here, that's for the core team to look at - but they're very reasonable). The overall goals of this effort are: 1. Get a (mostly) feature-complete Meson based build system in place by the end of October, so we can ship it in the 1.8.0 release. If all goes well, get rid of the `numpy.distutils` based build 1 release later. 2. Resolve all our CI timeout issues. If the build system alone isn't enough (it should be though), analyze the remaining issues and figure out how to tackle whatever is still problematic. The deliverables are, in order of priority: 1. Add capability to build release artifacts (wheels, sdist). I did some initial tests at the start with mesonpep517, but I have no idea if it works for SciPy and what bugs are lurking there. 2. macOS support 3. Add support for things we need to a dev.py interface (basically, like runtests.py in that it can do everything, like building docs, running benchmarks, etc.) 4. Contribute a few things we need, or that would make the build setup better, to Meson. For example, I think we'd want to be able to use `py3.install_sources` on generated targets. Of course I'm still new-ish to Meson, so it could well be that after making a proposal on the Meson issue tracker, another solution will be preferred. 5. Windows support 6. Improve build dependency detection and configuration - in particular BLAS/LAPACK flavors. 7. Clean up as many build warnings as possible, and silence the rest. The goal is to have a CI job that is completely silent (passes with `-Werror`). 8. Optimize build time further, including on CI (e.g., reuse build cache between jobs). I'd like to use 75% of the funds to pay Matthew, and 25% of the funds to take on a talented intern, Smit Lunagariya. He is comfortable working in Python, C and C++, and will be able to help with deliverables 7 and 8, as well as with testing and other tasks. Overall I'm not certain that this amount of funds is enough to complete every last deliverable on this list; it's hard to estimate. But we should certainly get close, and be able to ship a Meson build in 1.8.0. For a quick sketch of the benefits: - On my machine the build time is currently 13 minutes with numpy.distutils and 90 seconds with Meson - The build log goes from >10,000 lines (mostly noise) to ~100 lines (mostly configuration info). Why should this be a funded activity? For two reasons: (1) build and CI issues are probably the number one maintenance issue we have, and (2) most people find working on build/CI boring and painful, and hence it doesn't happen with volunteer-only activities. Even regular maintenance of our CI is hard - recently jobs have been broken for 1-2 weeks at a time a couple of times. And the scipy-wheels repo is in even worse shape. My proposal is to designate this topic as high priority, let the core team approve the compensation levels for the funded people, and get the work started. For reporting on progress and to have some accountability about how we spent the funds, I propose a short report (say 1 page, giving a status update and linking the main PRs made or issues closed) when half the funds are spent, and another such report at the end. In addition I'll plan to write a small development grant for this when the next call for that opens up, because I think we want Matthew to continue working on this after this $12,000 has been spent. I'd like to get everything as good as it can be, so other projects can easily adopt Meson as well - to also achieve my not-so-secret goal of not having to integrate numpy.distutils capabilities into setuptools once distutils is removed from Python. Thoughts? Cheers, Ralf

5 6

directional derivatives in Wolfe line search
by Sara Fridovich-Keil Oct. 29, 2021

Oct. 29, 2021

Hi SciPy-Dev, I have been using scipy.optimize.fmin_bfgs on some derivative-free benchmark problems, and noticed that whenever the Wolfe line search requires a directional derivative, the current implementation estimates the entire gradient via finite differencing, and then computes the directional derivative by taking the inner product of the gradient and the search direction. In my experiments, replacing this full gradient estimation with a single extra function evaluation to estimate the directional derivative directly, is faster. The key places this happens in the code are: https://github.com/scipy/scipy/blob/701ffcc8a6f04509d115aac5e5681c538b5265a… <https://github.com/scipy/scipy/blob/701ffcc8a6f04509d115aac5e5681c538b5265a…> https://github.com/scipy/scipy/blob/701ffcc8a6f04509d115aac5e5681c538b5265a… <https://github.com/scipy/scipy/blob/701ffcc8a6f04509d115aac5e5681c538b5265a…> Both of the Wolfe line search implementations take an input function fprime that computes/estimates the full gradient, even though only derphi, the directional derivative, is used. What I’d like to do is have an option for fprime to be either provided or not provided to the Wolfe line search. If the objective has a nice/cheap gradient then the current behavior is fine (passing the gradient function as fprime, and computing directional derivatives with an inner product), but if the objective is derivative-free then derphi should be computed with finite differencing along the search direction (just one extra function evaluation) instead of using fprime. Do people agree this would be a good change? If so I can make a pull request. Best, Sara

4 10

Refactorize levy_stable _fitstart
by Todd Bailey July 31, 2021

July 31, 2021

Hi all. I have analyses that need parts of the calculations in stats.levy_stable._fitstart(). I am willing to put them into a formal PR if people think it would be useful to refactorize _fitstart() to re-use these calculations. Current processing .fitstart() takes an array of data samples, and returns parameter estimates for a stable distribution fit to those samples. Stage 1 calculates quintiles from the samples (percentiles 5, 25, 50, 75, 95). Stage 2 estimates shape parameters from the quintiles. Stage 3 estimates location and scale parameters from the quintiles and the estimated shape parameters. Use cases I have some data that is already aggregated to percentiles. For this use case, I need Stage 2 and Stage 3. I also have some related data that is aggregated to quintiles (percentiles 10, 25, 50, 75, 90). These quintiles do not go far enough into the tails to inform an estimate of alpha. However, if I assume that the shape parameters are the same throughout, then I need a slightly-adapted Stage 3 to estimate the location and scale for each quintile distribution (using fixed alpha and beta that have been determined from other data). Refactorization Would it be useful to make some form of Stages 2 and 3 available separately within scipy.stats.levy_stable? If so, should it be via public API? Thanks in advance for your thoughts and any advice on how best to proceed. Todd

1 0

Tests failing #14371
by Yacine Thabet July 29, 2021

July 29, 2021

Hi guys, I have jumped to work on small issues to jump on the code of scipy and get used to it. I have worked on the PR https://github.com/scipy/scipy/pull/14371, but I have an issue all tests are passing on my local environment but are failing when I push to github. I tried ot pull/ merge and rebase but nothing works and I created a mess on my PR. Could you help me please, because I don't understand the problem here. Cheers.

2 1

PROPACK Sparse SVD Integration
by Nicholas McKibben July 19, 2021

July 19, 2021

Hi all, Matt Haberland and I are happy to open PR gh-14433 <https://github.com/scipy/scipy/pull/14433> to close the long-standing issue gh-857 <https://github.com/scipy/scipy/issues/857> through PROPACK integration into scipy.sparse.linalg.svds based on the previous work done by @jakevdp in pypropack <https://github.com/jakevdp/pypropack/>. This PR includes: - PROPACK wrappers exposing a new 'propack' solver backend for scipy.sparse.linalg.svds - an overhaul and expansion of the scipy.sparse.linalg.svds test suite, documentation, and benchmarks - inclusion of a new PROPACK submodule under the SciPy organization which includes patches for known bugs, Fortran compiler support, and OpenMP issues Thanks, Nicholas

2 3

(no subject)
by Charles R Harris July 18, 2021

July 18, 2021

Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.21.1. The NumPy 1.21.1 is a maintenance release that fixes bugs discovered after the 1.21.0 release. OpenBLAS has also been updated to v0.3.17 to deal with arm64 problems. The Python versions supported for this release are 3.7-3.9. The 1.21.x series is compatible with development Python 3.10 and Python 3.10 will be officially supported after it is released. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.21.1/>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.21.1>. Linux users will need pip >= 0.19.3 in order to install manylinux2010 and manylinux2014 wheels. *Contributors* A total of 11 people contributed to this release. People with a \"+\" by their names contributed a patch for the first time. - Bas van Beek - Charles Harris - Ganesh Kathiresan - Gregory R. Lee - Hugo Defois + - Kevin Sheppard - Matti Picus - Ralf Gommers - Sayed Adel - Sebastian Berg - Thomas J. Fan *Pull requests merged* A total of 26 pull requests were merged for this release. - #19311: REV,BUG: Replace `NotImplemented` with `typing.Any` - #19324: MAINT: Fixed the return-dtype of `ndarray.real` and `imag` - #19330: MAINT: Replace `"dtype[Any]"` with `dtype` in the definiton of\... - #19342: DOC: Fix some docstrings that crash pdf generation. - #19343: MAINT: bump scipy-mathjax - #19347: BUG: Fix arr.flat.index for large arrays and big-endian machines - #19348: ENH: add `numpy.f2py.get_include` function - #19349: BUG: Fix reference count leak in ufunc dtype handling - #19350: MAINT: Annotate missing attributes of `np.number` subclasses - #19351: BUG: Fix cast safety and comparisons for zero sized voids - #19352: BUG: Correct Cython declaration in random - #19353: BUG: protect against accessing base attribute of a NULL subarray - #19365: BUG, SIMD: Fix detecting AVX512 features on Darwin - #19366: MAINT: remove `print()`\'s in distutils template handling - #19390: ENH: SIMD architectures to show\_config - #19391: BUG: Do not raise deprecation warning for all nans in unique\... - #19392: BUG: Fix NULL special case in object-to-any cast code - #19430: MAINT: Use arm64-graviton2 for testing on travis - #19495: BUILD: update OpenBLAS to v0.3.17 - #19496: MAINT: Avoid unicode characters in division SIMD code comments - #19499: BUG, SIMD: Fix infinite loop during count non-zero on GCC-11 - #19500: BUG: fix a numpy.npiter leak in npyiter\_multi\_index\_set - #19501: TST: Fix a `GenericAlias` test failure for python 3.9.0 - #19502: MAINT: Start testing with Python 3.10.0b3. - #19503: MAINT: Add missing dtype overloads for object- and ctypes-based\... - #19510: REL: Prepare for NumPy 1.21.1 release. Cheers, Charles Harris

1 0

Integrating UNU.RAN in scipy.stats
by Tirth Patel July 16, 2021

July 16, 2021

Hi all, I, Christoph, and Nicholas have been working on gh-14215 [1] to integrate UNU.RAN library in SciPy. We'd appreciate your thoughts on it! We have designed an object-oriented API for sampling from any continuous or discrete distributions using universal generators in UNU.RAN. For now, only the `TransformedDensityRejection` (for continuous distributions) and `DiscreteAliasUrn` (for discrete distributions) methods have been added. These methods take a distribution object with required methods like PDF, dPDF, CDF, etc as input and set-up a generator which can then be used to sample from that distribution: >>> from scipy.stats import TransformedDensityRejection >>> import numpy as np >>> >>> class StdNorm: ... def pdf(self, x: float) -> float: ... # notice that normalization constant is not required ... # and the pdf accepts and returns scalars. ... return np.exp(-0.5 * x*x) ... def dpdf(self, x: float) -> float: ... return -x * self.pdf(x) ... >>> dist = StdNorm() >>> rng = TransformedDensityRejection(dist, seed=123) >>> rng.rvs() 0.474548717355228 One of the tricky parts about this PR is handling errors occurring in Python callbacks and UNU.RAN C library. We could use some reviews and ideas to build a maintainable infrastructure. Use of non-local returns causes memory leaks making the code for error handling a lot less trivial and possibly much more complex. I would really appreciate it if someone could take a look at the Cython and C code in the PR and help verify the approach or suggest an alternative approach, if any. There are some discussions about this in [2] and [3] and you can also look at the review comments on the main PR. Although the API design is not final yet, please feel free to comment on it as well. Thanks! [1]: https://github.com/scipy/scipy/pull/14215 [2]: https://github.com/tirthasheshpatel/scipy/pull/9 [3]: https://github.com/tirthasheshpatel/unuran/pull/1 -- Kind Regards, Tirth Patel

1 0

pythran 0.9.12 - heskenn
by Serge Guelton July 13, 2021

July 13, 2021

Hi folks, I just shipped a new release of Pythran—a compiler for scientific kernels written in python. It means it is available on PyPI and github, but not yet on conda, fedora, gentoo etc. This one fixes a bunch of issues for Scipy (again), restores compatibility with Cython, but the biggest change is definitively the new dependency tree. Indeed, Pythran no longer depends on networkx, six nor decorator. The dependency on gast and beniget received a version bump for compatibility with python 3.10 changes. Big thanks to the contributors, every commit counts! $ git shortlog -s 0.9.11..0.9.12 1 Ashwin Vishnu 1 Christian Clauss 1 Jochen Schröder 1 Miro Hrončok 2 Ralf Gommers 35 serge-sans-paille 1 刘星雨 A summarized changelog $ git diff 0.9.11..0.9.12 -- Changelog | cut -c 2- (...) 2021-07-06 Serge Guelton <serge.guelton(a)telecom-bretagne.eu> * Remove six, networkx and decorator dependency * Bump gast and Beniget requirements to support python 3.10 * Bump xsimd to 7.5.0 * Minimal default support for non-linux, non-osx, now-windows platform * Numpy improvements for np.bincount, np.transpose, np.searchsorted * Restore (and test) cython compatibility * Expose pythran.get_include for toolchain integration * Improve error message on invalid spec * Handle static dispatching based on keyword signature * Raise Memory Error upon (too) large numpy alloc * Support scalar case of scipy.special.binom * Trim the number of warnings in pythonic codebase (...)

1 0

better distinguishing public and private APIs in SciPy
by Ralf Gommers July 13, 2021

July 13, 2021

Hi all, Last week I filed https://github.com/scipy/scipy/issues/14360. The summary of that is: I think it's time to make private APIs really private, and have a better-defined public API. People don't read docs, even maintainers of other libraries like Dask and CuPy. While we have very prominently documented what our public API is, on the front page of the reference guide (http://scipy.github.io/devdocs/reference/index.html#api-definition), this is not enough. Now that more and more other libraries (Dask, CuPy, JAX, Pytorch, RAPIDS, etc.) are starting to reimplement SciPy APIs, this is becoming more of an issue. GitHub has gotten better (I think) at following renames, so `git blame` still works as expected after renaming a file - which was my main concern. My proposed approach is to add a test that ensures we won't add any new namespaces with missing underscores (we can steal this test from NumPy), add underscores to file names that are missing them, and deprecate accessing `scipy.submodule.some_private_namespace` with a deprecation that only expires in SciPy 2.0. There may be a few places where we find that some semi-private API is used quite a lot in the wild. In those cases, I suggest to still do the rename but don't add the deprecation (just use a code comment instead to explain), to avoid too much downstream churn. Please look at https://github.com/scipy/scipy/issues/14360 for more details on this, and comment if you see a potential issue with this. Cheers, Ralf

1 0