Mailman 3 May 2022 - SciPy-Dev

Deprecate scipy.misc and Add scipy.datasets submodule
by Anirudh Dagar 07 Jan '24

07 Jan '24

Hi all, I just filed https://github.com/scipy/scipy/pull/15607 after an offline discussion with Ralf earlier this month. The summary of that PR is: This is a good time to get rid of the `scipy.misc` submodule since it houses only a handful of functions. As of today, most of the methods have already been deprecated from the `misc` module. They have generally moved under some other submodule. In this case, it is best to create a `scipy.datasets` submodule that can be home to the dataset functions (ascent, face, ecg). Adding a datasets submodule is not a new idea, and was proposed a couple of years ago by Warren (see https://github.com/scipy/scipy/pull/8707), but unfortunately, it never went ahead. Adding this submodule is not only sensible from an organizational/structural point of view but will also help reduce the wheel size of SciPy package. This is possible if we decouple the dataset files from the SciPy repository making use of Pooch to download and cache datasets kept in separate repos. See the PR for more details. I also filed https://github.com/scipy/scipy/issues/15608 Out of the 5 methods in `scipy.misc`, three dataset functions can be moved to this datasets submodule and will slowly be deprecated from misc. The question about the other two methods i.e (derivative, central_diff_weights) is up for discussion and I'd like to understand what will be the best way to deprecate them. Should we move them to a submodule like integrate (keeping derivative makes sense to me)? On the other hand, considering these two methods are not used extensively so one possibility is to deprecate both of them completely and remove them from SciPy. Please share your opinion/concerns, and comment if you see a potential issue with this. Cheers, Anirudh

4 10

SciPy Community Meeting on Wed, April 13 at 8pm UTC
by Melissa Mendonça 18 Oct '22

18 Oct '22

Hello, all! The next SciPy Community Meeting will happen this week, on* Wednesday March 16th, at 8pm UTC*. Here's the hackmd we are using to take notes: https://hackmd.io/pyhudrC5TgSwdJtpPldHgQ If you wish to join on Zoom, use this (updated) link: https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09 <https://us06web.zoom.us/j/6345425936> Our community calendar is now also available here: https://scientific-python.org/calendars/ Cheers, Melissa

1 12

Add Symmetric LQ method for symmetric indefinite systems of linear equations
by 赵刚 25 Sep '22

25 Sep '22

Dear all, I submitted a PR #16202 to introduce the Symmetric LQ (SYMMLQ) method in SciPy and fix the user's issue #13537. The PR website is as follows: https://github.com/scipy/scipy/pull/16202 Just like Minimal Residual method (MINRES), the SYMMLQ is also a key and important Krylov subspace method for symmetric indefinite systems, which was originally proposed by C.C.Paige and M.A.Saunders (cited by 2283 in Google) and widely used in some well-known math libraries such as MATLAB, PETSc, Trilinos and PyKrylov mentioned in the issue #13537. SYMMLQ is a variant of Conjugate Gradient method (CG). For symmetric positive-definite problems, SYMMLQ and CG has same convergence, but for symmetric indefinite problems, SYMMLQ has a smoother convergence curve than CG (CG is easier to diverge), its test results are shown in `Additional Information` of #16202. This PR is a python translation of PETSc implementation (It has been cited in `References` of `symmlq.py`) and some changes have been made based on the operation features of NumPy/SciPy. If anyone is interested in this PR, please feel free to help review it. Hope you like it, thanks. Best Regards, Gang

2 1

New feature: fitting distributions to censored data.
by Warren Weckesser 29 Aug '22

29 Aug '22

Hey folks, A new enhancement for fitting probability distributions to censored data by maximum likelihood estimation is in progress in https://github.com/scipy/scipy/pull/13699 This is a feature in the statistics roadmap, and is part of the work for the CZI EOSS cycle 1 grant. Censored data is data where the true value of the measurement is unknown, but there is a known bound. The common terminology is: * left-censored: The true value is greater than some known bound. * right-censored: The true value is less than some known bound. * interval-censored: The true value is between known bounds. (By allowing the bounds to be +inf and -inf, we can think of all these cases as interval-censored.) In the PR, a new class, `CensoredData`, is defined to represent censored data. The `fit` method of the univariate continuous distributions is updated to accept an instance of `CensoredData`. The `CensoredData` class constructor accepts two arrays, `lower` and `upper`, to hold the known bounds of the data. (If `lower[i] == upper[i]`, it means that data value is not censored.) Because it is quite common for data to be encountered where the only censored values in the data set are either all right-censored or all left-censored, two convenience methods, `CensoredData.right_censored(x, censored)` and `CensoredData.left_censored(x, censored)`, are provide to create a `CensoredData` instance from an array of values and a corresponding boolean array that indicates if the value is censored. Here's a quick example, with data from https://www.itl.nist.gov/div898/handbook/apr/section4/apr413.htm. The data table shows 10 regular values and 10 right-censored values. The results reported there for fitting the two-parameter Weibull distribution (location fixed at 0) to that data are shape = 1.7208 scale = 606.5280 Here's the calculation using the proposed API in SciPy: In [55]: from scipy.stats import weibull_min, CensoredData Create the `CensoredData` instance: In [56]: x = np.array([54, 187, 216, 240, 244, 335, 361, 373, 375, 386] + [500]*10) In [57]: data = CensoredData.right_censored(x, x == 500) In [58]: print(data) CensoredData(20 values: 10 not censored, 10 right-censored) Fit `weibull_min` (with the location fixed at 0) to the censored data: In [59]: shape, loc, scale = weibull_min.fit(data, floc=0) In [60]: shape Out[60]: 1.720797180719942 In [61]: scale Out[61]: 606.527565269458 Matt Haberland has already suggested quite a few improvements to the PR. Additional comments would be appreciated. Warren

4 7

introducing a more well-defined "legacy" API status
by Ralf Gommers 01 Jun '22

01 Jun '22

Hi all, We've got a number of (sets of) APIs that we keep around for backwards compatibility reasons, but that are not recommended to use in new code. The way we mark those APIs varies, and often it's not exactly clear that they shouldn't be used. Examples of legacy APIs: 1. `scipy.fftpack` 2. the functional interface to FITPACK (see, e.g., the Notes section of http://scipy.github.io/devdocs/reference/generated/scipy.interpolate.splev.… ) What do you think about adding a formal API status "legacy", and having a uniform way of marking such APIs? That could be for example a `.. legacy::` reST directive that gives a warning box near the top of the docs. The benefits would be (a) that it's clearer to end users that there is a better alternative, (b) it's clearer to other projects that they should not copy these APIs (thinking of CuPy, Dask, etc. here), and (c) there is a good intermediate state between "supported" and "deprecated" so that if we're not in agreement on the cost of a deprecation, we can move something to "legacy" instead rather than do nothing. Thoughts? Cheers, Ralf

5 5

gh-5286: `multivariate_normal` returns nonzero pdf for input outside its support when covariance is singular
by Matt Haberland 01 Jun '22

01 Jun '22

Hi Team, gh-5286 <https://github.com/scipy/scipy/issues/5286> reported that when the covariance matrix is singular, `multivariate_normal.pdf` returns nonzero values for `x` outside the support. Specifically, `x` is projected onto the lower-dimensional subspace (for which a proper PDF can be defined) before calculating the PDF. This comment <https://github.com/scipy/scipy/issues/5286#issuecomment-1097518655> depicts the issue graphically. Here are a few possibilities: 0. Close the issue as `wontfix` and call it a feature 1. Deprecate support for PDF when the covariance matrix is singular, and eventually raise an error (like Matlab and Mathematica) 2. Accept the fix proposed in gh-5288 <https://github.com/scipy/scipy/pull/5288> (or something like it) 3. Defer judgement and continue to wait for another alternative If there are no comments, the plan is to accept gh-5288 as a bug fix, but it is possible that some libraries might rely on the existing behavior as a "feature", so we thought we should check before merging gh-5288. Please join the discussion in gh-5288 if you're interested! Thanks, Matt -- Matt Haberland Assistant Professor BioResource and Agricultural Engineering 08A-3K, Cal Poly

1 0

main branch is open for development of SciPy 1.10.0
by Tyler Reddy 31 May '22

31 May '22

Hi, I've branched maintenance/1.9.x and the main branch is now open for the development of SciPy 1.10.0. Best wishes, Tyler

1 0

ENH: stats: adding the truncated Pareto distribution
by habitus＠tuta.io 31 May '22

31 May '22

Hello all, I submitted a PR to add the upper bounded Pareto (power-law) distribution to scipy, in the line of the existing trunc- norm, expon and weibull_min. It has a bounded support and pops up regularly when explaining scientific observations. I overloaded most methods from rv_continuous, and parameter fitting works somehow (not close to target parameters, but qualitatively good agreement between e.g. the fitted PDF and a histogram). Cheers, Nico PR: https://github.com/scipy/scipy/pull/16317 Wikipedia: https://en.wikipedia.org/wiki/Pareto_distribution#Bounded_Pareto_distributi…

1 0

NumPy 1.23.0rc2 released
by Charles R Harris 31 May '22

31 May '22

Hi All, On behalf of the NumPy team, I'm pleased to announce the release of NumPy 1.23.0rc2. The NumPy 1.23.0 release continues the ongoing work to improve the handling and promotion of dtypes, increase the execution speed, clarify the documentation, and expire old deprecations. The highlights are: - Implementation of ``loadtxt`` in C, greatly improving its performance. - Exposing DLPack at the Python level for easy data exchange. - Changes to the promotion and comparisons of structured dtypes. - Improvements to f2py. The Python versions supported in this release are 3.8-3.10, 3.11 will be supported when it comes out. Note that 32 bit wheels are only provided for Windows, all other wheels are 64 bits on account of Ubuntu, Fedora, and other Linux distributions dropping 32 bit support. All 64 bit wheels are also linked with 64 bit OpenBLAS. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.23.0rc2/>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.23.0rc2>. *Contributors* A total of 150 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - @DWesl - @GalaxySnail + - @code-review-doctor + - @h-vetinari - Aaron Meurer - Alexander Shadchin - Alexandre de Siqueira - Allan Haldane - Amrit Krishnan - Andrei Batomunkuev - Andrew J. Hesford + - Andrew Murray + - Andrey Andreyevich Bienkowski + - André Elimelek de Weber + - Andy Wharton + - Arryan Singh - Arushi Sharma - Bas van Beek - Bharat Raghunathan - Bhavuk Kalra + - Brigitta Sipőcz - Brénainn Woodsend + - Burlen Loring + - Caio Agiani + - Charles Harris - Chiara Marmo - Cornelius Roemer + - Dahyun Kim + - Damien Caliste - David Prosin + - Denis Laxalde - Developer-Ecosystem-Engineering - Devin Shanahan + - Diego Wang + - Dimitri Papadopoulos Orfanos - Ding Liu + - Diwakar Gupta + - Don Kirkby + - Emma Simon + - Eric Wieser - Evan Miller + - Evgeni Burovski - Evgeny Posenitskiy + - Ewout ter Hoeven + - Felix Divo - Francesco Andreuzzi + - Ganesh Kathiresan - Gaëtan de Menten - Geoffrey Gunter + - Hans Meine - Harsh Mishra + - Henry Schreiner - Hood Chatham + - Ilhan Polat - Inessa Pawson - Isuru Fernando - Ivan Gonzalez + - Ivan Meleshko + - Ivan Yashchuk + - Janus Heide + - Jarrod Millman - Jason Thai + - Jeremy Volkman + - Jesús Carrete Montaña + - Jhong-Ken Chen (陳仲肯) + - John Kirkham - John-Mark Gurney + - Jonathan Deng + - Joseph Fox-Rabinovitz - Jouke Witteveen + - Junyan Ou + - Jérôme Richard + - Kassian Sun + - Kazuki Sakamoto + - Kenichi Maehashi - Kevin Sheppard - Kilian Lieret + - Kushal Beniwal + - Leo Singer - Logan Thomas + - Lorenzo Mammana + - Margret Pax - Mariusz Felisiak + - Markus Mohrhard + - Mars Lee - Marten van Kerkwijk - Masamichi Hosoda + - Matthew Barber - Matthew Brett - Matthias Bussonnier - Matthieu Darbois - Matti Picus - Melissa Weber Mendonça - Michael Burkhart + - Morteza Mirzai + - Motahhar Mokf + - Muataz Attaia + - Muhammad Motawe + - Mukulika Pahari - Márton Gunyhó + - Namami Shanker + - Nihaal Sangha + - Niyas Sait - Omid Rajaei + - Oscar Gustafsson + - Ovee Jawdekar + - P. L. Lim + - Pamphile Roy + - Pantelis Antonoudiou + - Pearu Peterson - Peter Andreas Entschev - Peter Hawkins - Pierre de Buyl - Pieter Eendebak + - Pradipta Ghosh + - Rafael Cardoso Fernandes Sousa + - Raghuveer Devulapalli - Ralf Gommers - Raphael Kruse - Raúl Montón Pinillos - Robert Kern - Rohit Goswami - Ross Barnowski - Ruben Garcia + - Sadie Louise Bartholomew + - Saswat Das + - Sayed Adel - Sebastian Berg - Serge Guelton - Simon Surland Andersen + - Siyabend Ürün + - Somasree Majumder + - Soumya + - Stefan van der Walt - Stefano Miccoli + - Stephan Hoyer - Stephen Worsley + - Tania Allard - Thomas Duvernay + - Thomas Green + - Thomas J. Fan - Thomas Li + - Tim Hoffmann - Ting Sun + - Tirth Patel - Toshiki Kataoka - Tyler Reddy - Warren Weckesser - Yang Hau - Yoon, Jee Seok + Cheers, Charles Harris Reply allReplyForward <https://drive.google.com/u/0/settings/storage?hl=en&utm_medium=web&utm_sour…> <https://www.google.com/intl/en/policies/terms/> <https://www.google.com/intl/en/policies/privacy/> <https://www.google.com/gmail/about/policy/>

1 0

Warning on deprecation of stats.binom_test
by Ulyumdzhiev Dmitry 30 May '22

30 May '22

Hello, We are about to include a deprecation warning for `stats.binom_test` in https://github.com/scipy/scipy/pull/16307 as requested in #16256. Previously, it was only deprecated in the documentation. Please let us know if something is off. Kind regards, Dmitry

1 0