Mailman 3 September 2022 - SciPy-Dev

RFC: decouple doctesting from refguide-check
by Evgeni Burovski 23 Feb '24

23 Feb '24

Hi, Code examples in the SciPy and NumPy documentation are doctested, with a modified doctesting machinery which understands floating point, numpy formatting and some details of our documentation and API. Our modified doctesting machinery is buried in the refguide-check utility, which also does several unrelated things, all of which are tightly coupled to each other and to the libraries themselves. It has been sometimes described as hard to understand or work with or extend due to this tight coupling and a lack of a dedicated test suite. The numpy and scipy versions of the utility are both vendored, and the numpy version has diverged from the scipy version somewhat. Following a discussion in https://github.com/numpy/numpy/issues/21070 I did a small experiment to decouple the doctesting into a separate package, so that it's easier to consolidate the two versions. Plus, a separate repo is generally easier to maintain, configure, and possibly extend or adopt to other projects. The work-in-progress result is here: https://github.com/ev-br/scpdt It currently can run the full doctesting of the scipy API documentation (docstrings of objects) --- see https://github.com/ev-br/scpdt/pull/33 (the log of a test run with warnings turned to errors is in the GH actions: https://github.com/ev-br/scpdt/runs/6743881766?check_suite_focus=true). Note that it shows, among other things, a couple of deprecation warnings our docs have accumulated :-). The API of the tool closely follows that of the standard library doctest module and provides (nearly) drop-in replacements for doctest checking, parsing, finding and running. Various configuration options for our modifications are collected into a single bag object which is internally passed around. This way, it's user-configurable all the way from a plain standard doctest module behavior to what refguide-check does now. There are a couple of wrinkles to iron out; overall it does what refguide-check does already. One missing bit is doctesting rst or other text files, but it's coming soon. The current plan is to: - verify that the standalone version does not miss things checked by the refguide-check - plumb it through the SciPy dev interface and rip out the refguide-check bundled doctesting utilities. - Sync changes that NumPy version of refguide-check accumulated over time - Make sure it correctly tests the NumPy docs, too. - Better document the internals, there is currently only a readme file. If someone's interested to join me working on these, great, the more the merrier :-). I think it could make sense to move the tool's repository to the scipy github org (or maybe even numpy org?). I'm offering to maintain it regardless of the location. Thoughts? Cheers, Evgeni

5 8

Deprecate scipy.misc and Add scipy.datasets submodule
by Anirudh Dagar 07 Jan '24

07 Jan '24

Hi all, I just filed https://github.com/scipy/scipy/pull/15607 after an offline discussion with Ralf earlier this month. The summary of that PR is: This is a good time to get rid of the `scipy.misc` submodule since it houses only a handful of functions. As of today, most of the methods have already been deprecated from the `misc` module. They have generally moved under some other submodule. In this case, it is best to create a `scipy.datasets` submodule that can be home to the dataset functions (ascent, face, ecg). Adding a datasets submodule is not a new idea, and was proposed a couple of years ago by Warren (see https://github.com/scipy/scipy/pull/8707), but unfortunately, it never went ahead. Adding this submodule is not only sensible from an organizational/structural point of view but will also help reduce the wheel size of SciPy package. This is possible if we decouple the dataset files from the SciPy repository making use of Pooch to download and cache datasets kept in separate repos. See the PR for more details. I also filed https://github.com/scipy/scipy/issues/15608 Out of the 5 methods in `scipy.misc`, three dataset functions can be moved to this datasets submodule and will slowly be deprecated from misc. The question about the other two methods i.e (derivative, central_diff_weights) is up for discussion and I'd like to understand what will be the best way to deprecate them. Should we move them to a submodule like integrate (keeping derivative makes sense to me)? On the other hand, considering these two methods are not used extensively so one possibility is to deprecate both of them completely and remove them from SciPy. Please share your opinion/concerns, and comment if you see a potential issue with this. Cheers, Anirudh

4 10

Next newcomers' meeting: Friday, July 22 at 8pm UTC
by Melissa Mendonça 17 Aug '23

17 Aug '23

Hello, folks! This friday, *July 22 at 8pm UTC* we are having another newcomers' meeting. All are welcome to join and ask questions about contributing, setting up your development environment, PRs or issues you may want to work on, or just chat. Join us on Zoom <https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09>, and if you need more information check out some notes from our previous meeting here: https://hackmd.io/@melissawm/scipy-newcomers Hope to see you around! - Melissa

1 14

Basic spherical statistics in scipy
by Daniel Schmitz 10 Apr '23

10 Apr '23

Hi everyone, recently questions came up with regards to statistics on the unit sphere in scipy. Issue 12041 <https://github.com/scipy/scipy/issues/12041> raises the question if spherical mean/variance should be added as descriptive statistics similar to circmean/circvar <https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.circvar.ht…> . The canonical reference for all things related to statistics on the unit sphere is "Directional Statistics" by Mardia & Jupp. Another important basic functionality is the generation of random samples on the unit sphere: discussion was started in this issue <https://github.com/scipy/scipy/issues/16205> . Any strong opinion against these? I think at least the random number generation is reimplemented every day around the world (done so myself at least 2 times) and would strongly benefit from being part of scipy. Best

4 9

SciPy Community Meeting on Wed, April 13 at 8pm UTC
by Melissa Mendonça 18 Oct '22

18 Oct '22

Hello, all! The next SciPy Community Meeting will happen this week, on* Wednesday March 16th, at 8pm UTC*. Here's the hackmd we are using to take notes: https://hackmd.io/pyhudrC5TgSwdJtpPldHgQ If you wish to join on Zoom, use this (updated) link: https://us06web.zoom.us/j/6345425936?pwd=aDVFQzVmbk9SVU5jU0Jwc0s3YWUrdz09 <https://us06web.zoom.us/j/6345425936> Our community calendar is now also available here: https://scientific-python.org/calendars/ Cheers, Melissa

1 12

Question about the fit and pdf methods for discrete distributions
by Julian CHAMBRIER 29 Sep '22

29 Sep '22

Dear all, In the context of my research on a way to find the distribution (discrete or continuous) the most adapted to a set of data (as well as the associated parameters), I use scipy a lot. Concerning continuous distributions, the present methods are very useful for my work (especially fit, pdf and ppf), but I have more difficulties with discrete distributions. I saw that some topics have been opened concerning a fit method for discrete distributions. Is this work still going on? Indeed, I would like to be able to estimate the parameters of a sample of data with a discrete distribution (like Poisson, Bernoulli, ...) and thus be able to use a PDF and fit method. On another note, I would like to know if the PERT distribution will be integrated in scipy? Don't hesitate to come back to me for more information. Best regards, Julian CHAMBRIER

3 11

Pythran 0.12.0 - arar
by Serge Guelton 29 Sep '22

29 Sep '22

Hey folks, I just uploaded a new revision of Pythran, namely 0.12.0 codenamed `arar'. As usual with these releases, it comes along with numerous bugfixes and enhacements, some of which are listed in the changelog below. I'd like to highlight that translation speed from Python to C++ is now much faster, sometimes by an order of magnitude. And we now generate code that Intel compiler can understands, which may please HPC users. All in one, I'm very happy with what has been done so far. As usual the package is already available on PyPI[0] and the source is on GitHub[1], Fedora, Debian, Ubuntu, Gentoo and Conda package will flow in your favorite distributions when packager feel it's the right time :-) Serge [0] https://pypi.org/project/pythran/ [1] https://github.com/serge-sans-paille/pythran -- Changelog extract 2022-09-20 Serge Guelton <serge.guelton(a)telecom-bretagne.eu> * Improve numpy expression computation speed * Decent ICC support (fixing an issue from 2018!) * Much faster C++ code generation * Complete rework of constant folder * Support C++-time evaluation of numpy.ndarray.ndim * Improved omp declare reduction support * Allow indexing of ndarray by integers of mixed types * A lot of small pesty bug fixes in the C++ headers

2 1

Scipy curve_fit does not estimate offset values and throws initial guess as the final estimate for every fit.
by amk2358＠columbia.edu 29 Sep '22

29 Sep '22

Hello everyone, I am currently working on a curve_fitting problem that needs to estimate 4 parameters namely time to peak (tmax), maximum amplitude (ymax), shape (b) and offset. I am using Scipy's curve_fit library that is giving reliable results for the first three parameters but not for offset. My objective function looks like this: def objective2(t,tmax, ymax, b, offset): offInt = math.ceil(offset) offArr = np.zeros(offInt) offArr = np.append(offArr,ymax*((t[:(len(t)-offInt)]/tmax)*np.exp((b*tmax)))* 2.713*np.exp((tmax-t[:(len(t)-offInt)])*b)) return offArr I am fitting a 4 parameter gamma function in this objective function. If my initial guess for these parameters is p0 = np.array([100, 5, 0.005, 10]), all my estimated fits have an offset of 10 which is not right. Can someone help me with this or is there some issue with scipy curve_fit that can be corrected?

3 2

Add Symmetric LQ method for symmetric indefinite systems of linear equations
by 赵刚 25 Sep '22

25 Sep '22

Dear all, I submitted a PR #16202 to introduce the Symmetric LQ (SYMMLQ) method in SciPy and fix the user's issue #13537. The PR website is as follows: https://github.com/scipy/scipy/pull/16202 Just like Minimal Residual method (MINRES), the SYMMLQ is also a key and important Krylov subspace method for symmetric indefinite systems, which was originally proposed by C.C.Paige and M.A.Saunders (cited by 2283 in Google) and widely used in some well-known math libraries such as MATLAB, PETSc, Trilinos and PyKrylov mentioned in the issue #13537. SYMMLQ is a variant of Conjugate Gradient method (CG). For symmetric positive-definite problems, SYMMLQ and CG has same convergence, but for symmetric indefinite problems, SYMMLQ has a smoother convergence curve than CG (CG is easier to diverge), its test results are shown in `Additional Information` of #16202. This PR is a python translation of PETSc implementation (It has been cited in `References` of `symmlq.py`) and some changes have been made based on the operation features of NumPy/SciPy. If anyone is interested in this PR, please feel free to help review it. Hope you like it, thanks. Best Regards, Gang

2 1

Multivariate Normal Enhancements (gh-16002)
by Matt Haberland 25 Sep '22

25 Sep '22

Hi team, Several issues and PRs have proposed enhancements to the multivariate normal distribution (e.g. gh-8999, gh-9921, gh-9942, gh-11053, gh-11772, gh-12184, gh-15675), but the functionality has remained essentially untouched for at least six years. gh-16002 begins to make progress toward these issues by introducing a `Covariance` class and public subclasses (e.g. `CovViaPrecision`) that allow the user to provide the desired decomposition of the covariance matrix instead the covariance matrix itself. After this and planned followup PRs are merged, the user will be able to specify a covariance matrix via its diagonal elements, Cholesky decomposition, LDL decomposition, eigenvalue decomposition, and inverse. (Of course, many other ideas are possible - e.g. sparse decompositions, tridiagonal, etc.) Advantages include better treatment of singular covariance matrices, avoiding repeated decomposition of the same matrix, and more efficient computation of the Mahalanobis distance. The `Covariance` classes implement essential methods and properties to separate the logic of these computations from the `multivariate_normal` class, and therefore they can be reused in other multivariate distributions (e.g. `multivariate_t`, `wishart`, `invwishart`) and directly in user code. It also sets the stage for enabling efficient broadcasting over n-dimensional shape parameter arrays (e.g. stacks of covariance matrices and means). Please join the discussion in gh-16002 if you're interested. https://github.com/scipy/scipy/pull/16002 We'll leave the PR open to feedback for at least a week. Thanks, Matt -- Matt Haberland Assistant Professor BioResource and Agricultural Engineering 08A-3K, Cal Poly

1 0