Mailman 3 September 2022 - SciPy-Dev

Use of another CI provider for native linux_aarch64 and macosx_arm64
by Andrew Nelson 16 Sep '22

16 Sep '22

Hi all, I've been investigating the use of cirrus-ci for native building of wheels for linux_aarch64. Currently it takes about 2 hours to build a wheel on Github Actions using emulation, but on cirrus-ci it's about 5 minutes. This will enable us to fully test the wheels that are produced, rather than just a subset of tests. It would also be possible to build macosx_arm64 wheels natively rather than a cross compile. Native builds are a huge plus, as it's a lot less complicated when compiling code. It would also be a possibility to have regular CI on those platforms, which are currently not being tested at all in our CI suite. Does anyone have comments on the use of yet another CI provider? I have a PR open in anticipation of a warm response, https://github.com/scipy/scipy/pull/17029. You can see a build, and the artifacts produced at https://cirrus-ci.com/build/6727646015913984. cheers, Andrew.

3 2

scipy 1.9.1 failures on macOS 12.5.1 with M1
by Babar Mehta 12 Sep '22

12 Sep '22

I am running `scipy.test()` on my SciPy 1.9.1 installation (from conda-forge) and depending on which version of OpenBLAS I use (also from conda-forge), I get different results: For OpenBLAS 0.3.21 (openmp_hf78f355_3) I see: ======================================================================== FAILED fft/_pocketfft/tests/test_real_transforms.py::test_dst4_definition_ortho[1-float32] - AssertionError: FAILED linalg/tests/test_solvers.py::test_solve_discrete_are - AssertionError: FAILED ndimage/tests/test_datatypes.py::test_uint64_max - AssertionError FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_maxiter_worsening[gmres] - AssertionError FAILED special/tests/test_hyp2f1.py::TestHyp2f1::test_a_b_neg_int_after_euler_hypergeometric_transformation[hyp2f1_test_case0] - AssertionError: FAILED special/tests/test_hyp2f1.py::TestHyp2f1::test_region4[hyp2f1_test_case30] - AssertionError: FAILED special/tests/test_hyp2f1.py::TestHyp2f1::test_region4[hyp2f1_test_case34] - AssertionError: FAILED special/tests/test_hyp2f1.py::TestHyp2f1::test_region4[hyp2f1_test_case47] - AssertionError: FAILED special/tests/test_hyp2f1.py::TestHyp2f1::test_region5[hyp2f1_test_case3] - AssertionError: =========================== 9 failed, 37060 passed, 3306 skipped, 12055 deselected, 137 xfailed, 11 xpassed, 86 warnings in 340.02s (0:05:40) =========================== For OpenBLAS 0.3.20 (openmp_h7efc2c9_1) I see: ======================================================================== FAILED fft/_pocketfft/tests/test_real_transforms.py::test_dst4_definition_ortho[1-float32] - AssertionError: FAILED linalg/tests/test_basic.py::TestSolve::test_simple_her_actuallysym - AssertionError: FAILED linalg/tests/test_basic.py::TestLstsq::test_simple_overdet_complex - AssertionError: FAILED linalg/tests/test_basic.py::TestLstsq::test_random_complex_exact - AssertionError: FAILED linalg/tests/test_basic.py::TestLstsq::test_random_complex_overdet - AssertionError: FAILED linalg/tests/test_blas.py::TestFBLAS2Simple::test_syr_her - AssertionError: FAILED linalg/tests/test_decomp.py::TestEig::test_make_eigvals - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigBanded::test_zhbevd - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigBanded::test_zhbevx - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigBanded::test_eig_banded - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_various_drivers_standard[ev-complex64] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_various_drivers_standard[ev-complex128] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_various_drivers_standard[evd-complex64] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_various_drivers_standard[evd-complex128] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_various_drivers_standard[evx-complex64] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_various_drivers_standard[evx-complex128] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_eigh[6-F-True-True-True-None] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_eigh[6-F-True-True-False-None] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_eigh[6-F-True-False-True-None] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_eigh[6-F-True-False-False-None] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_eigh[6-F-False-True-True-None] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_eigh[6-F-False-True-False-None] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_eigh[6-F-False-False-True-None] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_eigh[6-F-False-False-False-None] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_eigh[6-D-True-True-True-None] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_eigh[6-D-True-True-False-None] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_eigh[6-D-True-False-True-None] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_eigh[6-D-True-False-False-None] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_eigh[6-D-False-True-True-None] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_eigh[6-D-False-True-False-None] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_eigh[6-D-False-False-True-None] - AssertionError: FAILED linalg/tests/test_decomp.py::TestEigh::test_eigh[6-D-False-False-False-None] - AssertionError: FAILED linalg/tests/test_decomp.py::TestSVD_GESVD::test_simple_complex - AssertionError: FAILED linalg/tests/test_decomp.py::TestSVD_GESVD::test_random_complex - AssertionError: FAILED linalg/tests/test_decomp.py::TestQZ::test_qz_complex - AssertionError: FAILED linalg/tests/test_decomp.py::TestQZ::test_qz_complex64 - AssertionError: FAILED linalg/tests/test_decomp.py::TestQZ::test_qz_double_complex - AssertionError: FAILED linalg/tests/test_decomp.py::TestOrdQZ::test_lhp - AssertionError: FAILED linalg/tests/test_decomp.py::TestOrdQZ::test_rhp - AssertionError: FAILED linalg/tests/test_decomp.py::TestOrdQZ::test_iuc - AssertionError: FAILED linalg/tests/test_decomp.py::TestOrdQZ::test_ouc - AssertionError: FAILED linalg/tests/test_decomp.py::TestOrdQZ::test_ref - AssertionError: FAILED linalg/tests/test_decomp.py::TestOrdQZ::test_cef - AssertionError: FAILED linalg/tests/test_decomp.py::TestOrdQZWorkspaceSize::test_decompose - ValueError: Reordering of (A, B) failed because the transformed matrix pair (A, B) would ... FAILED linalg/tests/test_decomp_cossin.py::test_cossin[True-4-2-2-complex64] - AssertionError: FAILED linalg/tests/test_decomp_cossin.py::test_cossin[True-4-2-2-complex128] - AssertionError: FAILED linalg/tests/test_decomp_cossin.py::test_cossin[True-40-12-20-complex64] - AssertionError: FAILED linalg/tests/test_decomp_cossin.py::test_cossin[True-40-12-20-complex128] - AssertionError: FAILED linalg/tests/test_decomp_cossin.py::test_cossin[True-100-50-50-complex64] - AssertionError: FAILED linalg/tests/test_decomp_cossin.py::test_cossin[True-100-50-50-complex128] - AssertionError: FAILED linalg/tests/test_decomp_cossin.py::test_cossin[False-4-2-2-complex64] - AssertionError: FAILED linalg/tests/test_decomp_cossin.py::test_cossin[False-4-2-2-complex128] - AssertionError: FAILED linalg/tests/test_decomp_cossin.py::test_cossin[False-40-12-20-complex64] - AssertionError: FAILED linalg/tests/test_decomp_cossin.py::test_cossin[False-40-12-20-complex128] - AssertionError: FAILED linalg/tests/test_decomp_cossin.py::test_cossin[False-100-50-50-complex64] - AssertionError: FAILED linalg/tests/test_decomp_cossin.py::test_cossin[False-100-50-50-complex128] - AssertionError: FAILED linalg/tests/test_decomp_cossin.py::test_cossin_mixed_types - AssertionError: FAILED linalg/tests/test_decomp_ldl.py::test_simple - AssertionError: FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex64] - AssertionError: FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[30-complex128] - AssertionError: FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex64] - AssertionError: FAILED linalg/tests/test_decomp_ldl.py::test_ldl_type_size_combinations_complex[150-complex128] - AssertionError: FAILED linalg/tests/test_lapack.py::TestLeastSquaresSolvers::test_gelsd - AssertionError: FAILED linalg/tests/test_lapack.py::TestBlockedQR::test_tpqrt_tpmqrt - AssertionError: FAILED linalg/tests/test_lapack.py::test_pteqr[1-complex64-float32] - AssertionError: FAILED linalg/tests/test_lapack.py::test_pteqr[1-complex128-float64] - AssertionError: FAILED linalg/tests/test_lapack.py::test_pteqr[2-complex64-float32] - AssertionError: FAILED linalg/tests/test_lapack.py::test_pteqr[2-complex128-float64] - AssertionError: FAILED linalg/tests/test_lapack.py::test_orcsd_uncsd[complex64] - AssertionError: FAILED linalg/tests/test_lapack.py::test_orcsd_uncsd[complex128] - AssertionError: FAILED linalg/tests/test_lapack.py::test_gges_tgexc[complex64] - AssertionError: FAILED linalg/tests/test_lapack.py::test_gges_tgexc[complex128] - AssertionError: FAILED linalg/tests/test_solvers.py::test_solve_discrete_are - numpy.linalg.LinAlgError: The associated symplectic pencil has eigenvaluestoo close to the unit circle FAILED ndimage/tests/test_datatypes.py::test_uint64_max - AssertionError FAILED sparse/linalg/_eigen/arpack/tests/test_arpack.py::test_hermitian_modes - scipy.sparse.linalg._eigen.arpack.arpack.ArpackError: ARPACK error -8: Error return fr... FAILED sparse/linalg/_eigen/arpack/tests/test_arpack.py::test_real_nonsymmetric_modes - numpy.linalg.LinAlgError: generalized eig algorithm (ggev) did not converge (L... FAILED sparse/linalg/_eigen/arpack/tests/test_arpack.py::test_complex_nonsymmetric_modes - numpy.linalg.LinAlgError: generalized eig algorithm (ggev) did not converge... FAILED sparse/linalg/_eigen/arpack/tests/test_arpack.py::test_standard_nonsymmetric_starting_vector - numpy.linalg.LinAlgError: generalized eig algorithm (ggev) did n... FAILED sparse/linalg/_eigen/arpack/tests/test_arpack.py::test_general_nonsymmetric_starting_vector - numpy.linalg.LinAlgError: generalized eig algorithm (ggev) did no... FAILED sparse/linalg/_eigen/lobpcg/tests/test_lobpcg.py::test_hermitian - AssertionError: FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_ARPACK::test_svd_linop - scipy.sparse.linalg._eigen.arpack.arpack.ArpackError: ARPACK error -8: Error return... FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_small_sigma_sparse[complex-shape0] - ValueError: array must not contain infs or NaNs FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_small_sigma_sparse[complex-shape1] - ValueError: array must not contain infs or NaNs FAILED sparse/linalg/_eigen/tests/test_svds.py::Test_SVDS_LOBPCG::test_small_sigma_sparse[complex-shape2] - ValueError: array must not contain infs or NaNs FAILED sparse/linalg/_isolve/tests/test_iterative.py::test_maxiter_worsening[gmres] - AssertionError FAILED special/tests/test_hyp2f1.py::TestHyp2f1::test_a_b_neg_int_after_euler_hypergeometric_transformation[hyp2f1_test_case0] - AssertionError: FAILED special/tests/test_hyp2f1.py::TestHyp2f1::test_region4[hyp2f1_test_case30] - AssertionError: FAILED special/tests/test_hyp2f1.py::TestHyp2f1::test_region4[hyp2f1_test_case34] - AssertionError: FAILED special/tests/test_hyp2f1.py::TestHyp2f1::test_region4[hyp2f1_test_case47] - AssertionError: FAILED special/tests/test_hyp2f1.py::TestHyp2f1::test_region5[hyp2f1_test_case3] - AssertionError: ========================== 90 failed, 36979 passed, 3306 skipped, 12055 deselected, 138 xfailed, 10 xpassed, 98 warnings in 292.97s (0:04:52) =========================== On my Linux aarch64 machine all tests pass. So I guess my questions are: 1) Is it possible that these 90 failures are due to OpenBLAS? Or is it something else that SciPy uses? As the number of failures went from 90 to 9 I am inclined to believe so but I am not a dev. 2) How can I further debug the 9 failures? If they are indeed due to OpenBLAS then it's possible that the latest version fixed a bunch of things but not everything, so in order to further help them I guess it would be better if we provide minimal reproducible examples? Apologies in advance for any mistakes and happy to provide any further information.

2 2

new nightly builds (Python 3.11, Meson) available for testing
by Ralf Gommers 12 Sep '22

12 Sep '22

Hi all, Andrew and I finished a bunch of improvements and cleanups related to wheel builds. There are now development builds for SciPy 3.10.0-dev available on our regular pre-release bucket: https://anaconda.org/scipy-wheels-nightly/scipy. The following has changed: - Python 3.11 wheels were added, - 64-bit Windows wheels were added, - 64-bit ARM (aarch64) Linux wheels were added, - 32-bit Linux wheels were removed, - macOS universal2 wheels were removed, - wheels are now built with Meson instead of `numpy.distutils`, - the wheel builds are run from https://github.com/scipy/scipy/blob/main/.github/workflows/wheels.yml instead of https://github.com/MacPython/scipy-wheels This is the setup that will be used for future releases. Please do give those wheels a spin, and open an issue if you see anything that is wrong. Cheers, Ralf

2 1

NumPy 1.23. 3 Release
by Charles R Harris 09 Sep '22

09 Sep '22

Hi All, On behalf of the NumPy team, I am pleased to announce the release of NumPy 1.23.3. NumPy 1.23.3 is a maintenance release that fixes bugs discovered after the 1.23.2 release. There is no major theme for this release, the main improvements are for some downstream builds and some annotation corner cases. The Python versions supported for this release are 3.8-3.11. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.23.3>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.23.3>. Note that we will move to MacOS 11 for the NumPy 1.23.4 release, the 10.15 version currently used will no longer be supported by our build infrastructure at that point. *Contributors* A total of 16 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Aaron Meurer - Bas van Beek - Charles Harris - Ganesh Kathiresan - Gavin Zhang + - Iantra Solari+ - Jyn Spring 琴春 + - Matti Picus - Rafael Cardoso Fernandes Sousa - Rafael Sousa + - Ralf Gommers - Rin Cat (鈴猫) + - Saransh Chopra + - Sayed Adel - Sebastian Berg - Serge Guelton *Pull requests merged* A total of 14 pull requests were merged for this release. - #22136: BLD: Add Python 3.11 wheels to aarch64 build - #22148: MAINT: Update setup.py for Python 3.11. - #22155: CI: Test NumPy build against old versions of GCC(6, 7, 8) - #22156: MAINT: support IBM i system - #22195: BUG: Fix circleci build - #22214: BUG: Expose heapsort algorithms in a shared header - #22215: BUG: Support using libunwind for backtrack - #22216: MAINT: fix an incorrect pointer type usage in f2py - #22220: BUG: change overloads to play nice with pyright. - #22221: TST,BUG: Use fork context to fix MacOS savez test - #22222: TYP,BUG: Reduce argument validation in C-based ``__class_getitem__`` - #22223: TST: ensure ``np.equal.reduce`` raises a ``TypeError`` - #22224: BUG: Fix the implementation of numpy.array_api.vecdot - #22230: BUG: Better report integer division overflow (backport) Cheers, Charles Harris

1 0

gh-16923: add QMC quadrature
by Matt Haberland 07 Sep '22

07 Sep '22

Hi Team, gh-16923 proposes the addition of scipy.integrate.qmc_quad, a function for numerical integration using Quasi Monte Carlo points. The algorithm is quite simple and very similar to Monte Carlo integration, but the convergence rate tends to be faster. The proposed interface will feel familiar to users of `scipy.integrate.quad`, but it departs slightly from that of `scipy.integrate.nquad` as discussed in the issue. Please join the discussion if you're interested! https://github.com/scipy/scipy/pull/16923 Thanks, Matt -- Matt Haberland Assistant Professor BioResource and Agricultural Engineering 08A-3K, Cal Poly

1 0

Re: Whittaker−Henderson smoothing to signal
by Jonathan Stickel 06 Sep '22

06 Sep '22

On 9/1/2022 10:00, scipy-dev-request(a)python.org wrote: > From: Robert Kern<robert.kern(a)gmail.com> > Subject: [SciPy-Dev] Re: WIP: Whittaker−Henderson smoothing to signal > To: SciPy Developers List<scipy-dev(a)python.org> > Cc: andythe_great<andythe_great(a)protonmail.com> > Message-ID: > <CAF6FJiuLq6vU2NutrdeogqJWgpt_dCuZyX21PjPqXT0izHd4Kw(a)mail.gmail.com> > Content-Type: multipart/alternative; > boundary="000000000000bbfb6505e79dd68a" > > On Thu, Sep 1, 2022 at 5:43 AM Ralf Gommers<ralf.gommers(a)gmail.com> wrote: > >> Hi Andy, >> >> >> On Thu, Sep 1, 2022 at 1:55 AM andythe_great<andythe_great(a)protonmail.com> >> wrote: >> >>> Hello, >>> I would like to add Whittaker−Henderson smoothing to /scipy/signal in >>> addition to the existing savitzky_golay. >>> This function smooths the data. >>> >>> Whittaker−Henderson smoothing provides fewer artifacts compared to >>> Savitzky-Golay.https://pubs.acs.org/doi/10.1021/acsmeasuresciau.1c00054 >> Thanks for proposing this. It sounds like a useful improvement. The >> article you link to here is very new and has only 3 citations. For new >> methods we typically like to get a better sense of how good/popular/useful >> it is, either via a higher-cited article or another resource. Searching for >> Whittaker-Henderson method yields >> https://www.sciencedirect.com/science/article/abs/pii/S0167947306004713. >> Is that what you are implementing? >> > Yes. The algorithm is about a century old (and essentially amounts to > Tikhonov regularization of the identity operator). His current docstring > cites "A Perfect Smoother" from 2003 which has about 1000 citations. > > https://pubs.acs.org/doi/abs/10.1021/ac034173t > > These recent papers are more about how to implement it efficiently using > MATLAB-like (and thus scipy-like) functionality. I implemented regularization-based smoothing for the scipy ecosystem several years ago: https://github.com/jjstickel/scikit-datasmooth/ https://pypi.org/project/scikit.datasmooth/ I would be happy to see it used more. I would welcome contributions to the that package or assist in porting the code elsewhere if preferred. Jonathan

3 7

Merging main gave completely wrong results in a PR unit test test_small_sigma2 just for Linux Meson tests / Meson build (3.9, false) and Linux Tests / Python 3.8-dbg
by Andrew Knyazev 06 Sep '22

06 Sep '22

Could someone please help resolving https://github.com/scipy/scipy/pull/16712#issuecomment-1237401746 ? Merging main in https://github.com/scipy/scipy/commit/6c0ce33c407745fb24c3626234105cec3f528… gave completely wrong results in the unit test test_small_sigma2 just for Linux Meson tests / Meson build (3.9, false) and Linux Tests / Python 3.8-dbg I have no idea why and how to fix it.

1 0

Standard interface for scipy.stat functions
by Romain Jacob 06 Sep '22

06 Sep '22

Hello everyone, A looooooooooog time ago, I proposed to add a method to compute non-parametric confidence intervals for quantiles into scipy.stat [1]. Matt is now kindly trying to help me finishing this PR. Beyond the technical points (see [1] if you are interested), we got to discuss a "meta-point" that we would like to bring up with everyone. Right now, almost all functions in scipy.stat return a p-value; some provide confidence intervals in addition. Because consistency is good, we were discussing that, rather than having a function returning _only_ confidence intervals (which was my original PR proposal), we should have a function named something like `quantile_test` which would return a `res` object with the p-value of the test as an attribute and a method to compute the confidence interval. So far, so good. But... * AFAIK, there isn't really an established hypothesis test for quantiles (except for the median). We can fairly easily define one, but no one will ever look for it. * Conversely, reporting CI is (thankfully) becoming more and more common (at least, in my field). I think it would help to have this functionality available and easily findable in scipy would help. I tend to think that having a function called something like `quantile_ci` would make it much easier to find. So, what do you think? `quantile_test` with CI as a method, or `quantile_ci` directly? Cheers, [1] https://github.com/scipy/scipy/pull/12680 -- Romain Jacob Postdoctoral Researcher ETH Zurich - Networked Systems Group (NSG) www.romainjacob.net <https://www.romainjacob.net/> @RJacobPartner <https://twitter.com/RJacobPartner> Gloriastrasse 35, ETZ G78 8092 Zurich +41 7 68 16 88 22

1 0

Change behavior of stats / mstats.gmean with non-positive input?
by Matt Haberland 01 Sep '22

01 Sep '22

Hi team, The correct geometric mean of an array containing a zero is zero. This is apparent from the usual definition, but SciPy performs the calculation in the log-space, and `np.log(0)` is `-np.inf`. This requires a little bit of care. When the input to `scipy.stats.mstats.gmean` is a masked array, the `-np.inf` gets masked when it is calculated. This makes `gmean` effectively ignore the zero and produce an incorrect result. Negative elements, for which the true value would be complex, are also ignored. This was mentioned as a "feature" in previous versions of the documentation: > Use masked arrays to ignore any non-finite values ... that arise in the calculations > such as Not a Number and infinity because masked arrays automatically mask > any non-finite values. https://github.com/scipy/scipy/pull/16680 proposes to remove this "feature". After the fix, the user would still be able to use masked arrays (or `nan`s with `nan_policy='omit') to explicitly mask non-positive elements, but they would not be ignored automatically. Please join the discussion if you support or have any concerns with the proposed change. Thanks! Matt

1 0

WIP: Whittaker−Henderson smoothing to signal
by andythe_great 01 Sep '22

01 Sep '22

Hello, I would like to add Whittaker−Henderson smoothing to /scipy/signal in addition to the existing savitzky_golay. This function smooths the data. Whittaker−Henderson smoothing provides fewer artifacts compared to Savitzky-Golay. https://pubs.acs.org/doi/10.1021/acsmeasuresciau.1c00054 I would like some comments if this is a good idea. Currently, the code is in progress and hosted here. https://github.com/kevinsmia1939/scipy/blob/main/scipy/signal/_whittaker_he… At the moment, I did not implement the way to handle inconsistent data interval. Andy

4 3