Mailman 3 October 2020 - SciPy-Dev

proposal to drop Python 3.6 and NumPy 1.14 / 1.15
by Ralf Gommers 14 Nov '20

14 Nov '20

Hi all, Python 3.9 is coming out before the next SciPy release, and NEP 29 recommends to support only Python >=3.7 and NumPy >= 1.16 from last month onwards [1]. I think supporting 3 Python versions and 5 NumPy versions in SciPy 1.6.0 is easily enough. That would also bring us back in line with SciPy on conda-forge, which built against NumPy 1.16 for 1.5.2. Any objections? Cheers, Ralf [1] https://numpy.org/neps/nep-0029-deprecation_policy.html#support-table

10 21

Reviewing/dev/merge traction on some optimize PRs
by Andrew Nelson 29 Oct '20

29 Oct '20

Hi all, there are a few optimize PRs that I could do with help on the reviewing/dev/merge cycle on. They're mainly to solve various issues which a few people have been reporting related to the numerical differentiation changes for minimize methods in 1.5. In order of importance: https://github.com/scipy/scipy/pull/12889 https://github.com/scipy/scipy/pull/13009 https://github.com/scipy/scipy/pull/12998 (https://github.com/scipy/scipy/pull/11263 this one needs me to do a rebase and fix a merge conflict, and I don't think there are reviewer objections to this.) I'm asking for help because they're non-trivial changes and I'd like to see them merged (or the issues addressed) well before 1.6 to give a chance for further issues to be resolved. Also, they're all touching the same area of code and when one gets merged they might create further merge conflicts. -- _____________________________________ Dr. Andrew Nelson _____________________________________

1 0

buglet in signal.bilinear_zpk example
by Neal Becker 29 Oct '20

29 Oct '20

The labels on the 2 plots in the example here are swapped: https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.bilinear_… The 1st plot is the z domain and the 2nd plot is the s domain. -- *Those who don't understand recursion are doomed to repeat it*

1 0

NumPy 1.19.3 release
by Charles R Harris 28 Oct '20

28 Oct '20

Hi All, On behalf of the NumPy team I am pleased to announce that NumPy 1.19.3 has been released. NumPy 1.19.3 is a small maintenance release with two major improvements: - Python 3.9 binary wheels on all supported platforms, - OpenBLAS fixes for Windows 10 version 2004 fmod bug. This release supports Python 3.6-3.9 and is linked with OpenBLAS 3.12 to avoid some of the fmod problems on Windows 10 version 2004. Microsoft is aware of the problem and users should upgrade when the fix becomes available, the fix here is limited in scope. NumPy Wheels for this release can be downloaded from the PyPI <https://pypi.org/project/numpy/1.19.3/>, source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.19.3>. Linux users will need pip >= 0.19.3 in order to install manylinux2010 and manylinux2014 wheels. *Contributors* A total of 8 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Charles Harris - Chris Brown + - Daniel Vanzo + - E. Madison Bray + - Hugo van Kemenade + - Ralf Gommers - Sebastian Berg - @danbeibei + *Pull requests merged* A total of 10 pull requests were merged for this release. - #17298: BLD: set upper versions for build dependencies - #17336: BUG: Set deprecated fields to null in PyArray_InitArrFuncs - #17446: ENH: Warn on unsupported Python 3.10+ - #17450: MAINT: Update test_requirements.txt. - #17522: ENH: Support for the NVIDIA HPC SDK nvfortran compiler - #17568: BUG: Cygwin Workaround for #14787 on affected platforms - #17647: BUG: Fix memory leak of buffer-info cache due to relaxed strides - #17652: MAINT: Backport openblas_support from master. - #17653: TST: Add Python 3.9 to the CI testing on Windows, Mac. - #17660: TST: Simplify source path names in test_extending. Cheers, Charles Harris

2 1

Adding non-parametric methods to scipy.stats
by Romain Jacob 27 Oct '20

27 Oct '20

Hello everyone, I have been working for some time on the implementation of non-parametric methods to compute confidence intervals for percentiles. There are some very interesting results in the literature (see e.g. a nice pitch in [1]) which I think it would be great to add to SciPy to make them more readily available. It also seems to be rather in line with "recent" discussions of the roadmap for scipy.stats [2]. I would be interested in contributing this. What do you think? Cheers, -- Romain [1] https://ieeexplore.ieee.org/document/6841797 [2] https://github.com/scipy/scipy/issues/10577 -- Romain Jacob Postdoctoral Researcher ETH Zurich - Computer Engineering and Networks Laboratory www.romainjacob.net <https://www.romainjacob.net/> @RJacobPartner <https://twitter.com/RJacobPartner> Gloriastrasse 35, ETZ G75 8092 Zurich +41 7 68 16 88 22

6 18

Determining License for fortran inclusion in SciPy
by Sam Wallan 27 Oct '20

27 Oct '20

Hello All, We are working to add support for the Tukey-Kramer statistical test, and it relies upon an involved computation of the critical values of the Studentized Range, for which a Fortran routine has been published. Using this Fortran would be much quicker in terms of development and likely quicker in computation than our best Python implementation. The code is published in a journal article, but I'm not sure if that means that the code is publicly available, or what license or copyright might be retained. I have contacted one of the original authors to ask him about it and am awaiting a response. The paper the code is in is available from JSTOR, and the raw code is available in various places <http://ftp.uni-bayreuth.de/math/statlib/apstat/190> on the internet both in original form and sometimes with improvements. Algorithm AS 190: Probabilities and Upper Quantiles for the Studentized Range <https://www.jstor.org/stable/2347300?seq=1> R. E. Lund and J. R. Lund Journal of the Royal Statistical Society. Series C (Applied Statistics) Vol. 32, No. 2 (1983), pp. 204-210 (7 pages) Published By: Wiley DOI: 10.2307/2347300 In SciPy there is already use of several algorithms also published by Journal of the Royal Statistical Society (these can also be found on JSTOR). In git history I see they were added some 18 years ago: PRHO: Algorithm AS 89 Appl. Statist. (1975) Vol.24, No. 3, P377 in scipy/stats/statlib/spearman.f POLY: ALGORITHM AS 181.2 APPL. STATIST. (1982) VOL. 31, NO. 2 in scipy/stats/statlib/swilk.f I’ve investigated online for other licensed use of these algorithms, but I have not seen anything concrete. I closest use I was able to find is a direct translation of the algorithm <https://foundry.sandia.gov/releases/latest/javadoc-api/gov/sandia/cognition…> we are interested in, AS 190, to Java by Sandia National Laboratories, which "is released under the open source BSD License.” <https://foundry.sandia.gov/> Does anyone have experience with this source or insight into whether the inclusion of this Fortran in SciPy is fair game? Cheers, Sam Wallan

2 1

help needed
by Amardeep Singh 23 Oct '20

23 Oct '20

Hi All I am a new joiner.Using macbook. Can someone please guide me how to debug scipy using clion? I am able to build but not sure about debugging the c code. ProductName: Mac OS X ProductVersion: 10.15.7 BuildVersion: 19H2 thx

1 0

Re ENH 12755: adding a fast ill-condition sensing capability to arls()
by rondall jones 20 Oct '20

20 Oct '20

Early comments regarding this package mentioned the preference for integration in some manner with existing scipy solvers. It took me a while to see a way to do that. The problem was the lack of a quick method for detecting the likelihood of ill-conditioning (including singularity) which would make a regularized solver preferable to a standard solver like lstsq(). But I was able to develop such a quick method, called "strange()" -- because it detects strange behavior that is usually indicative of an ill-conditioned system. This method takes only about 4n^2 flops, rather than O(n^3) which the regularized solvers seem to take. So: lsmr and arls take a few times as long as lstsq. And lstsq takes a few times as long as "strange()". So inside arls() it is low-cost to call strange() first to see if it is safe to call the faster lstsq(). The special solvers -- arlseq, arlsgt, and arlsnn -- all get the benefit of this change as they depend on arls(). But that this capability applies only to the case of single right-hand-sides, as the benefit of this approach dissolves rapidly when multiple right-hand-sides are used, as all their solutions in arls() share a single call for the SVD. Also updating the in-code user documentation. Ron Jones

1 0

ANN: SciPy 1.5.3
by Tyler Reddy 19 Oct '20

19 Oct '20

Hi all, On behalf of the SciPy development team I'm pleased to announce the release of SciPy 1.5.3, which is a bug fix release that includes Linux ARM64 wheels for the first time. Sources and binary wheels can be found at: https://pypi.org/project/scipy/ and at: https://github.com/scipy/scipy/releases/tag/v1.5.3 One of a few ways to install this release with pip: pip install scipy==1.5.3 ========================== SciPy 1.5.3 Release Notes ========================== SciPy 1.5.3 is a bug-fix release with no new features compared to 1.5.2. In particular, Linux ARM64 wheels are now available and a compatibility issue with XCode 12 has been fixed. Authors ====== * Peter Bell * CJ Carey * Thomas Duvernay + * Gregory Lee * Eric Moore * odidev * Dima Pasechnik * Tyler Reddy * Simon Segerblom Rex + * Daniel B. Smith * Will Tirone + * Warren Weckesser A total of 12 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 1.5.3 ------------------------------ * `#9611 <https://github.com/scipy/scipy/issues/9611>`__: Overflow error with new way of p-value calculation in kendall... * `#10069 <https://github.com/scipy/scipy/issues/10069>`__: scipy.ndimage.watershed_ift regression in 1.0.0 * `#11260 <https://github.com/scipy/scipy/issues/11260>`__: BUG: DOP853 with complex data computes complex error norm, causing... * `#11479 <https://github.com/scipy/scipy/issues/11479>`__: RuntimeError: dictionary changed size during iteration on loading... * `#11972 <https://github.com/scipy/scipy/issues/11972>`__: BUG (solved): Error estimation in DOP853 ODE solver fails for... * `#12543 <https://github.com/scipy/scipy/issues/12543>`__: BUG: Picture rotated 180 degrees and rotated -180 degrees should... * `#12613 <https://github.com/scipy/scipy/issues/12613>`__: Travis X.4 and X.7 failures in master * `#12654 <https://github.com/scipy/scipy/issues/12654>`__: scipy.stats.combine_pvalues produces wrong results with method='mudholkar_george' * `#12819 <https://github.com/scipy/scipy/issues/12819>`__: BUG: Scipy Sparse slice indexing assignment Bug with zeros * `#12834 <https://github.com/scipy/scipy/issues/12834>`__: BUG: ValueError upon calling Scipy Interpolator objects * `#12836 <https://github.com/scipy/scipy/issues/12836>`__: ndimage.median can return incorrect values for integer inputs * `#12860 <https://github.com/scipy/scipy/issues/12860>`__: Build failure with Xcode 12 Pull requests for 1.5.3 ----------------------------- * `#12611 <https://github.com/scipy/scipy/pull/12611>`__: MAINT: prepare for SciPy 1.5.3 * `#12614 <https://github.com/scipy/scipy/pull/12614>`__: MAINT: prevent reverse broadcasting * `#12617 <https://github.com/scipy/scipy/pull/12617>`__: MAINT: optimize: Handle nonscalar size 1 arrays in fmin_slsqp... * `#12623 <https://github.com/scipy/scipy/pull/12623>`__: MAINT: stats: Loosen some test tolerances. * `#12638 <https://github.com/scipy/scipy/pull/12638>`__: CI, MAINT: pin pytest for Azure win * `#12668 <https://github.com/scipy/scipy/pull/12668>`__: BUG: Ensure factorial is not too large in mstats.kendalltau * `#12705 <https://github.com/scipy/scipy/pull/12705>`__: MAINT: \`openblas_support\` added sha256 hash * `#12706 <https://github.com/scipy/scipy/pull/12706>`__: BUG: fix incorrect 1d case of the fourier_ellipsoid filter * `#12721 <https://github.com/scipy/scipy/pull/12721>`__: BUG: use special.sindg in ndimage.rotate * `#12724 <https://github.com/scipy/scipy/pull/12724>`__: BUG: per #12654 adjusted mudholkar_george method to combine p... * `#12726 <https://github.com/scipy/scipy/pull/12726>`__: BUG: Fix DOP853 error norm for complex problems * `#12730 <https://github.com/scipy/scipy/pull/12730>`__: CI: pin xdist for Azure windows * `#12786 <https://github.com/scipy/scipy/pull/12786>`__: BUG: stats: Fix formula in the \`stats\` method of the ARGUS... * `#12795 <https://github.com/scipy/scipy/pull/12795>`__: CI: Pin setuptools on windows CI * `#12830 <https://github.com/scipy/scipy/pull/12830>`__: [BUG] sparse: Avoid using size attribute in LIL __setitem__ * `#12833 <https://github.com/scipy/scipy/pull/12833>`__: BUG: change list of globals items to list of a copy * `#12842 <https://github.com/scipy/scipy/pull/12842>`__: BUG: Use uint16 for cost in NI_WatershedElement * `#12845 <https://github.com/scipy/scipy/pull/12845>`__: BUG: avoid boolean or integer addition error in ndimage.measurements.median * `#12864 <https://github.com/scipy/scipy/pull/12864>`__: BLD: replace the #include of libqull_r.h with with this of qhull_ra.h... * `#12867 <https://github.com/scipy/scipy/pull/12867>`__: BUG: Fixes a ValueError yielded upon calling Scipy Interpolator... * `#12902 <https://github.com/scipy/scipy/pull/12902>`__: CI: Remove 'env' from pytest.ini * `#12913 <https://github.com/scipy/scipy/pull/12913>`__: MAINT: Ignore pytest's PytestConfigWarning Checksums ========= MD5 ~~~ fd75941594b22a322f63a27d53c1bdda scipy-1.5.3-cp36-cp36m-macosx_10_9_x86_64.whl 85d547117ae6d9fd447120c1768ff2d0 scipy-1.5.3-cp36-cp36m-manylinux1_i686.whl 8e8ca0d9123c4f4bf59f74ec474fc936 scipy-1.5.3-cp36-cp36m-manylinux1_x86_64.whl 855db93424cf97e5b4685c4cf74be346 scipy-1.5.3-cp36-cp36m-manylinux2014_aarch64.whl 20405ee4157858d33e38c62b873b6420 scipy-1.5.3-cp36-cp36m-win32.whl be3f3ba1018e7b1f3a30738aa502a4b5 scipy-1.5.3-cp36-cp36m-win_amd64.whl 12be163517b748a025cd399e6e9ce7df scipy-1.5.3-cp37-cp37m-macosx_10_9_x86_64.whl b2d761876d1f1d27e289b08f44034ede scipy-1.5.3-cp37-cp37m-manylinux1_i686.whl 1cd15a513ad76c64a4353a63eee6b3a8 scipy-1.5.3-cp37-cp37m-manylinux1_x86_64.whl 00bb4f63f4ee40869193c8e639da3274 scipy-1.5.3-cp37-cp37m-manylinux2014_aarch64.whl f783a76397dfe2a73752ac86f32fa474 scipy-1.5.3-cp37-cp37m-win32.whl 3c2927e6adf9b522f7f1745308b4d1f2 scipy-1.5.3-cp37-cp37m-win_amd64.whl faaa0cab95b6f352508120b8f628aaae scipy-1.5.3-cp38-cp38-macosx_10_9_x86_64.whl 77e8cb222c1868f01f048a444ef49260 scipy-1.5.3-cp38-cp38-manylinux1_i686.whl a2a035b15f78106090b9550f1383b40f scipy-1.5.3-cp38-cp38-manylinux1_x86_64.whl 86d52a963596a4cf6e1930ac5cf79a03 scipy-1.5.3-cp38-cp38-manylinux2014_aarch64.whl 8dd29aceb8dae5b5cc4f8100d8b35423 scipy-1.5.3-cp38-cp38-win32.whl 14f617e43a37827a29e8ee9ad97eda4b scipy-1.5.3-cp38-cp38-win_amd64.whl ecf5c58e4df1d257abf1634d51cb9205 scipy-1.5.3.tar.gz 61bcc473115587a66acfb42d3021479b scipy-1.5.3.tar.xz a9c549f19c661ab1b44e5b2cc707d4c1 scipy-1.5.3.zip SHA256 ~~~~~~ f574558f1b774864516f3c3fe072ebc90a29186f49b720f60ed339294b7f32ac scipy-1.5.3-cp36-cp36m-macosx_10_9_x86_64.whl e527c9221b6494bcd06a17f9f16874406b32121385f9ab353b8a9545be458f0b scipy-1.5.3-cp36-cp36m-manylinux1_i686.whl b9751b39c52a3fa59312bd2e1f40144ee26b51404db5d2f0d5259c511ff6f614 scipy-1.5.3-cp36-cp36m-manylinux1_x86_64.whl d5e3cc60868f396b78fc881d2c76460febccfe90f6d2f082b9952265c79a8788 scipy-1.5.3-cp36-cp36m-manylinux2014_aarch64.whl 1fee28b6641ecbff6e80fe7788e50f50c5576157d278fa40f36c851940eb0aff scipy-1.5.3-cp36-cp36m-win32.whl ffcbd331f1ffa82e22f1d408e93c37463c9a83088243158635baec61983aaacf scipy-1.5.3-cp36-cp36m-win_amd64.whl 07b083128beae040f1129bd8a82b01804f5e716a7fd2962c1053fa683433e4ab scipy-1.5.3-cp37-cp37m-macosx_10_9_x86_64.whl e2602f79c85924e4486f684aa9bbab74afff90606100db88d0785a0088be7edb scipy-1.5.3-cp37-cp37m-manylinux1_i686.whl aebb69bcdec209d874fc4b0c7ac36f509d50418a431c1422465fa34c2c0143ea scipy-1.5.3-cp37-cp37m-manylinux1_x86_64.whl bc0e63daf43bf052aefbbd6c5424bc03f629d115ece828e87303a0bcc04a37e4 scipy-1.5.3-cp37-cp37m-manylinux2014_aarch64.whl 8cc5c39ed287a8b52a5509cd6680af078a40b0e010e2657eca01ffbfec929468 scipy-1.5.3-cp37-cp37m-win32.whl 0edd67e8a00903aaf7a29c968555a2e27c5a69fea9d1dcfffda80614281a884f scipy-1.5.3-cp37-cp37m-win_amd64.whl 66ec29348444ed6e8a14c9adc2de65e74a8fc526dc2c770741725464488ede1f scipy-1.5.3-cp38-cp38-macosx_10_9_x86_64.whl 12fdcbfa56cac926a0a9364a30cbf4ad03c2c7b59f75b14234656a5e4fd52bf3 scipy-1.5.3-cp38-cp38-manylinux1_i686.whl a1a13858b10d41beb0413c4378462b43eafef88a1948d286cb357eadc0aec024 scipy-1.5.3-cp38-cp38-manylinux1_x86_64.whl 5163200ab14fd2b83aba8f0c4ddcc1fa982a43192867264ab0f4c8065fd10d17 scipy-1.5.3-cp38-cp38-manylinux2014_aarch64.whl 33e6a7439f43f37d4c1135bc95bcd490ffeac6ef4b374892c7005ce2c729cf4a scipy-1.5.3-cp38-cp38-win32.whl a3db1fe7c6cb29ca02b14c9141151ebafd11e06ffb6da8ecd330eee5c8283a8a scipy-1.5.3-cp38-cp38-win_amd64.whl ddae76784574cc4c172f3d5edd7308be16078dd3b977e8746860c76c195fa707 scipy-1.5.3.tar.gz cab92f8dab54a3be66525ea23e4f6568145abd1e94681cce19258a140f4de416 scipy-1.5.3.tar.xz 32dd070e203363e3a63bc184afb1d8e165c0a36a49b83b097642f78ef4da6077 scipy-1.5.3.zip

2 1

changes to autosol.py / arls.py
by rondall jones 16 Oct '20

16 Oct '20

To Ilhan, Ralf, Stefan, others... Ilhan really wants arls() to give the user some feedback about the solution. I didn't understand his/your point at first but I eventually realized that I can and should return the artifacts I compute about the problem. And I really like the result. Ilhan was right. (There are no changes whatsoever to algorithms.) Also, Ilhan objected to the name, autosol, feeling that it implies too much. I had already changed all the routine names to arlsxxx, for "automatically regularized least squares". I guess I should have changed the file name, too, and am doing that too. I hope that doesn't foul up your review process. So I just uploaded arls() and its test_arls() with these changes" * each solver now returns: * the tradition numerical rank of A ( I checked other current solvers in scipy to be sure I was doing this in the same exact way) * the "usable rank" which involves both A and b, and is the crucial artifact needed to do autoregularization * the estimate of the right-hand-side RMS error, sigma. Some people will be surprised to see that this is possible. * the resulting Tikhonov regularization parameter, lambda * The in-code comments have been updated * The test_arls() code has been updated to use one function per test, as best as I understand how to do it at this point Unfortunately I don't know how to delete the old autosol.py and test_autosol.py. Maybe one of you code managers have to do that? Thanks for your interest and tolerance of my areas of ignorance. By the way, I am not seeking to REPLACE any current scipy solvers. Arls should only be an additional feature for a long time. But I think it will be a significant asset for scipy. I have mostly publicized these codes strictly via my web site in the past, as C++ has no central code repository like python. Rondall Jones

3 2