Mailman 3 July 2020 - SciPy-Dev

Adding non-parametric methods to scipy.stats
by Romain Jacob 27 Oct '20

27 Oct '20

Hello everyone, I have been working for some time on the implementation of non-parametric methods to compute confidence intervals for percentiles. There are some very interesting results in the literature (see e.g. a nice pitch in [1]) which I think it would be great to add to SciPy to make them more readily available. It also seems to be rather in line with "recent" discussions of the roadmap for scipy.stats [2]. I would be interested in contributing this. What do you think? Cheers, -- Romain [1] https://ieeexplore.ieee.org/document/6841797 [2] https://github.com/scipy/scipy/issues/10577 -- Romain Jacob Postdoctoral Researcher ETH Zurich - Computer Engineering and Networks Laboratory www.romainjacob.net <https://www.romainjacob.net/> @RJacobPartner <https://twitter.com/RJacobPartner> Gloriastrasse 35, ETZ G75 8092 Zurich +41 7 68 16 88 22

6 18

Re: [SciPy-Dev] Discontinued Rackspace Open Compute Discount
by Robert Lucente 05 Oct '20

05 Oct '20

Several CI services are free for open source projects although there are throughput limits https://about.gitlab.com https://github.com/features/actions https://travis-ci.org https://www.appveyor.com -----Original Message----- >From: Olivier Grisel <olivier.grisel(a)ensta.org> >Sent: Feb 8, 2020 11:30 AM >To: SciPy Developers List <scipy-dev(a)python.org> >Subject: [SciPy-Dev] Discontinued Rackspace Open Compute Discount > >Hi all, > >I received an email end of January that I just discovered today >telling me that the Rackspace Open Compute Discount grant we use to >host http://wheels.scipy.org for nightly builds and release management >of our community has been discontinued on December 31st 2019. > >I would like to take down the Rackspace hosted cloudfiles as soon as >possible but we need to find a replacement as this will break all the >CIs that rely on nightly wheels and also all the wheel release >infrastructure for several project in the ecosystem. > >Apparently the fees for January and the beginning of February already >sum to $936.51. This is a large cost because there were a bunch of >unused left-over block volumes that I already deleted. > >To host wheels for nightly builds and release automation, a possible >alternatives include using the github release pages but they are not >really designed for this kind of workflow. > >Alternatively there is a PyPI repo hosting feature in Azure Pipelines >in public preview but I am not sure how it works, if it's free for >open source projects and if we could use it as a replacement for the >Rackspace cloud storage we currently use to host >http://wheels.scipy.org . > >https://docs.microsoft.com/en-us/azure/devops/pipelines/artifacts/pypi?view… > >Important: if you have uploaded any important files to one of the >cloud file storage / blob containers on Rackspace, please download a >copy as soon as possible. > >But please do not download things you do not really want to keep to >avoid adding any unnecessary bandwidth costs. If you have file write >permission on one of those containers, please delete the files you own >and do not need anymore. > >Feel free to join this room to discuss the matter interactively: >https://gitter.im/scikit-learn/nightly-build-hosting > >Feel free to reply to this email on the scipy-dev mailing list if you >have suggestions for setting up a free replacement. > >-- >Olivier >http://twitter.com/ogrisel - http://github.com/ogrisel >_______________________________________________ >SciPy-Dev mailing list >SciPy-Dev(a)python.org >https://mail.python.org/mailman/listinfo/scipy-dev

12 25

Wallenius hypergeometric distribution
by Harry Gulliver 10 Aug '20

10 Aug '20

Hi all, new (potential) contributor here! I'm working on a project using Wallenius' non-central hypergeometric distribution atm and noticed it's not yet available in scipy.stats, so thought I'd volunteer to add it. Could possibly also do Fisher's hypergeom while I'm at it. Anything I ought to know before just writing some code and making a pull request? How can I best help? Thanks! Harry

3 4

ENH: Add an approximate Linear Sum Assignment Solver
by Ali Saad-Eldin 04 Aug '20

04 Aug '20

Hi all, Hi all, I would like to add an approximate Linear Assignment Problem<http://www.assignmentproblems.com/doc/LSAPIntroduction.pdf> (aLAP) solver to scipy.optimize. The specific approximation algorithm I propose adding can be found here<https://link.springer.com/content/pdf/10.1007%2F978-3-540-68111-3_74.pdf>. Scipy.optimize currently houses an exact solver for LAP in linear_sum_assignment()<https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.linear_…>, and I was thinking this could be added either as a new method within linear_sum_assignment(), or as its own function. The approximation algorithm has time complexity O(n2log(n)), compared to the algorithm implemented in linear_sum_assignment() with complexity ~O(n2.3 ), while guaranteeing a score >= 1/2 optimum (though in practice the scores appear to be much closer to optimum). I've attached two plots demonstrating runtime and accuracy, running linear_sum_assignment() and aLAP on simulated, dense nxn cost_matrices with entries randomly selected from a uniform(0,100) distribution, for n sampled from [0, 3000]. Note that linear_sum_assignment() is essentially C/C++, while the aLAP implementation is native Python, so a cython implementation could make aLAP even faster. Additionally, an advantage to this algorithm is that it is parallelizable, which could cause a further speed up. Current version of the implementation can be found here<https://github.com/neurodata/graspy/blob/ase-alap/graspy/match/alap.py>, and proof of effectiveness here<https://nbviewer.jupyter.org/github/asaadeldin11/nbview/blob/master/aLAP_co…>. Best, Ali Saad-Eldin

2 1

Entropy Calculations
by Michael Nowotny 31 Jul '20

31 Jul '20

Dear SciPy developers, I have noticed that the statistical functions for the calculation of entropy and KL divergence currently only support discrete distributions for which the probability mass function is known. I recently needed to compute various information theoretic measures from samples of distributions and created the package `Divergence`. It offers functionality for entropy, cross entropy, relative entropy, Jensen-Shannon divergence, joint entropy, conditional entropy, and mutual information and is available on GitHub at https://github.com/michaelnowotny/divergence <https://github.com/michaelnowotny/divergence>. It supports samples from both discrete and continuous distributions. Continuous distributions are implement via numerical integration of kernel density estimates generated from the sample. I would be happy to contribute some or all of its functionality to SciPy. Please let me know if you are interested. Thank you, Michael

2 2

Fwd: [rlucas7/scipy] Run failed: Nightly build - master (c5e0ca5)
by rlucas7＠vt.edu 28 Jul '20

28 Jul '20

Hi scipy-dev, I think my branch is failing the python 3.9 (experimental) build. I don’t see the failure every night but instead maybe once a month. Has anyone else seen this and know what needs to be changed? Presumably there is a configuration file that I need to change an entry to turn off the build. Begin forwarded message: > From: Lucas Roberts <notifications(a)github.com> > Date: July 16, 2020 at 8:03:43 PM EDT > To: rlucas7/scipy <scipy(a)noreply.github.com> > Cc: Ci activity <ci_activity(a)noreply.github.com> > Subject: [rlucas7/scipy] Run failed: Nightly build - master (c5e0ca5) > Reply-To: rlucas7/scipy <noreply(a)github.com> > > > Run failed for master (c5e0ca5) > > Repository: rlucas7/scipy > Workflow: Nightly build > Duration: 35.0 seconds > Finished: 2020-07-17 00:03:24 UTC > > View results > > Jobs: > test_nightly (3.9) failed (1 annotation) > — > You are receiving this because this workflow ran on your branch. > Manage your GitHub Actions notifications here.

3 3

Global Optimization Benchmarks
by Andrea Gavana 27 Jul '20

27 Jul '20

Dear SciPy developers & users, I have a couple of new derivative-free, global optimization algorithms I’ve been working on lately - plus some improvements to AMPGO and a few more benchmark functions - and I’d like to rerun the benchmarks as I did back in 2013 (!!!). In doing so, I’d like to remove some of the least interesting/worst performing algorithms (Firefly, MLSL, Galileo, the original DE) and replace them with the ones currently available in SciPy - differential_evolution, SHGO and dual_annealing. Everything seems good and dandy, but it appears to me that SHGO does not accept an initial point for the optimization process - which makes the whole “run the optimization from 100 different starting points for each benchmark” a bit moot. I am no expert on SHGO, so maybe there is an alternative way to “simulate” the changing of the starting point for the optimization? Or maybe some other approach to make it consistent across optimizers? Any suggestion is more than welcome. Andrea.

2 6

Re: [SciPy-Dev] Wallenius hypergeometric distribution
by Harry Gulliver 25 Jul '20

25 Jul '20

Serendipitous timing! In principle I'd be happy to do that, but I know almost no C++, so I'm not sure if I'd be able to... Harry Date: Fri, 24 Jul 2020 07:54:26 -0700 From: Matt Haberland <mhaberla(a)calpoly.edu> To: SciPy Developers List <scipy-dev(a)python.org> Subject: Re: [SciPy-Dev] Wallenius hypergeometric distribution Message-ID: <CADuxUizWkhy1nkns+09hAkTXQ7TkyOhaVYwzMhdbp2TyZMOvaQ(a)mail.gmail.com> Content-Type: text/plain; charset="utf-8" Funny that you should mention this now! On Tuesday we got permission from Agner Fog to use his C++ implementation (used in the R package BiasedUrn, https://cran.r-project.org/web/packages/BiasedUrn/index.html) under SciPy's license. Would you be interested in wrapping a C++ implementation? On Fri, Jul 24, 2020, 2:53 AM Harry Gulliver <gulliver.harry(a)gmail.com> wrote: > Hi all, new (potential) contributor here! > > I'm working on a project using Wallenius' non-central hypergeometric > distribution atm and noticed it's not yet available in scipy.stats, so > thought I'd volunteer to add it. Could possibly also do Fisher's hypergeom > while I'm at it. > > Anything I ought to know before just writing some code and making a pull > request? How can I best help? > > Thanks! > Harry

2 1

ANN: SciPy 1.5.2
by Tyler Reddy 24 Jul '20

24 Jul '20

Hi all, On behalf of the SciPy development team I'm pleased to announce the release of SciPy 1.5.2, which is a bug fix release. Sources and binary wheels can be found at: https://pypi.org/project/scipy/ and at: https://github.com/scipy/scipy/releases/tag/v1.5.2 One of a few ways to install this release with pip: pip install scipy==1.5.2 ========================== SciPy 1.5.2 Release Notes ========================== SciPy 1.5.2 is a bug-fix release with no new features compared to 1.5.1. Authors ====== * Peter Bell * Tobias Biester + * Evgeni Burovski * Thomas A Caswell * Ralf Gommers * Sturla Molden * Andrew Nelson * ofirr + * Sambit Panda * Ilhan Polat * Tyler Reddy * Atsushi Sakai * Pauli Virtanen A total of 13 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Issues closed for 1.5.2 ------------------------------ * `#3847 <https://github.com/scipy/scipy/issues/3847>`__: Crash of interpolate.splprep(task=-1) * `#7395 <https://github.com/scipy/scipy/issues/7395>`__: splprep segfaults if fixed knots are specified * `#10761 <https://github.com/scipy/scipy/issues/10761>`__: scipy.signal.convolve2d produces incorrect values for large arrays * `#11971 <https://github.com/scipy/scipy/issues/11971>`__: DOC: search in devdocs returns wrong link * `#12155 <https://github.com/scipy/scipy/issues/12155>`__: BUG: Fix permutation of distance matrices in scipy.stats.multiscale_graphcorr * `#12203 <https://github.com/scipy/scipy/issues/12203>`__: Unable to install on PyPy 7.3.1 (Python 3.6.9) * `#12316 <https://github.com/scipy/scipy/issues/12316>`__: negative scipy.spatial.distance.correlation * `#12422 <https://github.com/scipy/scipy/issues/12422>`__: BUG: slsqp: ValueError: failed to initialize intent(inout) array... * `#12428 <https://github.com/scipy/scipy/issues/12428>`__: stats.truncnorm.rvs() never returns a scalar in 1.5 * `#12441 <https://github.com/scipy/scipy/issues/12441>`__: eigvalsh inconsistent eigvals= subset_by_index= * `#12445 <https://github.com/scipy/scipy/issues/12445>`__: DOC: scipy.linalg.eigh * `#12449 <https://github.com/scipy/scipy/issues/12449>`__: Warnings are not filtered in csr_matrix.sum() * `#12469 <https://github.com/scipy/scipy/issues/12469>`__: SciPy 1.9 exception in LSQSphereBivariateSpline * `#12487 <https://github.com/scipy/scipy/issues/12487>`__: BUG: optimize: incorrect result from approx_fprime * `#12493 <https://github.com/scipy/scipy/issues/12493>`__: CI: GitHub Actions for maintenance branches * `#12533 <https://github.com/scipy/scipy/issues/12533>`__: eigh gives incorrect results * `#12579 <https://github.com/scipy/scipy/issues/12579>`__: BLD, MAINT: distutils issues in wheels repo Pull requests for 1.5.2 ------------------------------- * `#12156 <https://github.com/scipy/scipy/pull/12156>`__: BUG: Fix permutation of distance matrices in scipy.stats.multiscale_graphcorr * `#12238 <https://github.com/scipy/scipy/pull/12238>`__: BUG: Use 64-bit indexing in convolve2d to avoid overflow * `#12256 <https://github.com/scipy/scipy/pull/12256>`__: BLD: Build lsap as a single extension instead of extension +... * `#12320 <https://github.com/scipy/scipy/pull/12320>`__: BUG: spatial: avoid returning negative correlation distance * `#12383 <https://github.com/scipy/scipy/pull/12383>`__: ENH: Make cKDTree.tree more efficient * `#12392 <https://github.com/scipy/scipy/pull/12392>`__: DOC: update scipy-sphinx-theme * `#12430 <https://github.com/scipy/scipy/pull/12430>`__: BUG: truncnorm and geninvgauss never return scalars from rvs * `#12437 <https://github.com/scipy/scipy/pull/12437>`__: BUG: optimize: cast bounds to floats in new_bounds_to_old/old_bounds_to_new * `#12442 <https://github.com/scipy/scipy/pull/12442>`__: MAINT:linalg: Fix for input args of eigvalsh * `#12461 <https://github.com/scipy/scipy/pull/12461>`__: MAINT: sparse: write matrix/asmatrix wrappers without warning... * `#12478 <https://github.com/scipy/scipy/pull/12478>`__: BUG: fix array_like input defects and add tests for all functions... * `#12488 <https://github.com/scipy/scipy/pull/12488>`__: BUG: fix approx_derivative step size. Closes #12487 * `#12500 <https://github.com/scipy/scipy/pull/12500>`__: CI: actions branch trigger fix * `#12501 <https://github.com/scipy/scipy/pull/12501>`__: CI: actions branch trigger fix * `#12504 <https://github.com/scipy/scipy/pull/12504>`__: BUG: cKDTreeNode use after free * `#12529 <https://github.com/scipy/scipy/pull/12529>`__: MAINT: allow graceful docs re-upload * `#12538 <https://github.com/scipy/scipy/pull/12538>`__: BUG:linalg: eigh type parameter handling corrected * `#12560 <https://github.com/scipy/scipy/pull/12560>`__: MAINT: truncnorm.rvs compatibility for \`Generator\` * `#12562 <https://github.com/scipy/scipy/pull/12562>`__: redo gh-12188: fix segfaults in splprep with fixed knots * `#12586 <https://github.com/scipy/scipy/pull/12586>`__: BLD: Add -std=c99 to sigtools to compile with C99 * `#12590 <https://github.com/scipy/scipy/pull/12590>`__: CI: Add GCC 4.8 entry to travis build matrix * `#12591 <https://github.com/scipy/scipy/pull/12591>`__: BLD: fix cython error on master-branch cython Checksums ========= MD5 ~~~ 2e046d26cdc4241a6a5b2907d57528df scipy-1.5.2-cp36-cp36m-macosx_10_9_x86_64.whl 902dea66453e2fa0616e9479970986f5 scipy-1.5.2-cp36-cp36m-manylinux1_i686.whl e130db080706d9f4ce22d8493c8e1ce2 scipy-1.5.2-cp36-cp36m-manylinux1_x86_64.whl 721f16bae600731e479a5b4e98ce9a97 scipy-1.5.2-cp36-cp36m-win32.whl a3171cfe38618d51acbfb8d1b39ac612 scipy-1.5.2-cp36-cp36m-win_amd64.whl c9f733d4d2e82c098c08760963dafaf8 scipy-1.5.2-cp37-cp37m-macosx_10_9_x86_64.whl 53ba6c502d09145b38e0e857b2d4a273 scipy-1.5.2-cp37-cp37m-manylinux1_i686.whl b9db33944ac4147936a7f42df8e95ad2 scipy-1.5.2-cp37-cp37m-manylinux1_x86_64.whl be9e8bfdf0e5e0914d1e1605be26d9c0 scipy-1.5.2-cp37-cp37m-win32.whl 848fa7b82a25d0ce36710ccc47ebc2ca scipy-1.5.2-cp37-cp37m-win_amd64.whl 590cd3b70a2dc8664896d6b9e2e5fc6d scipy-1.5.2-cp38-cp38-macosx_10_9_x86_64.whl 7fdbb19c15702b98319ea4ea32df8458 scipy-1.5.2-cp38-cp38-manylinux1_i686.whl 301f3a873e1bfef70d6f594c489fafe8 scipy-1.5.2-cp38-cp38-manylinux1_x86_64.whl 8c08ac0f55810e89e336eb3bf5a7b337 scipy-1.5.2-cp38-cp38-win32.whl 711f5c47c801dc79bead7d40669fd8c9 scipy-1.5.2-cp38-cp38-win_amd64.whl 620fc39f371e04a76af5d0290f8d3753 scipy-1.5.2.tar.gz 5bc188f21054a2ecff74fae40dd298da scipy-1.5.2.tar.xz 17bc80802955d100f6c1335594eda29a scipy-1.5.2.zip SHA256 ~~~~~~ cca9fce15109a36a0a9f9cfc64f870f1c140cb235ddf27fe0328e6afb44dfed0 scipy-1.5.2-cp36-cp36m-macosx_10_9_x86_64.whl 1c7564a4810c1cd77fcdee7fa726d7d39d4e2695ad252d7c86c3ea9d85b7fb8f scipy-1.5.2-cp36-cp36m-manylinux1_i686.whl 07e52b316b40a4f001667d1ad4eb5f2318738de34597bd91537851365b6c61f1 scipy-1.5.2-cp36-cp36m-manylinux1_x86_64.whl d56b10d8ed72ec1be76bf10508446df60954f08a41c2d40778bc29a3a9ad9bce scipy-1.5.2-cp36-cp36m-win32.whl 8e28e74b97fc8d6aa0454989db3b5d36fc27e69cef39a7ee5eaf8174ca1123cb scipy-1.5.2-cp36-cp36m-win_amd64.whl 6e86c873fe1335d88b7a4bfa09d021f27a9e753758fd75f3f92d714aa4093768 scipy-1.5.2-cp37-cp37m-macosx_10_9_x86_64.whl a0afbb967fd2c98efad5f4c24439a640d39463282040a88e8e928db647d8ac3d scipy-1.5.2-cp37-cp37m-manylinux1_i686.whl eecf40fa87eeda53e8e11d265ff2254729d04000cd40bae648e76ff268885d66 scipy-1.5.2-cp37-cp37m-manylinux1_x86_64.whl 315aa2165aca31375f4e26c230188db192ed901761390be908c9b21d8b07df62 scipy-1.5.2-cp37-cp37m-win32.whl ec5fe57e46828d034775b00cd625c4a7b5c7d2e354c3b258d820c6c72212a6ec scipy-1.5.2-cp37-cp37m-win_amd64.whl fc98f3eac993b9bfdd392e675dfe19850cc8c7246a8fd2b42443e506344be7d9 scipy-1.5.2-cp38-cp38-macosx_10_9_x86_64.whl a785409c0fa51764766840185a34f96a0a93527a0ff0230484d33a8ed085c8f8 scipy-1.5.2-cp38-cp38-manylinux1_i686.whl 0a0e9a4e58a4734c2eba917f834b25b7e3b6dc333901ce7784fd31aefbd37b2f scipy-1.5.2-cp38-cp38-manylinux1_x86_64.whl dac09281a0eacd59974e24525a3bc90fa39b4e95177e638a31b14db60d3fa806 scipy-1.5.2-cp38-cp38-win32.whl 92eb04041d371fea828858e4fff182453c25ae3eaa8782d9b6c32b25857d23bc scipy-1.5.2-cp38-cp38-win_amd64.whl 066c513d90eb3fd7567a9e150828d39111ebd88d3e924cdfc9f8ce19ab6f90c9 scipy-1.5.2.tar.gz 28d5d2e9af6ca5c0352cd83fb64191f2d8e883ab5287a221ba7a175c8cc2ccbe scipy-1.5.2.tar.xz a9054595a370f24d68f7a694037316b69ae80f5837323d567f76cde055189c08 scipy-1.5.2.zip

1 0

ENH: Adding Graph Embedding Functionality to SciPy
by Ali Saad-Eldin 24 Jul '20

24 Jul '20

Hi all, I’m curious whether the development team would be interested in the addition of graph embedding functionality, specifically adding Adjacency<https://doi.org/10.1080/01621459.2012.699795>/Laplacian<https://projecteuclid.org/euclid.aos/1534492839> spectral embedding to scipy.sparse.csgraph (or anywhere else it would make sense to have it). Both methods are useful in a variety of graph applications, such as clustering nodes with similar connective structure; in essence, they both amount to an SVD. Our current implementations in GraSPy<https://graspy.neurodata.io/tutorials/embedding/adjacencyspectralembed>, (available at ASE<https://github.com/neurodata/graspy/blob/ase-embed/graspy/embed/ase.py> and LSE<https://github.com/neurodata/graspy/blob/ase-embed/graspy/embed/lse.py>) input a graph represented as a dense or sparse matrix, and return the appropriate embedding. Please let me know if you'd like to add these functions to scipy. Best, Ali Saad-Eldin

1 0