Mailman 3 March 2021 - SciPy-Dev

New feature: fitting distributions to censored data.
by Warren Weckesser Aug. 29, 2022

Aug. 29, 2022

Hey folks, A new enhancement for fitting probability distributions to censored data by maximum likelihood estimation is in progress in https://github.com/scipy/scipy/pull/13699 This is a feature in the statistics roadmap, and is part of the work for the CZI EOSS cycle 1 grant. Censored data is data where the true value of the measurement is unknown, but there is a known bound. The common terminology is: * left-censored: The true value is greater than some known bound. * right-censored: The true value is less than some known bound. * interval-censored: The true value is between known bounds. (By allowing the bounds to be +inf and -inf, we can think of all these cases as interval-censored.) In the PR, a new class, `CensoredData`, is defined to represent censored data. The `fit` method of the univariate continuous distributions is updated to accept an instance of `CensoredData`. The `CensoredData` class constructor accepts two arrays, `lower` and `upper`, to hold the known bounds of the data. (If `lower[i] == upper[i]`, it means that data value is not censored.) Because it is quite common for data to be encountered where the only censored values in the data set are either all right-censored or all left-censored, two convenience methods, `CensoredData.right_censored(x, censored)` and `CensoredData.left_censored(x, censored)`, are provide to create a `CensoredData` instance from an array of values and a corresponding boolean array that indicates if the value is censored. Here's a quick example, with data from https://www.itl.nist.gov/div898/handbook/apr/section4/apr413.htm. The data table shows 10 regular values and 10 right-censored values. The results reported there for fitting the two-parameter Weibull distribution (location fixed at 0) to that data are shape = 1.7208 scale = 606.5280 Here's the calculation using the proposed API in SciPy: In [55]: from scipy.stats import weibull_min, CensoredData Create the `CensoredData` instance: In [56]: x = np.array([54, 187, 216, 240, 244, 335, 361, 373, 375, 386] + [500]*10) In [57]: data = CensoredData.right_censored(x, x == 500) In [58]: print(data) CensoredData(20 values: 10 not censored, 10 right-censored) Fit `weibull_min` (with the location fixed at 0) to the censored data: In [59]: shape, loc, scale = weibull_min.fit(data, floc=0) In [60]: shape Out[60]: 1.720797180719942 In [61]: scale Out[61]: 606.527565269458 Matt Haberland has already suggested quite a few improvements to the PR. Additional comments would be appreciated. Warren

4 7

Global Optimization Benchmarks
by Andrea Gavana July 11, 2021

July 11, 2021

Dear SciPy Developers & Users, long time no see :-) . I thought to start 2021 with a bit of a bang, to try and forget how bad 2020 has been... So I am happy to present you with a revamped version of the Global Optimization Benchmarks from my previous exercise in 2013. This new set of benchmarks pretty much superseeds - and greatly expands - the previous analysis that you can find at this location: http://infinity77.net/global_optimization/ . The approach I have taken this time is to select as many benchmark test suites as possible: most of them are characterized by test function *generators*, from which we can actually create almost an unlimited number of unique test problems. Biggest news are: 1. This whole exercise is made up of *6,825* test problems divided across *16* different test suites: most of these problems are of low dimensionality (2 to 6 variables) with a few benchmarks extending to 9+ variables. With all the sensitivities performed during this exercise on those benchmarks, the overall grand total number of functions evaluations stands at *3,859,786,025* - close to *4 billion*. Not bad. 2. A couple of "new" optimization algorithms I have ported to Python: - MCS: Multilevel Coordinate Search <https://www.mat.univie.ac.at/~neum/software/mcs/>, it’s my translation to Python of the original Matlab code from A. Neumaier and W. Huyer (giving then for free also GLS and MINQ) I have added a few, minor improvements compared to the original implementation. - BiteOpt: BITmask Evolution OPTimization <https://github.com/avaneev/biteopt> , I have converted the C++ code into Python and added a few, minor modifications. Enough chatting for now. The 13 tested algorithms are described here: http://infinity77.net/go_2021/ High level description & results of the 16 benchmarks: http://infinity77.net/go_2021/thebenchmarks.html Each benchmark test suite has its own dedicated page, with more detailed results and sensitivities. List of tested algorithms: 1. *AMPGO*: Adaptive Memory Programming for Global Optimization: this is my Python implementation of the algorithm described here: http://leeds-faculty.colorado.edu/glover/fred%20pubs/416%20-%20AMP%20(TS)%2… I have added a few improvements here and there based on my Master Thesis work on the standard Tunnelling Algorithm of Levy, Montalvo and Gomez. After AMPGO was integrated in lmfit <https://github.com/lmfit/lmfit-py>, I have improved it even more - in my opinion. 2. *BasinHopping*: Basin hopping is a random algorithm which attempts to find the global minimum of a smooth scalar function of one or more variables. The algorithm was originally described by David Wales: http://www-wales.ch.cam.ac.uk/ BasinHopping is now part of the standard SciPy distribution. 3. *BiteOpt*: BITmask Evolution OPTimization, based on the algorithm presented in this GitHub link: https://github.com/avaneev/biteopt I have converted the C++ code into Python and added a few, minor modifications. 4. *CMA-ES*: Covariance Matrix Adaptation Evolution Strategy, based on the following algorithm: http://www.lri.fr/~hansen/cmaesintro.html http://www.lri.fr/~hansen/cmaes_inmatlab.html#python (Python code for the algorithm) 5. *CRS2*: Controlled Random Search with Local Mutation, as implemented in the NLOpt <http://ab-initio.mit.edu/wiki/index.php/NLopt> package: http://ab-initio.mit.edu/wiki/index.php/NLopt_Algorithms#Controlled_Random_… 6. *DE*: Differential Evolution, described in the following page: http://www1.icsi.berkeley.edu/~storn/code.html DE is now part of the standard SciPy distribution, and I have taken the implementation as it stands in SciPy: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.differe… 7. *DIRECT*: the DIviding RECTangles procedure, described in: https://www.tol-project.org/export/2776/tolp/OfficialTolArchiveNetwork/NonL… http://ab-initio.mit.edu/wiki/index.php/NLopt_Algorithms#DIRECT_and_DIRECT-L (Python code for the algorithm) 8. *DualAnnealing*: the Dual Annealing algorithm, taken directly from the SciPy implementation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.dual_an… 9. *LeapFrog*: the Leap Frog procedure, which I have been recommended for use, taken from: https://github.com/flythereddflagg/lpfgopt 10. *MCS*: Multilevel Coordinate Search, it’s my translation to Python of the original Matlab code from A. Neumaier and W. Huyer (giving then for free also GLS <https://www.mat.univie.ac.at/~neum/software/ls/> and MINQ <https://www.mat.univie.ac.at/~neum/software/minq/>): https://www.mat.univie.ac.at/~neum/software/mcs/ I have added a few, minor improvements compared to the original implementation. See the MCS <http://infinity77.net/go_2021/mcs.html#mcs> section for a quick and dirty comparison between the Matlab code and my Python conversion. 11. *PSWARM*: Particle Swarm optimization algorithm, it has been described in many online papers. I have used a compiled version of the C source code from: http://www.norg.uminho.pt/aivaz/pswarm/ 12. *SCE*: Shuffled Complex Evolution, described in: Duan, Q., S. Sorooshian, and V. Gupta, Effective and efficient global optimization for conceptual rainfall-runoff models, Water Resour. Res., 28, 1015-1031, 1992. The version I used was graciously made available by Matthias Cuntz via a personal e-mail. 13. *SHGO*: Simplicial Homology Global Optimization, taken directly from the SciPy implementation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.shgo.ht… List of benchmark test suites: 1. SciPy Extended <http://infinity77.net/go_2021/scipy_extended.html#scipy-extended>: 235 multivariate problems (where the number of independent variables ranges from 2 to 17), again with multiple local/global minima. I have added about 40 new functions to the standard SciPy benchmarks <https://github.com/scipy/scipy/tree/master/benchmarks/benchmarks/go_benchma…> and fixed a few bugs in the existing benchmark models in the SciPy repository. 2. GKLS <http://infinity77.net/go_2021/gkls.html#gkls>: 1,500 test functions, with dimensionality varying from 2 to 6, generated with the super famous GKLS Test Functions Generator <https://arxiv.org/abs/1103.2695>. I have taken the original C code (available at http://netlib.org/toms/) and converted it to Python. 3. GlobOpt <http://infinity77.net/go_2021/globopt.html#globopt>: 288 tough problems, with dimensionality varying from 2 to 5, created with another test function generator which I arbitrarily named “GlobOpt”: https://www.researchgate.net/publication/225566516_A_new_class_of_test_func… . The original code is in C++ and I have bridged it to Python using Cython. *Many thanks* go to Professor Marco Locatelli for providing an updated copy of the C++ source code. 4. MMTFG <http://infinity77.net/go_2021/mmtfg.html#mmtfg>: sort-of an acronym for “Multi-Modal Test Function with multiple Global minima”, this test suite implements the work of Jani Ronkkonen: https://www.researchgate.net/publication/220265526_A_Generator_for_Multimod… . It contains 981 test problems with dimensionality varying from 2 to 4. The original code is in C and I have bridge it to Python using Cython. 5. GOTPY <http://infinity77.net/go_2021/gotpy.html#gotpy>: a generator of benchmark functions using the Bocharov-Feldbaum “Method-Min”, containing 400 test problems with dimensionality varying from 2 to 5. I have taken the Python implementation from https://github.com/redb0/gotpy and improved it in terms of runtime. Original paper from http://www.mathnet.ru/php/archive.phtml?wshow=paper&jrnid=at&paperid=11985&… . 6. Huygens <http://infinity77.net/go_2021/huygens.html#huygens>: this benchmark suite is very different from the rest, as it uses a “fractal” approach to generate test functions. It is based on the work of Cara MacNish on Fractal Functions <https://www.cs.bham.ac.uk/research/projects/ecb/displayDataset.php?id=150>. The original code is in Java, and at the beginning I just converted it to Python: given it was slow as a turtle, I have re-implemented it in Fortran and wrapped it using f2py <http://www.scipy.org/F2py>, then generating 600 2-dimensional test problems out of it. 7. LGMVG <http://infinity77.net/go_2021/lgmvg.html#lgmvg>: not sure about the meaning of the acronym, but the implementation follows the “Max-Set of Gaussians Landscape Generator” described in http://boyuan.global-optimization.com/LGMVG/index.htm . Source code is given in Matlab, but it’s fairly easy to convert it to Python. This test suite contains 304 problems with dimensionality varying from 2 to 5. 8. NgLi <http://infinity77.net/go_2021/ngli.html#ngli>: Stemming from the work of Chi-Kong Ng and Duan Li, this is a test problem generator for unconstrained optimization, but it’s fairly easy to assign bound constraints to it. The methodology is described in https://www.sciencedirect.com/science/article/pii/S0305054814001774 , while the Matlab source code can be found in http://www1.se.cuhk.edu.hk/~ckng/generator/ . I have used the Matlab script to generate 240 problems with dimensionality varying from 2 to 5 by outputting the generator parameters in text files, then used Python to create the objective functions based on those parameters and the benchmark methodology. 9. MPM2 <http://infinity77.net/go_2021/mpm2.html#mpm2>: Implementing the “Multiple Peaks Model 2”, there is a Python implementation at https://github.com/jakobbossek/smoof/blob/master/inst/mpm2.py . This is a test problem generator also used in the smoof <https://github.com/jakobbossek/smoof> library, I have taken the code almost as is and generated 480 benchmark functions with dimensionality varying from 2 to 5. 10. RandomFields <http://infinity77.net/go_2021/randomfields.html#randomfields>: as described in https://www.researchgate.net/publication/301940420_Global_optimization_test… , it generates benchmark functions by “smoothing” one or more multidimensional discrete random fields and composing them. No source code is given, but the implementation is fairly straightforward from the article itself. 11. NIST <http://infinity77.net/go_2021/nist.html#nist>: not exactly the realm of Global Optimization solvers, but the NIST StRD <https://www.itl.nist.gov/div898/strd/nls/nls_main.shtml> dataset can be used to generate a single objective function as “sum of squares”. I have used the NIST dataset as implemented in lmfit <https://github.com/lmfit/lmfit-py/tree/master/NIST_STRD>, thus creating 27 test problems with dimensionality ranging from 2 to 9. 12. GlobalLib <http://infinity77.net/go_2021/globallib.html#globallib>: Arnold Neumaier maintains <https://www.mat.univie.ac.at/~neum/glopt/coconut/Benchmark/Benchmark.html> a suite of test problems termed “COCONUT Benchmark” and Sahinidis has converted the GlobalLib and PricentonLib AMPL/GAMS dataset into C/Fortran code (http://archimedes.cheme.cmu.edu/?q=dfocomp ). I have used a simple C parser to convert the benchmarks from C to Python. The global minima are taken from Sahinidis <http://thales.cheme.cmu.edu/dfo/comparison/comp.html> or from Neumaier or refined using the NEOS server <https://neos-server.org/neos/> when the accuracy of the reported minima is too low. The suite contains 181 test functions with dimensionality varying between 2 and 9. 13. CVMG <http://infinity77.net/go_2021/cvmg.html#cvmg>: another “landscape generator”, I had to dig it out using the Wayback Machine at http://web.archive.org/web/20100612044104/https://www.cs.uwyo.edu/~wspears/… , the acronym stands for “Continuous Valued Multimodality Generator”. Source code is in C++ but it’s fairly easy to port it to Python. In addition to the original implementation (that uses the Sigmoid <https://en.wikipedia.org/wiki/Sigmoid_function> as a softmax/transformation function) I have added a few others to create varied landscapes. 360 test problems have been generated, with dimensionality ranging from 2 to 5. 14. NLSE <http://infinity77.net/go_2021/nlse.html#nlse>: again, not really the realm of Global optimization solvers, but Nonlinear Systems of Equations can be transformed to single objective functions to optimize. I have drawn from many different sources (Publications <https://www.sciencedirect.com/science/article/pii/S0898122111004299>, ALIAS/COPRIN <http://www-sop.inria.fr/coprin/logiciels/ALIAS/Benches/node1.html#SECTION00…> and many others) to create 44 systems of nonlinear equations with dimensionality ranging from 2 to 8. 15. Schoen <http://infinity77.net/go_2021/schoen.html#schoen>: based on the early work of Fabio Schoen and his short note <https://link.springer.com/article/10.1007%2FBF01096734> on a simple but interesting idea on a test function generator, I have taken the C code in the note and converted it into Python, thus creating 285 benchmark functions with dimensionality ranging from 2 to 6. *Many thanks* go to Professor Fabio Schoen for providing an updated copy of the source code and for the email communications. 16. Robust <http://infinity77.net/go_2021/robust.html#robust>: the last benchmark test suite for this exercise, it is actually composed of 5 different kind-of analytical test function generators, containing deceptive, multimodal, flat functions depending on the settings. Matlab source code is available at http://www.alimirjalili.com/RO.html , I simply converted it to Python and created 420 benchmark functions with dimensionality ranging from 2 to 6. Enjoy, and Happy 2021 :-) . Andrea.

6 21

Regarding contribution in Scipy organization for GSoC 21
by Amulya Gupta May 20, 2021

May 20, 2021

Dear Sir/Ma'am, I am Amulya Gupta, a sophomore from BITS,Pilani. I have been coding for the past 3 years. I am profound in Python(Modules: nose test, poetry, numpy, matplotlib, cmake, pipenv, tkinter, turtle and currently learning mypy etc), C++, Mysql, Java script. I have experience in open source with an internship with an organisation called NumFocus. I want to openly contribute to the project "*Add type annotations to a submodule*" from SciPy. Please can I talk to the mentor about his expectations from the student applying. How many more students have applied for the same project and how does the mentor has pre-planned about the project and his inputs for the proposal buildup. Regards Amulya Gupta

2 7

Adding Z-test to scipy.stats
by Xingyu Liu April 12, 2021

April 12, 2021

Hi everyone: I'm Xingyu Liu, a first year Data Science master student at Harvard University, and I'm thinking of adding Z-test to scipy.stats. Z-test is one of the most basic types of hypothesis test and is covered in almost all the statistics textbooks (e.g. Statistics by David Freedman, et,al.) . However, currently, Z-test is not implemented in scipy(see this feature request issue https://github.com/scipy/scipy/issues/13662 ). Its principles are quite similar to t-test except that it uses the population variance rather than the sample variance for calculation, which means that the population variance shoud be known in advance. In application, Z-test is used when sample size is large (n>50), or the population variance is known; t-test is used when sample size is small (n<50) and population variance is unknown(https://en.wikipedia.org/wiki/Z-test). There are mainly three types of Z-test and the coding part can be quite similar to t-test: 1. One Sample Z-Test: Does the mean of the sample differ from the expected mean? 2. Two Independent Sample Z-Test: Do the means of two independent samples differ? 3. Paired-Sample Z-Test: Do the means of the same sample differ before and after? Do you think it would be helpful to do this enhancement? If it is, I can work on it :) Cheers, Xingyu

2 2

Proposal to merge linalg.pinv and linalg.pinv2 and deprecate pinv2
by Ilhan Polat April 10, 2021

April 10, 2021

Currently, scipy has pinv, pinv2, pinvh pseudo inverse functions. The first one sends the array to the least squares solver to solve <rectangular matrix> @ X = Identity with minimum-norm solution And the other two forms the SVD based construction by inverting the SV/eig diagonal and reforming the product in the reverse order i.e. the typical pseudoinverse. However, the first one already does this in the lower level driver anyways since that's what gelss driver does [1]. Hence I think it is better to merge these and get rid of one as an alias in the deprecation cycle. Thoughts? ilhan [1] https://www.netlib.org/lapack/lug/node27.html

4 6

pydata-sphinx-theme
by Pamphile Roy April 6, 2021

April 6, 2021

Hi everyone, I was wondering if there was a plan about moving the doc to using PyData’s theme? (I could not find any issues or ref in mail, sorry if I missed something). IMO it’s good if we have a UX which is close to NumPy and the other libraries of the stack. Although for the landing page, I would propose to have fewer things like Pandas. I could propose to work on this, if this is something we want. Cheers, Pamphile

2 4

Using more functionalities from GitHub?
by Pamphile Roy April 5, 2021

April 5, 2021

Hi everyone, I would like to propose to use more functionalities of GitHub. I could not find any reference showing this was discussed before, so I apologize if the following would not be relevant. Teams Currently there is just the triage team, but we could create other teams like one per submodule or for given skill sets. This could allow easier pinging of people on PR. Project We have the roadmap and some issues are acting as meta-issues. But we could use, instead/on top, the integrated project management dashboard. It would be more convenient to keep track of what can be done, who is working on what area, etc. Also, this would clearly be in favor of openness and transparency with the management of the library. Cheers, Pamphile

3 4

Move some validations as asserts?
by Pamphile Roy March 31, 2021

March 31, 2021

Hi everyone, TL;DR: could we use assert to do validation? In a recent issue (https://github.com/scipy/scipy/issues/13629), the cost of validation was raised in context such as optimization. In situations like these, or even when you know how to use the API, there are lots of unnecessary validations happening. I would propose a general reflection around validation. Some ideas: A simple approach would be to systematically separate the logic in two: core function on one side, user interface on the other. This way the user could by-pass all the time the validation. Mock the validations. After some quick tests, it looks like the overhead is too big. I could be wrong. Using assert for all validation. The advantage of the assert is that validation would be cleanly identified in the code (starlette is full of these for instance). This would be an advanced feature as most users don’t know about asserts being removed if you launch python in optimized mode. I am interested in reading your thoughts. If one solution would be chosen, I could work on it too. Cheers, Pamphile

4 7

Github actions feedback
by Andrew Nelson March 30, 2021

March 30, 2021

I'm meeting with the github actions team tomorrow for a feedback session. I was going to mention our desire to test on aarch64. Does anyone else have feedback on what we'd like to see implemented in CI, Github actions or elsewhere? Andrew. -- _____________________________________ Dr. Andrew Nelson _____________________________________

3 2

NumPy 1.20.2 released.
by Charles R Harris March 27, 2021

March 27, 2021

Charles R Harris <charlesr.harris(a)gmail.com> Sun, Feb 7, 2:23 PM to numpy-discussion, SciPy, SciPy-User, bcc: python-announce-list Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.20.2. NumPy 1,20.2 is a bugfix release containing several fixes merged to the main branch after the NumPy 1.20.1 release. The Python versions supported for this release are 3.7-3.9. Wheels can be downloaded from PyPI <https://pypi.org/project/numpy/1.20.2/>; source archives, release notes, and wheel hashes are available on Github <https://github.com/numpy/numpy/releases/tag/v1.20.2>. Linux users will need pip >= 0.19.3 in order to install manylinux2010 and manylinux2014 wheels. *Contributors* A total of 7 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Allan Haldane - Bas van Beek - Charles Harris - Christoph Gohlke - Mateusz Sokół + - Michael Lamparski - Sebastian Berg *Pull requests merged* A total of 20 pull requests were merged for this release. - #18382: MAINT: Update f2py from master. - #18459: BUG: ``diagflat`` could overflow on windows or 32-bit platforms - #18460: BUG: Fix refcount leak in f2py ``complex_double_from_pyobj``. - #18461: BUG: Fix tiny memory leaks when ``like=`` overrides are used - #18462: BUG: Remove temporary change of descr/flags in VOID functions - #18469: BUG: Segfault in nditer buffer dealloc for Object arrays - #18485: BUG: Remove suspicious type casting - #18486: BUG: remove nonsensical comparison of pointer < 0 - #18487: BUG: verify pointer against NULL before using it - #18488: BUG: check if PyArray_malloc succeeded - #18546: BUG: incorrect error fallthrough in nditer - #18559: CI: Backport CI fixes from main. - #18599: MAINT: Add annotations for `dtype.__getitem__`, `__mul__` and... - #18611: BUG: NameError in numpy.distutils.fcompiler.compaq - #18612: BUG: Fixed ``where`` keyword for ``np.mean`` & ``np.var`` methods - #18617: CI: Update apt package list before Python install - #18636: MAINT: Ensure that re-exported sub-modules are properly annotated - #18638: BUG: Fix ma coercion list-of-ma-arrays if they do not cast to... - #18661: BUG: Fix small valgrind-found issues - #18671: BUG: Fix small issues found with pytest-leaks Cheers, Charles Harris

1 0