Mailman 3 October 2017 - SciPy-Dev

Are you using Bento to build scipy?
by CJ Carey 16 Jul '18

16 Jul '18

With the 1.0 release coming up, I figured this is a good time to take the pulse of the community regarding Bento. The project seems to have been abandoned before reaching stability, and it seems that the core problems it was trying to address have been at least partially fixed by the more official build systems. If you use Bento and would like to keep using it post 1.0, or if my previous sentence is wrong, please let me know!

10 15

Stochastic calculus package
by Maurizio Cipollina 15 Jun '18

15 Jun '18

Hi everybody, and first of all thanks for the great job you have been doing, on which I relied *A LOT* over time. Stochastic calculus and Stochastic Differential Equations are a significant branch of mathematics that to my understanding is underrepresented in scipy, as well as in the open source python stack at large. I may well be missing something, but as I can see one can find out there many recipies or collections of recipies, and a number of packages/projects skewed towards finance, but no structured framework for generating and handling stochastic processes and numerically solving SDEs. This is a pity, since scipy/numpy provide efficen implementations of all the basic ingredients needed to do the trick. Out of professional need, and prompted by this gap, I have developed a general purpose package of which I own the copyright, and which I would be happy to release under the BSD license as part of scipy, in case there is a consensus among the scipy community that it makes sense to do so (the package has no dependencies beyond standard library, numpy and scipy, and might probably fit best as a subpackage of scipy.stats). Before setting up a PR, I would be happy to hear your thoughts: I have pasted below the package docstring, that should give an overview of its scope and limitations. Please note that some of the realized processes stray into mathematical finance territory, somehow inevitably since finance is one major field of application of stochastic calculus, but the focus is the latter, not the former, at least in my intentions. Thanks in advance for taking your time to go through this, and for your comments and suggestions. Maurizio “”” =========================================================== Stochastic - Monte Carlo simulation of stochastic processes =========================================================== This package provides: 1. A `process` class, a subclass of `numpy.ndarray` representing a sequence of values in time, realized in one or several paths. Algebraic manipulations and ufunc computations are supported for instances that share the same timeline and comply with numpy broadcasting rules. Interpolation along the timeline is supported via callability of `process` instances. Process-specific functionalities, such as averaging and indexing along time or across paths, are delegated to process-specific methods, attributes and properties (no overriding of `numpy.ndarray` operations). 2. The `source` class and its subclasses, stochasticity sources providing numerical realizations of the differentials commonly found in stochastic differential equations (SDEs), with or without memory of formerly invoked realizations. 3. The `integrator` class, a general framework for numerical SDE integration, intended for subclassing, to assist the definition of step by step integration algorithms and the computation of numerical realizations of stochastic processes. 4. Several objects providing realizations of specific stochastic processes along a given timeline and across a requested number of paths. Realizations are obtained via explicit formulae, in case the relevant SDE has a closed form solution, or otherwise as `integrator` subclasses performing Euler-Maruyama numerical integration. 5. A `montecarlo` class, as an aid to cumulate the results of several Monte Carlo simulations of a given stochastic variable, and to extract summary estimates for its probability distribution function and statistics, thus supporting simulations across a number of paths that exceeds the maximum allowed by available memory. 6. Several analytical results relating to the supported stochastic processes, as a general reference and for testing purpouses. For all sources and realizations, process values can take any shape, scalar or multidimensional. Correlated multivariate stochasticity sources are supported. Time-varying process parameters (correlations, intensity of Poisson processes, volatilities etc.) are allowed whenever applicable. `process` instances act as valid stochasticity source realizations (as does any callable object complying with a `source` protocol), and may be passed as a source specification when computing the realization of a given process. Computations are fully vectorized across paths, providing an efficient infrastructure for simulating a large number of process realizations. Less so, for large number of time steps: integrating 1000 time steps across 100000 paths takes seconds, one million time steps across 100 paths takes minutes. General infrastructure ====================== .. autosummary:: :toctree: generated/ process -- Array representation of a process (a subclass of -- numpy.ndarray). kfunc -- Base class for functions with managed keyword arguments. source -- Base class for stochasticity sources. true_source -- Base class for stochasticity sources with memory. integrator -- General framework for numerical SDE integration. montecarlo -- Summmary statistics of Monte Carlo simulations. Stochasticity sources ===================== .. autosummary:: :toctree: generated/ wiener -- dW, a source of standard Wiener process (Brownian -- motion) increments. symmetric_wiener -- as above, with symmetric paths (averages exactly 0). true_wiener -- dW, a source of standard Wiener process (Brownian -- motion) increments, with memory. poisson -- dN, a source of Poisson process increments. compound_poisson -- dJ, a source of compound Poisson process increments. -- (jumps distributed in time as a Poisson process, -- and in size as a given `scipy.stats` distribution). compound_poisson_rv -- Preset jump size distributions for compound Poisson -- process increments. Stochastic process realizations =============================== .. autosummary:: :toctree: generated/ const_wiener_process -- Wiener process (Brownian motion), with -- time-independent parameters. const_lognorm_process -- Lognormal process, with time-independent -- parameters. wiener_process -- Wiener process (Brownian motion). lognorm_process -- Lognormal process. ornstein_uhlenbeck_process -- Ornstein-Uhlenbeck process (mean-reverting -- Brownian motion). hull_white_process -- F-factors Hull-White process (sum of F -- correlated mean-reverting Brownian -- motions). hull_white_1factor_process -- 1-factor Hull-White process (F=1 Hull-White -- process with F-index collapsed to a -- scalar). cox_ingersoll_ross_process -- Cox-Ingersoll-Ross mean-reverting process. full_heston_process -- Heston stochastic volatility process -- (returns both process and volatility). heston_process -- Heston stochastic volatility process -- (stores and returns process only). jump_diffusion_process -- Jump-diffusion process (lognormal process -- with compound Poisson jumps). merton_jump_diffusion_process -- Merton jump-diffusion process -- (jump-diffusion process with normal -- jump size distribution). kou_jump_diffusion_process -- Kou jump-diffusion process (jump-diffusion -- process with double exponential -- jump size distribution). Shortcuts:: lognorm -- lognorm_process oruh -- ornstein_uhlenbeck_process hwp -- hull_white_process hw1f -- hull_white_1factor_process cir -- cox_ingersoll_ross_process heston -- heston_process mjd -- merton_jump_diffusion_process kou -- kou_jump_diffusion_process Analytical results ================== Exact statistics for the realized stochastic processes are listed below, limited to the case of constant process parameters and with some further limitations to the parameters' domains. Function arguments are consistent with those of the corresponding processes. Suffixes `pdf`, `cdf` and `chf` stand respectively for probability distribution function, cumulative probability distribution function, and characteristic function. Black-Scholes formulae for the valuation of call and put options have been included (prefixed with `bs` below). .. autosummary:: :toctree: generated/ wiener_mean wiener_var wiener_std wiener_pdf wiener_cdf wiener_chf lognorm_mean lognorm_var lognorm_std lognorm_pdf lognorm_cdf lognorm_log_chf oruh_mean oruh_var oruh_std oruh_pdf oruh_cdf hw2f_mean hw2f_var hw2f_std hw2f_pdf hw2f_cdf cir_mean cir_var cir_std cir_pdf heston_log_mean heston_log_var heston_log_std heston_log_pdf heston_log_chf mjd_log_pdf mjd_log_chf kou_mean kou_log_pdf kou_log_chf bsd1d2 bscall bscall_delta bsput bsput_delta Notes ===== To the benefit of interactive and notebook sessions, and as an aid to fluently freeze or vary the plurality of parameters that define each stochastic process, all sources, process realizations and analytical results are objects with managed keywords (subclasses of `kfunc`): if the corresponding classes are instantiated with both variables (non keyword arguments) and parameters (keyword arguments), they behave as functions returning the computations' result; if called with parameters only, return an instance that stores the set parameters, and exposes the same calling pattern (call with variables and optionally with parameters to get results, call with parameters only to get a new instance storing modified parameters). Parameters that are not specified fall back to the class defaults, if calling the class, or to the instance's stored values, if calling an instance. """

4 7

wrapper for Scalapack
by marc 30 Oct '17

30 Oct '17

Dear Scipy developers, We are actually writing a python wrapper for Scalapack that seems to be cruelly missing for computational science with Python. Since Scipy has already wrappers for Blas and Lapack library we are hoping that you could be interested as well in a Scalapack wrapper. In this aim, we used f2py to wrap the routine as it is done in the Blas/Lapack wrapper of Scipy. The actual code can be found in the following repository, https://gitlab.com/mbarbry/python-scalapack Since the project just started, only few routines are implemented, but we manage to write a working test with sygv routine. We would like to know if the Scipy community would be interested to merge this wrapper in the Scipy code in the future? We believe that Scalapack is an important part of the Scientific tools for linear algebra, and that Scipy would only benefit of such wrapper as well than the Scientific community. Contributions to the code are obviously more than welcome. Best regards, Marc Barbry

6 10

Fixing a bug with scipy's hypergeometric function hyp2f1
by Adam 29 Oct '17

29 Oct '17

Hello guys, I've found a small region in the complex plane where scipy's implementation of the hypergeometric function hyp2f1 fails. I've documented this error in issue 8054 on github. I am willing to submit a PR that fixes this issue. My PR would basically implement the analytic continuation formula given in this paper: (Buhring, An Analytic Continuation of the Hypergeometric Series). I've already implemented this series in some prototype code written in Fortran and it agrees well with the values returned by mpmath's implementation of hyp2f1. Before I attempt a PR, I just wanted to touch base and ask the group the following: 1) Does this sound like a worthwhile PR? The failure region is somewhat small and I don't know with what urgency people would want this fixed. 2) Does the implementation sound reasonable? My background is physics and so I haven't done a complete literature search looking for the *fastest* algorithm. All I can say that the Buhring's formula works and my implementation only seems to be about %50 slower than the current hyp2f1 (at points in the complex plane where both methods converge). I would only apply Buhring's series in the region where hyp2f1 currently diverges. 3) Can the PR implement formulas/methods that don't appear in the literature? Buhring's paper *only* gives the analytic continuation for the case where the difference between the a/b parameters is NOT an integer. When a-b=m, the limit case of his series can be derived using a method described in "The Special Functions and Their Approximations" by Y. Luke (as Buhling mentions in his paper). I've derived the formula for this limit case and have an implementation of it that produces values in agreement with mpmath. Is it going to be a problem if I implement this limit case in the PR? I ask because I don't what reference I would place hyp2f1's doc string. I would be wiling to maybe add a latex doc to the PR (placed somewhere in the doc folder?) that contains the formula so that future scipy devs have something to reference when reviewing hyp2f1's source code. Anyways, let me know if my idea for a PR sounds like a good idea! I apologize for the longish email, but this is my first time trying to contribute to scipy... --Adam

5 13

SciPy 1.0 released!
by Ralf Gommers 25 Oct '17

25 Oct '17

Hi all, We are extremely pleased to announce the release of SciPy 1.0, 16 years after version 0.1 saw the light of day. It has been a long, productive journey to get here, and we anticipate many more exciting new features and releases in the future. Why 1.0 now? ------------ A version number should reflect the maturity of a project - and SciPy was a mature and stable library that is heavily used in production settings for a long time already. From that perspective, the 1.0 version number is long overdue. Some key project goals, both technical (e.g. Windows wheels and continuous integration) and organisational (a governance structure, code of conduct and a roadmap), have been achieved recently. Many of us are a bit perfectionist, and therefore are reluctant to call something "1.0" because it may imply that it's "finished" or "we are 100% happy with it". This is normal for many open source projects, however that doesn't make it right. We acknowledge to ourselves that it's not perfect, and there are some dusty corners left (that will probably always be the case). Despite that, SciPy is extremely useful to its users, on average has high quality code and documentation, and gives the stability and backwards compatibility guarantees that a 1.0 label imply. Some history and perspectives ----------------------------- - 2001: the first SciPy release - 2005: transition to NumPy - 2007: creation of scikits - 2008: scipy.spatial module and first Cython code added - 2010: moving to a 6-monthly release cycle - 2011: SciPy development moves to GitHub - 2011: Python 3 support - 2012: adding a sparse graph module and unified optimization interface - 2012: removal of scipy.maxentropy - 2013: continuous integration with TravisCI - 2015: adding Cython interface for BLAS/LAPACK and a benchmark suite - 2017: adding a unified C API with scipy.LowLevelCallable; removal of scipy.weave - 2017: SciPy 1.0 release **Pauli Virtanen** is SciPy's Benevolent Dictator For Life (BDFL). He says: *Truthfully speaking, we could have released a SciPy 1.0 a long time ago, so I'm happy we do it now at long last. The project has a long history, and during the years it has matured also as a software project. I believe it has well proved its merit to warrant a version number starting with unity.* *Since its conception 15+ years ago, SciPy has largely been written by and for scientists, to provide a box of basic tools that they need. Over time, the set of people active in its development has undergone some rotation, and we have evolved towards a somewhat more systematic approach to development. Regardless, this underlying drive has stayed the same, and I think it will also continue propelling the project forward in future. This is all good, since not long after 1.0 comes 1.1.* **Travis Oliphant** is one of SciPy's creators. He says: *I'm honored to write a note of congratulations to the SciPy developers and the entire SciPy community for the release of SciPy 1.0. This release represents a dream of many that has been patiently pursued by a stalwart group of pioneers for nearly 2 decades. Efforts have been broad and consistent over that time from many hundreds of people. From initial discussions to efforts coding and packaging to documentation efforts to extensive conference and community building, the SciPy effort has been a global phenomenon that it has been a privilege to participate in.* *The idea of SciPy was already in multiple people’s minds in 1997 when I first joined the Python community as a young graduate student who had just fallen in love with the expressibility and extensibility of Python. The internet was just starting to bringing together like-minded mathematicians and scientists in nascent electronically-connected communities. In 1998, there was a concerted discussion on the matrix-SIG, python mailing list with people like Paul Barrett, Joe Harrington, Perry Greenfield, Paul Dubois, Konrad Hinsen, David Ascher, and others. This discussion encouraged me in 1998 and 1999 to procrastinate my PhD and spend a lot of time writing extension modules to Python that mostly wrapped battle-tested Fortran and C-code making it available to the Python user. This work attracted the help of others like Robert Kern, Pearu Peterson and Eric Jones who joined their efforts with mine in 2000 so that by 2001, the first SciPy release was ready. This was long before Github simplified collaboration and input from others and the "patch" command and email was how you helped a project improve.* *Since that time, hundreds of people have spent an enormous amount of time improving the SciPy library and the community surrounding this library has dramatically grown. I stopped being able to participate actively in developing the SciPy library around 2010. Fortunately, at that time, Pauli Virtanen and Ralf Gommers picked up the pace of development supported by dozens of other key contributors such as David Cournapeau, Evgeni Burovski, Josef Perktold, and Warren Weckesser. While I have only been able to admire the development of SciPy from a distance for the past 7 years, I have never lost my love of the project and the concept of community-driven development. I remain driven even now by a desire to help sustain the development of not only the SciPy library but many other affiliated and related open-source projects. I am extremely pleased that SciPy is in the hands of a world-wide community of talented developers who will ensure that SciPy remains an example of how grass-roots, community-driven development can succeed.* **Fernando Perez** offers a wider community perspective: *The existence of a nascent Scipy library, and the incredible --if tiny by today's standards-- community surrounding it is what drew me into the scientific Python world while still a physics graduate student in 2001. Today, I am awed when I see these tools power everything from high school education to the research that led to the 2017 Nobel Prize in physics.* *Don't be fooled by the 1.0 number: this project is a mature cornerstone of the modern scientific computing ecosystem. I am grateful for the many who have made it possible, and hope to be able to contribute again to it in the future. My sincere congratulations to the whole team!* Highlights of this release -------------------------- Some of the highlights of this release are: - Major build improvements. Windows wheels are available on PyPI for the first time, and continuous integration has been set up on Windows and OS X in addition to Linux. - A set of new ODE solvers and a unified interface to them (`scipy.integrate.solve_ivp`). - Two new trust region optimizers and a new linear programming method, with improved performance compared to what `scipy.optimize` offered previously. - Many new BLAS and LAPACK functions were wrapped. The BLAS wrappers are now complete. Upgrading and compatibility --------------------------- There have been a number of deprecations and API changes in this release, which are documented below. Before upgrading, we recommend that users check that their own code does not use deprecated SciPy functionality (to do so, run your code with ``python -Wd`` and check for ``DeprecationWarning`` s). This release requires Python 2.7 or >=3.4 and NumPy 1.8.2 or greater. This is also the last release to support LAPACK 3.1.x - 3.3.x. Moving the lowest supported LAPACK version to >3.2.x was long blocked by Apple Accelerate providing the LAPACK 3.2.1 API. We have decided that it's time to either drop Accelerate or, if there is enough interest, provide shims for functions added in more recent LAPACK versions so it can still be used. New features ============ `scipy.cluster` improvements ---------------------------- `scipy.cluster.hierarchy.optimal_leaf_ordering`, a function to reorder a linkage matrix to minimize distances between adjacent leaves, was added. `scipy.fftpack` improvements ---------------------------- N-dimensional versions of the discrete sine and cosine transforms and their inverses were added as ``dctn``, ``idctn``, ``dstn`` and ``idstn``. `scipy.integrate` improvements ------------------------------ A set of new ODE solvers have been added to `scipy.integrate`. The convenience function `scipy.integrate.solve_ivp` allows uniform access to all solvers. The individual solvers (``RK23``, ``RK45``, ``Radau``, ``BDF`` and ``LSODA``) can also be used directly. `scipy.linalg` improvements ---------------------------- The BLAS wrappers in `scipy.linalg.blas` have been completed. Added functions are ``*gbmv``, ``*hbmv``, ``*hpmv``, ``*hpr``, ``*hpr2``, ``*spmv``, ``*spr``, ``*tbmv``, ``*tbsv``, ``*tpmv``, ``*tpsv``, ``*trsm``, ``*trsv``, ``*sbmv``, ``*spr2``, Wrappers for the LAPACK functions ``*gels``, ``*stev``, ``*sytrd``, ``*hetrd``, ``*sytf2``, ``*hetrf``, ``*sytrf``, ``*sycon``, ``*hecon``, ``*gglse``, ``*stebz``, ``*stemr``, ``*sterf``, and ``*stein`` have been added. The function `scipy.linalg.subspace_angles` has been added to compute the subspace angles between two matrices. The function `scipy.linalg.clarkson_woodruff_transform` has been added. It finds low-rank matrix approximation via the Clarkson-Woodruff Transform. The functions `scipy.linalg.eigh_tridiagonal` and `scipy.linalg.eigvalsh_tridiagonal`, which find the eigenvalues and eigenvectors of tridiagonal hermitian/symmetric matrices, were added. `scipy.ndimage` improvements ---------------------------- Support for homogeneous coordinate transforms has been added to `scipy.ndimage.affine_transform`. The ``ndimage`` C code underwent a significant refactoring, and is now a lot easier to understand and maintain. `scipy.optimize` improvements ----------------------------- The methods ``trust-region-exact`` and ``trust-krylov`` have been added to the function `scipy.optimize.minimize`. These new trust-region methods solve the subproblem with higher accuracy at the cost of more Hessian factorizations (compared to dogleg) or more matrix vector products (compared to ncg) but usually require less nonlinear iterations and are able to deal with indefinite Hessians. They seem very competitive against the other Newton methods implemented in scipy. `scipy.optimize.linprog` gained an interior point method. Its performance is superior (both in accuracy and speed) to the older simplex method. `scipy.signal` improvements --------------------------- An argument ``fs`` (sampling frequency) was added to the following functions: ``firwin``, ``firwin2``, ``firls``, and ``remez``. This makes these functions consistent with many other functions in `scipy.signal` in which the sampling frequency can be specified. `scipy.signal.freqz` has been sped up significantly for FIR filters. `scipy.sparse` improvements --------------------------- Iterating over and slicing of CSC and CSR matrices is now faster by up to ~35%. The ``tocsr`` method of COO matrices is now several times faster. The ``diagonal`` method of sparse matrices now takes a parameter, indicating which diagonal to return. `scipy.sparse.linalg` improvements ---------------------------------- A new iterative solver for large-scale nonsymmetric sparse linear systems, `scipy.sparse.linalg.gcrotmk`, was added. It implements ``GCROT(m,k)``, a flexible variant of ``GCROT``. `scipy.sparse.linalg.lsmr` now accepts an initial guess, yielding potentially faster convergence. SuperLU was updated to version 5.2.1. `scipy.spatial` improvements ---------------------------- Many distance metrics in `scipy.spatial.distance` gained support for weights. The signatures of `scipy.spatial.distance.pdist` and `scipy.spatial.distance.cdist` were changed to ``*args, **kwargs`` in order to support a wider range of metrics (e.g. string-based metrics that need extra keywords). Also, an optional ``out`` parameter was added to ``pdist`` and ``cdist`` allowing the user to specify where the resulting distance matrix is to be stored `scipy.stats` improvements -------------------------- The methods ``cdf`` and ``logcdf`` were added to `scipy.stats.multivariate_normal`, providing the cumulative distribution function of the multivariate normal distribution. New statistical distance functions were added, namely `scipy.stats.wasserstein_distance` for the first Wasserstein distance and `scipy.stats.energy_distance` for the energy distance. Deprecated features =================== The following functions in `scipy.misc` are deprecated: ``bytescale``, ``fromimage``, ``imfilter``, ``imread``, ``imresize``, ``imrotate``, ``imsave``, ``imshow`` and ``toimage``. Most of those functions have unexpected behavior (like rescaling and type casting image data without the user asking for that). Other functions simply have better alternatives. ``scipy.interpolate.interpolate_wrapper`` and all functions in that submodule are deprecated. This was a never finished set of wrapper functions which is not relevant anymore. The ``fillvalue`` of `scipy.signal.convolve2d` will be cast directly to the dtypes of the input arrays in the future and checked that it is a scalar or an array with a single element. ``scipy.spatial.distance.matching`` is deprecated. It is an alias of `scipy.spatial.distance.hamming`, which should be used instead. Implementation of `scipy.spatial.distance.wminkowski` was based on a wrong interpretation of the metric definition. In scipy 1.0 it has been just deprecated in the documentation to keep retro-compatibility but is recommended to use the new version of `scipy.spatial.distance.minkowski` that implements the correct behaviour. Positional arguments of `scipy.spatial.distance.pdist` and `scipy.spatial.distance.cdist` should be replaced with their keyword version. Backwards incompatible changes ============================== The following deprecated functions have been removed from `scipy.stats`: ``betai``, ``chisqprob``, ``f_value``, ``histogram``, ``histogram2``, ``pdf_fromgamma``, ``signaltonoise``, ``square_of_sums``, ``ss`` and ``threshold``. The following deprecated functions have been removed from `scipy.stats.mstats`: ``betai``, ``f_value_wilks_lambda``, ``signaltonoise`` and ``threshold``. The deprecated ``a`` and ``reta`` keywords have been removed from `scipy.stats.shapiro`. The deprecated functions ``sparse.csgraph.cs_graph_components`` and ``sparse.linalg.symeig`` have been removed from `scipy.sparse`. The following deprecated keywords have been removed in `scipy.sparse.linalg`: ``drop_tol`` from ``splu``, and ``xtype`` from ``bicg``, ``bicgstab``, ``cg``, ``cgs``, ``gmres``, ``qmr`` and ``minres``. The deprecated functions ``expm2`` and ``expm3`` have been removed from `scipy.linalg`. The deprecated keyword ``q`` was removed from `scipy.linalg.expm`. And the deprecated submodule ``linalg.calc_lwork`` was removed. The deprecated functions ``C2K``, ``K2C``, ``F2C``, ``C2F``, ``F2K`` and ``K2F`` have been removed from `scipy.constants`. The deprecated ``ppform`` class was removed from `scipy.interpolate`. The deprecated keyword ``iprint`` was removed from `scipy.optimize.fmin_cobyla`. The default value for the ``zero_phase`` keyword of `scipy.signal.decimate` has been changed to True. The ``kmeans`` and ``kmeans2`` functions in `scipy.cluster.vq` changed the method used for random initialization, so using a fixed random seed will not necessarily produce the same results as in previous versions. `scipy.special.gammaln` does not accept complex arguments anymore. The deprecated functions ``sph_jn``, ``sph_yn``, ``sph_jnyn``, ``sph_in``, ``sph_kn``, and ``sph_inkn`` have been removed. Users should instead use the functions ``spherical_jn``, ``spherical_yn``, ``spherical_in``, and ``spherical_kn``. Be aware that the new functions have different signatures. The cross-class properties of `scipy.signal.lti` systems have been removed. The following properties/setters have been removed: Name - (accessing/setting has been removed) - (setting has been removed) * StateSpace - (``num``, ``den``, ``gain``) - (``zeros``, ``poles``) * TransferFunction (``A``, ``B``, ``C``, ``D``, ``gain``) - (``zeros``, ``poles``) * ZerosPolesGain (``A``, ``B``, ``C``, ``D``, ``num``, ``den``) - () ``signal.freqz(b, a)`` with ``b`` or ``a`` >1-D raises a ``ValueError``. This was a corner case for which it was unclear that the behavior was well-defined. The method ``var`` of `scipy.stats.dirichlet` now returns a scalar rather than an ndarray when the length of alpha is 1. Other changes ============= SciPy now has a formal governance structure. It consists of a BDFL (Pauli Virtanen) and a Steering Committee. See `the governance document < https://github.com/scipy/scipy/blob/master/doc/source/dev/governance/govern… >`_ for details. It is now possible to build SciPy on Windows with MSVC + gfortran! Continuous integration has been set up for this build configuration on Appveyor, building against OpenBLAS. Continuous integration for OS X has been set up on TravisCI. The SciPy test suite has been migrated from ``nose`` to ``pytest``. ``scipy/_distributor_init.py`` was added to allow redistributors of SciPy to add custom code that needs to run when importing SciPy (e.g. checks for hardware, DLL search paths, etc.). Support for PEP 518 (specifying build system requirements) was added - see ``pyproject.toml`` in the root of the SciPy repository. In order to have consistent function names, the function ``scipy.linalg.solve_lyapunov`` is renamed to `scipy.linalg.solve_continuous_lyapunov`. The old name is kept for backwards-compatibility. Authors ======= * @arcady + * @xoviat + * Anton Akhmerov * Dominic Antonacci + * Alessandro Pietro Bardelli * Ved Basu + * Michael James Bedford + * Ray Bell + * Juan M. Bello-Rivas + * Sebastian Berg * Felix Berkenkamp * Jyotirmoy Bhattacharya + * Matthew Brett * Jonathan Bright * Bruno Jiménez + * Evgeni Burovski * Patrick Callier * Mark Campanelli + * CJ Carey * Robert Cimrman * Adam Cox + * Michael Danilov + * David Haberthür + * Andras Deak + * Philip DeBoer * Anne-Sylvie Deutsch * Cathy Douglass + * Dominic Else + * Guo Fei + * Roman Feldbauer + * Yu Feng * Jaime Fernandez del Rio * Orestis Floros + * David Freese + * Adam Geitgey + * James Gerity + * Dezmond Goff + * Christoph Gohlke * Ralf Gommers * Dirk Gorissen + * Matt Haberland + * David Hagen + * Charles Harris * Lam Yuen Hei + * Jean Helie + * Gaute Hope + * Guillaume Horel + * Franziska Horn + * Yevhenii Hyzyla + * Vladislav Iakovlev + * Marvin Kastner + * Mher Kazandjian * Thomas Keck * Adam Kurkiewicz + * Ronan Lamy + * J.L. Lanfranchi + * Eric Larson * Denis Laxalde * Gregory R. Lee * Felix Lenders + * Evan Limanto * Julian Lukwata + * François Magimel * Syrtis Major + * Charles Masson + * Nikolay Mayorov * Tobias Megies * Markus Meister + * Roman Mirochnik + * Jordi Montes + * Nathan Musoke + * Andrew Nelson * M.J. Nichol * Juan Nunez-Iglesias * Arno Onken + * Nick Papior + * Dima Pasechnik + * Ashwin Pathak + * Oleksandr Pavlyk + * Stefan Peterson * Ilhan Polat * Andrey Portnoy + * Ravi Kumar Prasad + * Aman Pratik * Eric Quintero * Vedant Rathore + * Tyler Reddy * Joscha Reimer * Philipp Rentzsch + * Antonio Horta Ribeiro * Ned Richards + * Kevin Rose + * Benoit Rostykus + * Matt Ruffalo + * Eli Sadoff + * Pim Schellart * Nico Schlömer + * Klaus Sembritzki + * Nikolay Shebanov + * Jonathan Tammo Siebert * Scott Sievert * Max Silbiger + * Mandeep Singh + * Michael Stewart + * Jonathan Sutton + * Deep Tavker + * Martin Thoma * James Tocknell + * Aleksandar Trifunovic + * Paul van Mulbregt + * Jacob Vanderplas * Aditya Vijaykumar * Pauli Virtanen * James Webber * Warren Weckesser * Eric Wieser + * Josh Wilson * Zhiqing Xiao + * Evgeny Zhurko * Nikolay Zinov + * Zé Vinícius + A total of 121 people contributed to this release. People with a "+" by their names contributed a patch for the first time. This list of names is automatically generated, and may not be fully complete. Cheers, Ralf

4 3

establishing a Code of Conduct for SciPy
by Ralf Gommers 24 Oct '17

24 Oct '17

Hi all, I propose that we as SciPy developers and community adopt a Code of Conduct. As you probably know, Code of Conduct (CoC) documents are becoming more common every year for open source projects, and there are a number of good reasons to adopt a CoC: 1. It gives us the opportunity to explicitly express the values and behaviors we'd like to see in our community. 2. It is designed to make everyone feel welcome (and while I think we're a welcoming community anyway, not having a CoC may look explicitly unwelcoming to some potential contributors nowadays). 3. It gives us a tool to address a set of problems if and when they occur, as well as a way for anyone to report issues or behavior that is unacceptable to them (much better than having those people potentially leave the community). 4. SciPy is not yet a fiscally sponsored project of NumFOCUS, however I think we'd like to be in the near future. NumFOCUS has started to require having a CoC as a prerequisite for new projects joining it. The PSF has the same requirement for any sponsorship for events/projects that it gives. Also note that GitHub has starting checking the presence of a CoC fairly prominently (https://github.com/scipy/scipy/community), and has also produced a guide with things to think about when formulating a CoC: https://opensource.guide/code-of-conduct/. I recommend reading that guide (as well as others guides on that site), it's really good. To get to a CoC document, a good approach is to borrow text from a CoC that has been in use for a while and has proven to be valuable, and then modify where needed (similar to a software license - don't invent your own). I considered three existing CoC's: - The Contributor Covenant (http://contributor-covenant.org/version/1/2/0/): simple, concise, the most widely used one. The NumFOCUS recommended one is based on it as well (https://www.numfocus.org/about/code-of-conduct/). - The Python Community Code of Conduct ( https://www.python.org/psf/codeofconduct/): also simple, addresses mostly the spirit in which the Python community is operating / should operate. - The Jupyter Code of Conduct ( https://github.com/jupyter/governance/tree/master/conduct): much more detailed, in part derived from the Speak up! and Django ones, more appropriate for large communities. I think the Python Community CoC isn't a good basis, it's more a statement of intent for a wider community while it's missing too many elements that a project CoC should have. That leaves the choice between the short and simple Contributor Covenant, and the more extensive Jupyter one. Personally I like the tone of the Jupyter one *much* more, so I have started from that one and made the following modifications to make it fit SciPy better: 1. Removed the part about reporting during events (we don't organize those) and removed the language about events from the faq. 2. Changes to reporting options: removed the form (email should be enough), and added reporting to NumFOCUS as a second option. 3. Changes to enforcement manual: reply within 72 hours rather than 24 hours (we don't have paid work on SciPy, so 24 hours is not very realistic). Changed the committee from 5 to 3 members (5 is a lot, and we're significantly smaller than Jupyter/Django). Here is a WIP PR with the CoC content: https://github.com/scipy/scipy/pull/7764. I suggest to bring up any larger questions/issues here, and detailed textual comments on the PR. Once the content is agreed upon I will change it to reST and integrate it with the rest of the docs. Thoughts? Volunteers for the committee? Cheers, Ralf

12 46

Linear extrapolation in univariate spline
by Andrea Arteaga 23 Oct '17

23 Oct '17

Dear SciPy devs, The UnivariateSpline class [1] allows to pass a parameter `ext` to control the behavior of the interpolant outside of the input data range, i.e, for the cases where an extrapolation is required. The four available modes are to perform actual k-degree extrapolation, to return 0, to do constant extrapolation or to fail. These modes are insufficient for my needs. What I would actually need is a linear extrapolation. By definition of the "natural" cubic spline [2], the second derivative will be zero at both ends of the domain. This has the physical meaning of modeling exactly the shape of a ruler bent in such a way to touch a set of points. Before the first point and after the last one, the rules will be straight. This is the kind of extrapolation I would need. I would like to ask if that feature has not been implemented for a good reason, if there are any plans on implementing it and if I can do something. I started to look at the code to see if I could easily implement this mode. In any case I will need some guidance with it. Another thing I should ask is whether there is already a way to specify custom extrapolation modes (e.g. by passing a python function). I think I found the place in the SciPy code where the extrapolation modes are handled, in the `splev.f` file [3]. It does not look too difficult to add a `e .eq. 4` case, which handles linear extrapolation. But I don't know how to access the slopes at both ends of the domain. Thanks and best Andrea Arteaga [1] https://docs.scipy.org/doc/scipy/reference/generated/scipy.interpolate.Univ… [2] https://en.wikipedia.org/wiki/Spline_interpolation#Example [3] https://github.com/scipy/scipy/blob/master/scipy/interpolate/fitpack/splev.…

1 0

ANN: second SciPy 1.0.0 release candidate
by Ralf Gommers 18 Oct '17

18 Oct '17

Hi all, I'm excited to be able to announce the availability of the second (and hopefully last) release candidate of Scipy 1.0. This is a big release, and a version number that has been 16 years in the making. It contains a few more deprecations and backwards incompatible changes than an average release. Therefore please do test it on your own code, and report any issues on the Github issue tracker or on the scipy-dev mailing list. Sources and binary wheels can be found at https://pypi.python.org/pypi/scipy and https://github.com/scipy/scipy/releases/tag/v1.0.0rc2. To install with pip: pip install --pre --upgrade scipy The most important issues fixed after v1.0.0rc1 is https://github.com/scipy/scipy/issues/7969 (missing DLL in Windows wheel). Pull requests merged after v1.0.0rc1: - `#7948 <https://github.com/scipy/scipy/pull/7948>`__: DOC: add note on checking for deprecations before upgrade to... - `#7952 <https://github.com/scipy/scipy/pull/7952>`__: DOC: update SciPy Roadmap for 1.0 release and recent discussions. - `#7960 <https://github.com/scipy/scipy/pull/7960>`__: BUG: optimize: revert changes to bfgs in gh-7165 - `#7962 <https://github.com/scipy/scipy/pull/7962>`__: TST: special: mark a failing hyp2f1 test as xfail - `#7973 <https://github.com/scipy/scipy/pull/7973>`__: BUG: fixed keyword in 'info' in ``_get_mem_available`` utility - `#7986 <https://github.com/scipy/scipy/pull/7986>`__: TST: Relax test_trsm precision to 5 decimals - `#8001 <https://github.com/scipy/scipy/pull/8001>`__: TST: fix test failures from Matplotlib 2.1 update - `#8010 <https://github.com/scipy/scipy/pull/8010>`__: BUG: signal: fix crash in lfilter - `#8019 <https://github.com/scipy/scipy/pull/8019>`__: MAINT: fix test failures with NumPy master Thanks to everyone who contributed to this release! Ralf ========================== SciPy 1.0.0 Release Notes ========================== .. note:: Scipy 1.0.0 is not released yet! .. contents:: SciPy 1.0.0 is the culmination of 8 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 1.0.x branch, and on adding new features on the master branch. Some of the highlights of this release are: - Major build improvements. Windows wheels are available on PyPI for the first time, and continuous integration has been set up on Windows and OS X in addition to Linux. - A set of new ODE solvers and a unified interface to them (`scipy.integrate.solve_ivp`). - Two new trust region optimizers and a new linear programming method, with improved performance compared to what `scipy.optimize` offered previously. - Many new BLAS and LAPACK functions were wrapped. The BLAS wrappers are now complete. This release requires Python 2.7 or 3.4+ and NumPy 1.8.2 or greater. This is also the last release to support LAPACK 3.1.x - 3.3.x. Moving the lowest supported LAPACK version to >3.2.x was long blocked by Apple Accelerate providing the LAPACK 3.2.1 API. We have decided that it's time to either drop Accelerate or, if there is enough interest, provide shims for functions added in more recent LAPACK versions so it can still be used. New features ============ `scipy.cluster` improvements ---------------------------- `scipy.cluster.hierarchy.optimal_leaf_ordering`, a function to reorder a linkage matrix to minimize distances between adjacent leaves, was added. `scipy.fftpack` improvements ---------------------------- N-dimensional versions of the discrete sine and cosine transforms and their inverses were added as ``dctn``, ``idctn``, ``dstn`` and ``idstn``. `scipy.integrate` improvements ------------------------------ A set of new ODE solvers have been added to `scipy.integrate`. The convenience function `scipy.integrate.solve_ivp` allows uniform access to all solvers. The individual solvers (``RK23``, ``RK45``, ``Radau``, ``BDF`` and ``LSODA``) can also be used directly. `scipy.linalg` improvements ---------------------------- The BLAS wrappers in `scipy.linalg.blas` have been completed. Added functions are ``*gbmv``, ``*hbmv``, ``*hpmv``, ``*hpr``, ``*hpr2``, ``*spmv``, ``*spr``, ``*tbmv``, ``*tbsv``, ``*tpmv``, ``*tpsv``, ``*trsm``, ``*trsv``, ``*sbmv``, ``*spr2``, Wrappers for the LAPACK functions ``*gels``, ``*stev``, ``*sytrd``, ``*hetrd``, ``*sytf2``, ``*hetrf``, ``*sytrf``, ``*sycon``, ``*hecon``, ``*gglse``, ``*stebz``, ``*stemr``, ``*sterf``, and ``*stein`` have been added. The function `scipy.linalg.subspace_angles` has been added to compute the subspace angles between two matrices. The function `scipy.linalg.clarkson_woodruff_transform` has been added. It finds low-rank matrix approximation via the Clarkson-Woodruff Transform. The functions `scipy.linalg.eigh_tridiagonal` and `scipy.linalg.eigvalsh_tridiagonal`, which find the eigenvalues and eigenvectors of tridiagonal hermitian/symmetric matrices, were added. `scipy.ndimage` improvements ---------------------------- Support for homogeneous coordinate transforms has been added to `scipy.ndimage.affine_transform`. The ``ndimage`` C code underwent a significant refactoring, and is now a lot easier to understand and maintain. `scipy.optimize` improvements ----------------------------- The methods ``trust-region-exact`` and ``trust-krylov`` have been added to the function `scipy.optimize.minimize`. These new trust-region methods solve the subproblem with higher accuracy at the cost of more Hessian factorizations (compared to dogleg) or more matrix vector products (compared to ncg) but usually require less nonlinear iterations and are able to deal with indefinite Hessians. They seem very competitive against the other Newton methods implemented in scipy. `scipy.optimize.linprog` gained an interior point method. Its performance is superior (both in accuracy and speed) to the older simplex method. `scipy.signal` improvements --------------------------- An argument ``fs`` (sampling frequency) was added to the following functions: ``firwin``, ``firwin2``, ``firls``, and ``remez``. This makes these functions consistent with many other functions in `scipy.signal` in which the sampling frequency can be specified. `scipy.signal.freqz` has been sped up significantly for FIR filters. `scipy.sparse` improvements --------------------------- Iterating over and slicing of CSC and CSR matrices is now faster by up to ~35%. The ``tocsr`` method of COO matrices is now several times faster. The ``diagonal`` method of sparse matrices now takes a parameter, indicating which diagonal to return. `scipy.sparse.linalg` improvements ---------------------------------- A new iterative solver for large-scale nonsymmetric sparse linear systems, `scipy.sparse.linalg.gcrotmk`, was added. It implements ``GCROT(m,k)``, a flexible variant of ``GCROT``. `scipy.sparse.linalg.lsmr` now accepts an initial guess, yielding potentially faster convergence. SuperLU was updated to version 5.2.1. `scipy.spatial` improvements ---------------------------- Many distance metrics in `scipy.spatial.distance` gained support for weights. The signatures of `scipy.spatial.distance.pdist` and `scipy.spatial.distance.cdist` were changed to ``*args, **kwargs`` in order to support a wider range of metrics (e.g. string-based metrics that need extra keywords). Also, an optional ``out`` parameter was added to ``pdist`` and ``cdist`` allowing the user to specify where the resulting distance matrix is to be stored `scipy.stats` improvements -------------------------- The methods ``cdf`` and ``logcdf`` were added to `scipy.stats.multivariate_normal`, providing the cumulative distribution function of the multivariate normal distribution. New statistical distance functions were added, namely `scipy.stats.wasserstein_distance` for the first Wasserstein distance and `scipy.stats.energy_distance` for the energy distance. Deprecated features =================== The following functions in `scipy.misc` are deprecated: ``bytescale``, ``fromimage``, ``imfilter``, ``imread``, ``imresize``, ``imrotate``, ``imsave``, ``imshow`` and ``toimage``. Most of those functions have unexpected behavior (like rescaling and type casting image data without the user asking for that). Other functions simply have better alternatives. ``scipy.interpolate.interpolate_wrapper`` and all functions in that submodule are deprecated. This was a never finished set of wrapper functions which is not relevant anymore. The ``fillvalue`` of `scipy.signal.convolve2d` will be cast directly to the dtypes of the input arrays in the future and checked that it is a scalar or an array with a single element. ``scipy.spatial.distance.matching`` is deprecated. It is an alias of `scipy.spatial.distance.hamming`, which should be used instead. Implementation of `scipy.spatial.distance.wminkowski` was based on a wrong interpretation of the metric definition. In scipy 1.0 it has been just deprecated in the documentation to keep retro-compatibility but is recommended to use the new version of `scipy.spatial.distance.minkowski` that implements the correct behaviour. Positional arguments of `scipy.spatial.distance.pdist` and `scipy.spatial.distance.cdist` should be replaced with their keyword version. Backwards incompatible changes ============================== The following deprecated functions have been removed from `scipy.stats`: ``betai``, ``chisqprob``, ``f_value``, ``histogram``, ``histogram2``, ``pdf_fromgamma``, ``signaltonoise``, ``square_of_sums``, ``ss`` and ``threshold``. The following deprecated functions have been removed from `scipy.stats.mstats`: ``betai``, ``f_value_wilks_lambda``, ``signaltonoise`` and ``threshold``. The deprecated ``a`` and ``reta`` keywords have been removed from `scipy.stats.shapiro`. The deprecated functions ``sparse.csgraph.cs_graph_components`` and ``sparse.linalg.symeig`` have been removed from `scipy.sparse`. The following deprecated keywords have been removed in `scipy.sparse.linalg`: ``drop_tol`` from ``splu``, and ``xtype`` from ``bicg``, ``bicgstab``, ``cg``, ``cgs``, ``gmres``, ``qmr`` and ``minres``. The deprecated functions ``expm2`` and ``expm3`` have been removed from `scipy.linalg`. The deprecated keyword ``q`` was removed from `scipy.linalg.expm`. And the deprecated submodule ``linalg.calc_lwork`` was removed. The deprecated functions ``C2K``, ``K2C``, ``F2C``, ``C2F``, ``F2K`` and ``K2F`` have been removed from `scipy.constants`. The deprecated ``ppform`` class was removed from `scipy.interpolate`. The deprecated keyword ``iprint`` was removed from `scipy.optimize.fmin_cobyla`. The default value for the ``zero_phase`` keyword of `scipy.signal.decimate` has been changed to True. The ``kmeans`` and ``kmeans2`` functions in `scipy.cluster.vq` changed the method used for random initialization, so using a fixed random seed will not necessarily produce the same results as in previous versions. `scipy.special.gammaln` does not accept complex arguments anymore. The deprecated functions ``sph_jn``, ``sph_yn``, ``sph_jnyn``, ``sph_in``, ``sph_kn``, and ``sph_inkn`` have been removed. Users should instead use the functions ``spherical_jn``, ``spherical_yn``, ``spherical_in``, and ``spherical_kn``. Be aware that the new functions have different signatures. The cross-class properties of `scipy.signal.lti` systems have been removed. The following properties/setters have been removed: Name - (accessing/setting has been removed) - (setting has been removed) * StateSpace - (``num``, ``den``, ``gain``) - (``zeros``, ``poles``) * TransferFunction (``A``, ``B``, ``C``, ``D``, ``gain``) - (``zeros``, ``poles``) * ZerosPolesGain (``A``, ``B``, ``C``, ``D``, ``num``, ``den``) - () ``signal.freqz(b, a)`` with ``b`` or ``a`` >1-D raises a ``ValueError``. This was a corner case for which it was unclear that the behavior was well-defined. The method ``var`` of `scipy.stats.dirichlet` now returns a scalar rather than an ndarray when the length of alpha is 1. Other changes ============= SciPy now has a formal governance structure. It consists of a BDFL (Pauli Virtanen) and a Steering Committee. See `the governance document <https://github.com/scipy/scipy/blob/master/doc/source/ dev/governance/governance.rst>`_ for details. It is now possible to build SciPy on Windows with MSVC + gfortran! Continuous integration has been set up for this build configuration on Appveyor, building against OpenBLAS. Continuous integration for OS X has been set up on TravisCI. The SciPy test suite has been migrated from ``nose`` to ``pytest``. ``scipy/_distributor_init.py`` was added to allow redistributors of SciPy to add custom code that needs to run when importing SciPy (e.g. checks for hardware, DLL search paths, etc.). Support for PEP 518 (specifying build system requirements) was added - see ``pyproject.toml`` in the root of the SciPy repository. In order to have consistent function names, the function ``scipy.linalg.solve_lyapunov`` is renamed to `scipy.linalg.solve_continuous_lyapunov`. The old name is kept for backwards-compatibility.

2 1

scipy.signal.residue function - extension to z, p, k input
by Leonhard Neuhaus 13 Oct '17

13 Oct '17

Hi all, I have written a function residues() which does the same as scipy.signal.residue, but that accepts an input in the form (zeros, poles, k) - k is the global prefactor - instead of the usual (b, a)-Polynomial form of the transfer function to transform. This was necessary since I wanted to do the partial fraction expansion of an transfer function of roughly 10th order (given in the zero-pole-form) and numerical imprecision issues prevented me from passing through the (b, a)-form. The partial fraction expansion was necessary in my case to decompose the transfer function into parallel biquads (opposed to the more common serial biquad decomposition). The way I implemented the transformation can be found here: https://github.com/lneuhaus/pyrpl/blob/55cb112a50e221f11eb28f106ba11272890c… This implementation does not suffer from the typical numerical imprecision issues associated with higher-order polynomials since the coefficients (i.e. the prefactors of x^n in the numerator/denominator of the transfer function) are never explicitly calculated. I would be happy to prepare a proper pull-request that extends the residue function with the (zero, pole, k)-input option if this is desired. Please let me know and I will get to work. Cheers, Leo

2 1

release schedule update
by Ralf Gommers 13 Oct '17

13 Oct '17

Hi all, Here's an update on the release schedule for scipy 1.0. We will need another release candidate, and are falling a little behind - here's my plan for the new dates: Sep 16: Branch 1.0.x --> done Sep 17: beta1 --> done Sep 27: rc1 --> done Oct 7: rc2 --> move to Oct 17 Oct 17: final release --> move to Oct 24 Here is the list of things that have to go into rc2: 1. https://github.com/scipy/scipy/pulls?q=is%3Apr+label%3Abackport-candidate+i… 2. Ensure we pick up the fix for https://github.com/scipy/scipy/issues/7969 to get the Windows wheels right. There are no unsolved blocking issues; the only reason for Oct 17 for rc2 is that I'm traveling now and didn't/don't have the bandwidth to do it this week. Cheers, Ralf

2 1