Mailman 3 September 2017 - SciPy-Dev

Re: [SciPy-Dev] Add feature to approx_fprime (Nicolas CELLIER)
by Jeffrey Manville Sept. 24, 2017

Sept. 24, 2017

Hello Nick, So, I am implementing Andrew Ng's Machine learning coursework in Python which was originally in Matlab/Octave. I was using the approx_fprime to evaluate by implementation of a back propagation algorithm. To be honest, I didn't know about automatic differentiation. I suspect that it can be prohibitively expensive for many machine learning algorithms, but probably not for my project. I could use a smaller epsilon, but then I am just getting closer and closer to the floating point error problem. *My project:* In my project, the "difference" between backprop and the numerical gradient was expected to be <1e-9 with an epsilon of 1e-4. I was getting about 2.48e-5 with the current approx_fprime vs 1.65e-11 with the two sided method and the same epsilon. The two sided was significantly more accurate. *The problem:* The problem with the one sided method is that it is inherently biased. The change isn't centered around the point in question for the one sided method. The two methods are essentially the same method with the difference being "where" they guess that calculated derivative is. One sided guesses that the calculated derivative is at one side of the two points. The two sided guesses the derivative is in the middle. *Proof:* I attached an excel doc that has a kind of example of the difference. I have the project on github. Here is the branch <https://github.com/manvillej/Python-AndrewNgML/tree/Ex4NNtraining> that I am currently working on. If you pull it and run ex4checker.py you will see the two sided. If you change line 3: import checker as op to import checker as op you will see the relative difference change from the two sided to the one sided. Cheers, Jeff On Sun, Sep 24, 2017 at 12:00 PM, <scipy-dev-request(a)python.org> wrote: > Send SciPy-Dev mailing list submissions to > scipy-dev(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request(a)python.org > > You can reach the person managing the list at > scipy-dev-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. Add feature to approx_fprime (Jeffrey Manville) > 2. Re: Add feature to approx_fprime (Nicolas CELLIER) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 23 Sep 2017 23:05:57 -0400 > From: Jeffrey Manville <jeffmanvillejr(a)gmail.com> > To: scipy-dev(a)python.org > Subject: [SciPy-Dev] Add feature to approx_fprime > Message-ID: > <CAFUATRLXKY6y=5VEANjrd2ATtZ8GHKxWEeLmZ=Tk6RkfxABEvw@mail. > gmail.com> > Content-Type: text/plain; charset="utf-8" > > Hello, > > I have been using the approx_fprime and I noticed that the source code as > it is only uses the one sided numerical differentiation. I had to build my > own two sided one by copying and modifying the one sided code. > > One sided: F'(x) = ( F(x+h) - F(x) ) / h aka Newton > method > Two sided: F'(x) = ( F(x+h) - F(x-h) ) / (2*h) aka > Symmetric method > > The two sided method tends to be more accurate, but can be more > computationally expensive. > > Is this something that would be good to add? I am newer to python and I > haven't contributed to open source yet, but it seems like a good fit. > > I think it would be good to pass in a kwarg called method and set the > default to one sided, so no one's code breaks, but they can choose which > method to use. > > Should I keep the decision logic at approx_fprime and have two separate > functions for the two methods? > > Here's a wikipedia article on it > <https://en.wikipedia.org/wiki/Numerical_differentiation> > Here's a link to the source code > <https://github.com/scipy/scipy/blob/v0.19.1/scipy/optimize/ > optimize.py#L633-L688> > > Cheers, > > Jeff >

2 1

welcome Ilhan to the core team
by Ralf Gommers Sept. 22, 2017

Sept. 22, 2017

Hi all, On behalf of the SciPy developers I'd like to welcome Ilhan Polat as a member of the core dev team. Ilhan has been contributing to scipy.linalg for a year now, and is responsible for a lot of the new BLAS and LAPACK wrappers as well as improvements to various solvers - https://github.com/scipy/scipy/pulls/ilayn I'm looking forward to his continued contributions! Cheers, Ralf

2 1

Enhancements to scipy.interpolate.RegularGridInterpolator
by Tristan H Sept. 22, 2017

Sept. 22, 2017

Hi, long time user, first time (potential) contributor. I recently developed some techniques for interpolation on regular grids that includes smooth spline interpolation in arbitrary dimensions. This also includes calculation of gradients with respect to the interpolations. This method is basically set of interpolations that acts like a separable tensor transformation: I apply instances of `UnivariateSpline` to the last dimension, and "fold" the values into the next dimension, and continue until a final value (and its gradient) is reached. This is useful to my work for various surrogate modelling purposes, but is general enough that it could be useful to others. To date, I found no existing n-dimensional interpolation library for structured data that includes gradients, so there is at least some novelty. I've actually gone ahead with a potential integration into `RegularGridInterpolator` as an additional set of 'method' options with tests, a rough cut at documentation, and a light refactor in `scipy.interpolate` to place `RegularGridInterpolator` and `interpn` into their own source files and test files. The refactor was originally done to get around a circular import issue based on a requirement on PPoly, though it is not strictly necessary any longer. I'd be happy to talk about the idea more or hear any opinions, or could open a PR if there is interest in seeing the code. Best, -Tristan

2 1

ANN: SfePy 2017.3
by Robert Cimrman Sept. 19, 2017

Sept. 19, 2017

I am pleased to announce release 2017.3 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method or by the isogeometric analysis (limited support). It is distributed under the new BSD license. Home page: http://sfepy.org Mailing list: https://mail.python.org/mm3/mailman3/lists/sfepy.python.org/ Git (source) repository, issue tracker: https://github.com/sfepy/sfepy Highlights of this release -------------------------- - support preconditioning in SciPy and PyAMG based linear solvers - user-defined preconditioners for PETSc linear solvers - parallel multiscale (macro-micro) homogenization-based computations - improved tutorial and installation instructions For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1 (rather long and technical). Cheers, Robert Cimrman --- Contributors to this release in alphabetical order: Robert Cimrman Lubos Kejzlar Vladimir Lukes Matyas Novak

1 0

Constrained Optimization with Scipy
by Masoud Mansouryar Sept. 19, 2017

Sept. 19, 2017

Hi Friends, I am working on an Optimization problem in Python, which is defined like this: def objective(x, sign=1.0): P_max = x[0] P_min = x[1] s_max = x[2] s_min = x[3] s_data, P_f, P_g = PG(a,b,P_max,P_min,P_f_max,s_max,s_min) A_tot = AM(s_data,c,d) a_cost_opt = a_costFunc(A_tot) g_cost_opt = g_costFunc(P_f,P_g) res_max = -(a_cost_opt + g_cost_opt) print((a_cost_opt + g_cost_opt)) return res_max x0 = (2.5,2.5,.8,.2) # Starting points bd0 = (1,5) bd1 = (1,5) bd2 = (0.7,0.9) bd3 = (0.1,0.3) bnds = (bd0,bd1,bd2,bd3) # bounds minimizer_kwargs = dict(method="L-BFGS-B", bounds=bnds) sol = sci.optimize.basinhopping(objective, x0, minimizer_kwargs=minimizer_kwargs, niter=2, niter_success=5, T=0.5, stepsize=.1, disp=True) As you see in my code, due to the complexity of my objective function, I am using basinhopping algorithm and so far it is giving me a good result. That negative sign in my res_max is to find the maximum indeed (maximum interest), so I thought a negative minimum is a maximum. The functions which are being called inside my objective function, have non-linear calculations, and all of their parameters are numpy array. But there are some points unclear for me about this optimization: 1. Is my objective function convex, differentiable, non-linear, discrete? 2. How can I find that my result is a global minimum, not a local minimum? (I want the global minimum) 3. Which algorithm is actually minimizing my function, basinhopping or L-BFGS-B? 4. Is scipy.optimize.basinhopping the best framework which I need? Your help is really appreciated. Thank you, Masoud

1 0

Re: [SciPy-Dev] [SciPy-User] ANN: SciPy 1.0 beta release
by Ralf Gommers Sept. 19, 2017

Sept. 19, 2017

On Mon, Sep 18, 2017 at 10:36 PM, Matthew Brett <matthew.brett(a)gmail.com> wrote: > Hi, > > On Mon, Sep 18, 2017 at 11:14 AM, Ralf Gommers <ralf.gommers(a)gmail.com> > wrote: > > > > > > On Mon, Sep 18, 2017 at 10:11 PM, Matthew Brett <matthew.brett(a)gmail.com > > > > wrote: > >> > >> Hi, > >> > >> On Mon, Sep 18, 2017 at 11:07 AM, Thomas Kluyver <takowl(a)gmail.com> > wrote: > >> > On 18 September 2017 at 10:59, Ralf Gommers <ralf.gommers(a)gmail.com> > >> > wrote: > >> >> > >> >> Binary wheels for Windows, Linux and OS X (for all supported Python > >> >> versions, 32-bit and 64-bit) can be found at http://wheels.scipy.org > . > >> >> To > >> >> install directly with pip: > >> >> > >> >> pip install scipy=='1.0.0b1' -f http://wheels.scipy.org > >> >> --trusted-host > >> >> wheels.scipy.org > >> > > >> > > >> > I don't want to criticise the hard work that has gone into making this > >> > available, but I'm disappointed that we're telling people to install > >> > software over an insecure HTTP connection. > >> > >> I personally prefer the following recipe: > >> > >> pip install -f > >> https://3f23b170c54c2533c070-1c8a9b3114517dc5fe17b7c3f8c63a4 > 3.ssl.cf2.rackcdn.com > >> scipy=='1.0.0b1' > >> > >> > Can the wheels not be uploaded to PyPI? > >> > >> Sounds like a good idea. I can do that - any objections? > > > > > > That would be helpful Matthew, I'm about to sign off for today. > > Done - new instructions for testing: > > pip install --pre --upgrade scipy > Thanks Matthew! Replying to all lists with the better install instructions. Cheers, Ralf

1 0

ANN: SciPy 1.0 beta release
by Ralf Gommers Sept. 18, 2017

Sept. 18, 2017

Hi all, I'm excited to be able to announce the availability of the first beta release of Scipy 1.0. This is a big release, and a version number that has been 16 years in the making. It contains a few more deprecations and backwards incompatible changes than an average release. Therefore please do test it on your own code, and report any issues on the Github issue tracker or on the scipy-dev mailing list. Sources: https://github.com/scipy/scipy/releases/tag/v1.0.0b1 Binary wheels: will follow tomorrow, I'll announce those when ready (TravisCI is under maintenance right now) Thanks to everyone who contributed to this release! Ralf Release notes (full notes including authors, closed issued and merged PRs at the Github Releases link above): ========================== SciPy 1.0.0 Release Notes ========================== .. note:: Scipy 1.0.0 is not released yet! .. contents:: SciPy 1.0.0 is the culmination of 8 months of hard work. It contains many new features, numerous bug-fixes, improved test coverage and better documentation. There have been a number of deprecations and API changes in this release, which are documented below. All users are encouraged to upgrade to this release, as there are a large number of bug-fixes and optimizations. Moreover, our development attention will now shift to bug-fix releases on the 1.0.x branch, and on adding new features on the master branch. Some of the highlights of this release are: - Major build improvements. Windows wheels are available on PyPI for the first time, and continuous integration has been set up on Windows and OS X in addition to Linux. - A set of new ODE solvers and a unified interface to them (`scipy.integrate.solve_ivp`). - Two new trust region optimizers and a new linear programming method, with improved performance compared to what `scipy.optimize` offered previously. - Many new BLAS and LAPACK functions were wrapped. The BLAS wrappers are now complete. This release requires Python 2.7 or 3.4+ and NumPy 1.8.2 or greater. This is also the last release to support LAPACK 3.1.x - 3.3.x. Moving the lowest supported LAPACK version to >3.2.x was long blocked by Apple Accelerate providing the LAPACK 3.2.1 API. We have decided that it's time to either drop Accelerate or, if there is enough interest, provide shims for functions added in more recent LAPACK versions so it can still be used. New features ============ `scipy.cluster` improvements ---------------------------- `scipy.cluster.hierarchy.optimal_leaf_ordering`, a function to reorder a linkage matrix to minimize distances between adjacent leaves, was added. `scipy.fftpack` improvements ---------------------------- N-dimensional versions of the discrete sine and cosine transforms and their inverses were added as ``dctn``, ``idctn``, ``dstn`` and ``idstn``. `scipy.integrate` improvements ------------------------------ A set of new ODE solvers have been added to `scipy.integrate`. The convenience function `scipy.integrate.solve_ivp` allows uniform access to all solvers. The individual solvers (``RK23``, ``RK45``, ``Radau``, ``BDF`` and ``LSODA``) can also be used directly. `scipy.linalg` improvements ---------------------------- The BLAS wrappers in `scipy.linalg.blas` have been completed. Added functions are ``*gbmv``, ``*hbmv``, ``*hpmv``, ``*hpr``, ``*hpr2``, ``*spmv``, ``*spr``, ``*tbmv``, ``*tbsv``, ``*tpmv``, ``*tpsv``, ``*trsm``, ``*trsv``, ``*sbmv``, ``*spr2``, Wrappers for the LAPACK functions ``*gels``, ``*stev``, ``*sytrd``, ``*hetrd``, ``*sytf2``, ``*hetrf``, ``*sytrf``, ``*sycon``, ``*hecon``, ``*gglse``, ``*stebz``, ``*stemr``, ``*sterf``, and ``*stein`` have been added. The function `scipy.linalg.subspace_angles` has been added to compute the subspace angles between two matrices. The function `scipy.linalg.clarkson_woodruff_transform` has been added. It finds low-rank matrix approximation via the Clarkson-Woodruff Transform. The functions `scipy.linalg.eigh_tridiagonal` and `scipy.linalg.eigvalsh_tridiagonal`, which find the eigenvalues and eigenvectors of tridiagonal hermitian/symmetric matrices, were added. `scipy.ndimage` improvements ---------------------------- Support for homogeneous coordinate transforms has been added to `scipy.ndimage.affine_transform`. The ``ndimage`` C code underwent a significant refactoring, and is now a lot easier to understand and maintain. `scipy.optimize` improvements ----------------------------- The methods ``trust-region-exact`` and ``trust-krylov`` have been added to the function `scipy.optimize.minimize`. These new trust-region methods solve the subproblem with higher accuracy at the cost of more Hessian factorizations (compared to dogleg) or more matrix vector products (compared to ncg) but usually require less nonlinear iterations and are able to deal with indefinite Hessians. They seem very competitive against the other Newton methods implemented in scipy. `scipy.optimize.linprog` gained an interior point method. Its performance is superior (both in accuracy and speed) to the older simplex method. `scipy.signal` improvements --------------------------- An argument ``fs`` (sampling frequency) was added to the following functions: ``firwin``, ``firwin2``, ``firls``, and ``remez``. This makes these functions consistent with many other functions in `scipy.signal` in which the sampling frequency can be specified. `scipy.signal.freqz` has been sped up significantly for FIR filters. `scipy.sparse` improvements --------------------------- Iterating over and slicing of CSC and CSR matrices is now faster by up to ~35%. The ``tocsr`` method of COO matrices is now several times faster. The ``diagonal`` method of sparse matrices now takes a parameter, indicating which diagonal to return. `scipy.sparse.linalg` improvements ---------------------------------- A new iterative solver for large-scale nonsymmetric sparse linear systems, `scipy.sparse.linalg.gcrotmk`, was added. It implements ``GCROT(m,k)``, a flexible variant of ``GCROT``. `scipy.sparse.linalg.lsmr` now accepts an initial guess, yielding potentially faster convergence. SuperLU was updated to version 5.2.1. `scipy.spatial` improvements ---------------------------- Many distance metrics in `scipy.spatial.distance` gained support for weights. The signatures of `scipy.spatial.distance.pdist` and `scipy.spatial.distance.cdist` were changed to ``*args, **kwargs`` in order to support a wider range of metrics (e.g. string-based metrics that need extra keywords). Also, an optional ``out`` parameter was added to ``pdist`` and ``cdist`` allowing the user to specify where the resulting distance matrix is to be stored `scipy.stats` improvements -------------------------- The methods ``cdf`` and ``logcdf`` were added to `scipy.stats.multivariate_normal`, providing the cumulative distribution function of the multivariate normal distribution. New statistical distance functions were added, namely `scipy.stats.wasserstein_distance` for the first Wasserstein distance and `scipy.stats.energy_distance` for the energy distance. Deprecated features =================== The following functions in `scipy.misc` are deprecated: ``bytescale``, ``fromimage``, ``imfilter``, ``imread``, ``imresize``, ``imrotate``, ``imsave``, ``imshow`` and ``toimage``. Most of those functions have unexpected behavior (like rescaling and type casting image data without the user asking for that). Other functions simply have better alternatives. ``scipy.interpolate.interpolate_wrapper`` and all functions in that submodule are deprecated. This was a never finished set of wrapper functions which is not relevant anymore. The ``fillvalue`` of `scipy.signal.convolve2d` will be cast directly to the dtypes of the input arrays in the future and checked that it is a scalar or an array with a single element. Backwards incompatible changes ============================== The following deprecated functions have been removed from `scipy.stats`: ``betai``, ``chisqprob``, ``f_value``, ``histogram``, ``histogram2``, ``pdf_fromgamma``, ``signaltonoise``, ``square_of_sums``, ``ss`` and ``threshold``. The following deprecated functions have been removed from `scipy.stats.mstats`: ``betai``, ``f_value_wilks_lambda``, ``signaltonoise`` and ``threshold``. The deprecated ``a`` and ``reta`` keywords have been removed from `scipy.stats.shapiro`. The deprecated functions ``sparse.csgraph.cs_graph_components`` and ``sparse.linalg.symeig`` have been removed from `scipy.sparse`. The following deprecated keywords have been removed in `scipy.sparse.linalg`: ``drop_tol`` from ``splu``, and ``xtype`` from ``bicg``, ``bicgstab``, ``cg``, ``cgs``, ``gmres``, ``qmr`` and ``minres``. The deprecated functions ``expm2`` and ``expm3`` have been removed from `scipy.linalg`. The deprecated keyword ``q`` was removed from `scipy.linalg.expm`. And the deprecated submodule ``linalg.calc_lwork`` was removed. The deprecated functions ``C2K``, ``K2C``, ``F2C``, ``C2F``, ``F2K`` and ``K2F`` have been removed from `scipy.constants`. The deprecated ``ppform`` class was removed from `scipy.interpolate`. The deprecated keyword ``iprint`` was removed from `scipy.optimize.fmin_cobyla`. The default value for the ``zero_phase`` keyword of `scipy.signal.decimate` has been changed to True. The ``kmeans`` and ``kmeans2`` functions in `scipy.cluster.vq` changed the method used for random initialization, so using a fixed random seed will not necessarily produce the same results as in previous versions. `scipy.special.gammaln` does not accept complex arguments anymore. The deprecated functions ``sph_jn``, ``sph_yn``, ``sph_jnyn``, ``sph_in``, ``sph_kn``, and ``sph_inkn`` have been removed. Users should instead use the functions ``spherical_jn``, ``spherical_yn``, ``spherical_in``, and ``spherical_kn``. Be aware that the new functions have different signatures. The cross-class properties of `scipy.signal.lti` systems have been removed. The following properties/setters have been removed: Name - (accessing/setting has been removed) - (setting has been removed) * StateSpace - (``num``, ``den``, ``gain``) - (``zeros``, ``poles``) * TransferFunction (``A``, ``B``, ``C``, ``D``, ``gain``) - (``zeros``, ``poles``) * ZerosPolesGain (``A``, ``B``, ``C``, ``D``, ``num``, ``den``) - () ``signal.freqz(b, a)`` with ``b`` or ``a`` >1-D raises a ``ValueError``. This was a corner case for which it was unclear that the behavior was well-defined. The method ``var`` of `scipy.stats.dirichlet` now returns a scalar rather than an ndarray when the length of alpha is 1. Other changes ============= SciPy now has a formal governance structure. It consists of a BDFL (Pauli Virtanen) and a Steering Committee. See `the governance document < https://github.com/scipy/scipy/blob/master/doc/source/dev/governance/govern… >`_ for details. It is now possible to build SciPy on Windows with MSVC + gfortran! Continuous integration has been set up for this build configuration on Appveyor, building against OpenBLAS. Continuous integration for OS X has been set up on TravisCI. The SciPy test suite has been migrated from ``nose`` to ``pytest``. ``scipy/_distributor_init.py`` was added to allow redistributors of SciPy to add custom code that needs to run when importing SciPy (e.g. checks for hardware, DLL search paths, etc.). Support for PEP 518 (specifying build system requirements) was added - see ``pyproject.toml`` in the root of the SciPy repository. In order to have consistent function names, the function ``scipy.linalg.solve_lyapunov`` is renamed to `scipy.linalg.solve_continuous_lyapunov`. The old name is kept for backwards-compatibility.

2 3

1.0.x branched
by Ralf Gommers Sept. 17, 2017

Sept. 17, 2017

Hi all, FYI: a 1.0.x branch has been created, and master is open for new features for 1.1.0 Cheers, Ralf

1 0

SciPy 1.0 release schedule proposal
by Ralf Gommers Sept. 16, 2017

Sept. 16, 2017

Hi all, We're now at 5 months after the 0.19.0 release, and well behind our planned 1.0 release date [1]. So it's high time for a more concrete plan - here it is: Sep 16: Branch 1.0.x Sep 17: beta1 Sep 27: rc1 Oct 7: rc2 Oct 17: final release This gives us one month to finish and merge everything that we really would like to see in 1.0 - should be enough time for things that are already in progress. The list of things now marked for 1.0 [2] is not too long, and not all of it is critical. It would be useful if everyone could add PRs/issues that they think are critical to the 1.0 milestone. Adding yourself as an "assignee" on PRs/issues you plan to tackle would also be helpful. Besides some really nice new features, we made major strides in infrastructure (CI, testing) and project organisation for this release. Fom my perspective there are three critical things left: 1. Windows wheels. Looks like we're pretty much there, but we need to ensure that 1.0 is the first release that has Windows wheels available. 2. Adding a code of conduct. This is the last thing we imho need from an "open source project maturity" perspective (an FSA would be nice, but can wait). 3. Merging a lot of PRs. There's ~150 PRs open now, I hope in the next month we can focus on reducing that number and getting some nice improvements in that have been waiting for quite some time. Finally, we've discussed before writing a paper about SciPy to coincide with this release. I'll follow up on that separately. Thoughts? Cheers, Ralf [1] https://mail.python.org/pipermail/scipy-dev/2016-September/021485.html [2] https://github.com/scipy/scipy/milestones/1.0

5 10

deprecating all misc.pilutil functions for 1.0
by Ralf Gommers Sept. 16, 2017

Sept. 16, 2017

Hi all, We've discussed this before, but never actually cleaned up all the misc.pilutil functions. I've just submitted https://github.com/scipy/scipy/pull/7869 which deprecates all of them - they're beyond rescuing and have better alternatives in Matplotlib, scikit-image and Pillow. Details in the PR description. Cheers, Ralf

1 0