Mailman 3 July 2018 - SciPy-Dev

SciPy 1.0 paper writing proposal
by Ralf Gommers Jan. 29, 2019

Jan. 29, 2019

Hi all, TL;DR, let's write the long journal paper on SciPy that we've wanted for a while, let's form a small committee to coordinate, and get it out the door in 2-3 months. Motivation --------------- (credits for most of this text: Evgeni) Many scipy contributors' day jobs are in academia. Bibliometry -- papers in refereed journals and citations of papers by other papers -- is one of the main performance indicators in most academic establishments. Since we do not generate papers, scipy contributions are all but invisible for the purposes of a contributor's annual report. Of course, details vary wildly; in many cases a contributor manages to balance their time, or to argue common sense with their superiors, or get an approval for scipy work, or just ignores the issue altogether -- but sooner or later there is a form to be filled or boxes to be checked, and scipy contributions simply do not fit in. A peer-reviewed journal paper on scipy will help contributors get the academic credit they deserve. We can write *the* paper for SciPy 1.0, with overall project structure, goals, etc., and for specific features/modules a focus on say the last 3 years. History ---------- For SciPy 1.0 we had three targets on the publicity/credits front: an interesting release announcement, interesting blogs/stories (NumFOCUS blog, Hacker News, etc.) and a paper. We didn't have the bandwidth for a paper in the end, the rest was successful. [1] is a previous announcement on this list about writing (a) paper(s) on SciPy. We wanted both "short papers" to cover one or two releases (target journal JOSS) and a full paper as the authoritative reference for SciPy. We had an earlier attempt for a "short paper", it's mostly written but has stalled (see [2]). We ran out of steam on that one. To avoid that this time around, it would be good to have a clear public plan, target dates, and a small committee rather than one person to drive things forward. Proposal ------------ Here's a proposal for all aspects of this exercise that I can think about now. Some parts stolen from the AstroPy paper [3] (because their process worked quite well). Form a small coordination committee of 3-5 people that set up the paper structure, move things along when parts stall, propose/take decisions as needed, invite co-authors, and organise paper submission/rework. Paper writing to be done by whoever volunteers for a section, not just the coordination committee. First outline/structure to be done by committee, which then asks for review of structure and volunteers for section writing. Scope: a 6-10 page paper, covering history, package scope and structure, community/organisational aspects, key features and recent enhancements per module, and roadmap. Authorship: anyone who made a substantial contribution in the history of the project. Here "substantial" is interpreted as anything beyond a one-line doc fix. Rationale: better to be too inclusive than exclusive. Sign-up via a web form, we send the link to that form to all email addresses in the commit history till v1.0. Author order (details tbd by committee): 1. The SciPy Developers 2. Maintainers, paper writers, other key contributors - in order of contribution level 3. All other authors - alphabetically ordered Submission target: mid-April, to either PeerJ Computer Science or Journal of Open Research Software (tbd by committee). Comments? Volunteers for committee? References ---------------- [1] https://mail.python.org/pipermail/scipy-dev/2016-August/021474.html [2] https://github.com/scipy/scipy-articles/pull/4 [3] https://github.com/astropy/astropy-v2.0-paper Cheers, Ralf

12 19

Re: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions
by Nikolay Mayorov Sept. 17, 2018

Sept. 17, 2018

Hi, Aditya! Glad that you set up the blog and good job on setting up the documentation build as well. Curious, what is this blogging platform like? How do you create posts in it? As for your first post: while not strictly necessary I think it would be nice to see a more thorough introductory post with a brief overview, motivation and/or an example. Do you want to work on it? Best, Nikolay

4 15

General discussion on parallelisation
by Andrew Nelson Sept. 8, 2018

Sept. 8, 2018

As part of https://github.com/scipy/scipy/pull/8259 I'm proposing that a `workers` keyword is added to optimize.differential_evolution to parallelise some computation. The proposal is that: 1. the workers keyword accepts either an integer or an object with a map-like method. 2. If an integer is supplied then the parallelisation is taken care of by scipy (more on that later), with -1 signifying that all processors are to be used. 3. If an object with a map-like method is supplied, e.g. `multiprocessing.Pool.map`, `mpi4py.futures.MPIPoolExecutor.map`, etc, then the parallelisation is taken care of by that object. This allows the user to specify the parallelisation configuration for their problem. 4. If workers=1, then computation will be done by the builtin `map` function. Now we come to the under the hood part. I've written something called PoolWrapper ( https://github.com/andyfaff/scipy/blob/b14bb513c0ffb9807a67663d39b9ab399375…) which wraps `multiprocessing.Pool` to achieve the behaviour outlined above. It can be used as a context manager, or the user of the object can decide when to close the resources opened by PoolWrapper. I've looked at using joblib instead of PoolWrapper and it seems useful but it doesn't have a couple of bits of functionality that are needed for this specific problem: viz: 5. joblib.Parallel doesn't have a map method (desirable to allow 3) so a small wrapper would have to be created anyway. 6. joblib.Parallel creates/destroys a multiprocessing.Pool each time the Parallel object is `__call__`ed. This leads to significant overhead. One can use the Parallel object with a context manager, which allows reuse of the Pool, but I don't think that's do-able in the context of using the DifferentialEvolutionSolver (DES) object as an iterator: >>> solver = DifferentialEvolutionSolver(func, bounds) >>> # use DES object as an iterator >>> for it in solver: ... res = next(solver) >>> print(res) >>> # use the DES.solve method >>> res = solver.solve() Whilst the DES object is not currently public (it's called by the differential_evolution function) it would be nice to expose it in the future, and people will want to use both approaches. Unfortunately with the first approach if we used joblib.Parallel we'd have to use Parallel.__call__ in DES.next() which has the overhead of creating/destroying Pools. For efficient use of resources the Pool should persist for the lifetime of the DES object. I also looked into `concurrent.futures.ProcessPoolExecutor`, but it's not available for Python 2.7. The purpose of this email is to elicit feedback for developing parallelisation strategy for scipy - what does the public interface look like, what does scipy do under the hood? Under the hood I think a mixture of PoolWrapper and joblib.Parallel could be used (with scipy vendoring joblib). A. -- _____________________________________ Dr. Andrew Nelson _____________________________________

5 18

Re: [SciPy-Dev] SciPy-Dev Digest, Vol 177, Issue 21
by Matt Newville Aug. 1, 2018

Aug. 1, 2018

Hi Andrew, Dickson, All, Sorry for barging in rather late to this discussion, and for responding to a digest, and if this appears to be too negative. > Date: Mon, 23 Jul 2018 23:36:37 -0400 > From: dickson41152 <dickson41152(a)protonmail.com> > To: Andrew Nelson <andyfaff(a)gmail.com> > Cc: scipy-dev <scipy-dev(a)python.org> > Subject: Re: [SciPy-Dev] Proposed Modifications to Nelder-Mead Simplex > Algorithm > Message-ID: > > <5xgSJzGhTqz3FgLpZlSQCqNBm-v5aytNC9kOBDasi_Mus8NNwoCKjPVEmrVqOpIn2B22oT1r0jOc8cNWJaBxRicbjjkAClPDRUwCRdPgAXs=@ > protonmail.com> > > Content-Type: text/plain; charset="utf-8" > > Thank you for your interest Andrew. > > > variable transformations to form unconstrained problems from bound > constrained problems > > I am considering writing a new transformation function to calculate the > equations on that web page that you linked to discussing variable > transformations. If it is as straight-forward as I think it will be to > write, then I may also do some comparisons between the "clipping" method > and the sin() / arcsin() / sqrt() method you highlighted. > > As an aside, I was told informally about the existence of that method you > referenced during a conversation I had last week, but I have been finding > it very difficult to find any written discussion (i.e. actual analytical > expressions) about that method via web searches before I made the > proposals. So thank you very much for linking that web page - it is very > useful information for my own personal work as well as the proposed Scipy > modifications! > > > user-specified objective function values for points in the > user-specified initial simplex > > I have indeed used a keyword in the manner you describe (and therefore > similar in style to the initial_simplex keyword) - it seemed to me to be a > "clean" way of implementing this modification and very straightforward to > implement since it is so similar to the initial_simplex handling. > > > evaluation budget count check > > I am thinking of writing checks which will effectively skip the execution > of the remainder of the solver algorithm if the budget has been met, before > finally sorting the current simplex of evaluations and returning the best. > My inclination is to avoid using StopIteration since it is an exception > i.e. it suggests some kind of problem has occurred, where as my proposed > idea is to handle evaluations up to the exact budget as a feature. > > Anyway I haven't yet made an attempt on this modification, so my thoughts > are tentative and may change - especially if it turns out that I am trying > to write spaghetti. > > dickson I think that adding bounds to Nelder-Mead Simplex is an interesting idea. Using the Minuit-style bounds - using smooth functions to transform infiniite "internal" values to finite "external" values - would definitely be an easy and stable way to do that. But, with all due respect, I also think it is the wrong approach ;). The main point of the lmfit package that Andrew linked to is that the fitting *algorithm* should not support bounds on variable values. I do understand that other scipy.optimize algorithms support bounds, and yes, I do think that is an unfortunate choice ;). Instead, the *parameters* should have bounds (and perhaps other properties), and the algorithm should work on these parameter objects. In this way one does not add bounds to Nelder-Mead, and then add bounds to Powell's method, and then add bounds to Levenberg-Marquardt, etc. Rather, one adds bounds to parameters, and has the methods use these parameters instead of scalar values, and thus one add bounds to *all* methods in a consistent and transferrable way. A few months ago Andrew suggested a class-based approach to implementing fitting algorithms. I think this idea has a lot of merit to it. I also believe that lmfit successfully demonstrates that having parameter objects as described briefly above does better encapsulate the desired properties (bounds, etc) *and* makes the code implementing the algorithms clearer and simpler. In fact, if I understand the original goals here to be: 1. Allowing bound constraints 2. Allowing the user to specify initial objective function values for any or all points in a user's initial simplex. 3. Checking the objective function evaluation count against the user-specified evaluation budget at each function evaluation . I would point out that lmfit has a Nelder-Mead solver that already does #1 and #3 and that (as with bounds), #3 may be best done not within the solver, but in outer code that calls the solver, so that this may be available in a consistent manner to multiple methods. If #2 means to memoize each step in the process to avoid repeated calculation, this does not currently exist within lmfit. Such a goal might really be best done in the solver method itself. It may be also be doable using a per-iteration callback (which lmfit does have) , though I don't know that anyone one has ever tried this. I apologize if this appears to be an outsider yelling "you're doing it all wrong". I rely heavily on scipy.optimize for my daily work, and would like to be able to make these tools better but I am over-committed and cannot contribute significantly to the scipy ecosystem myself other than try to support lmfit as best I can. Cheers, --Matt

2 1

Re: [SciPy-Dev] [cython-users] [Cython] [Scipy-Dev] FFT from Cython
by Ian Henriksen July 30, 2018

July 30, 2018

Actually, this seems like a pretty reasonable SciPy feature request. Feel free to open an issue there. Thanks, Ian Henriksen On Mon, Jul 30, 2018 at 3:18 PM 'Chris Barker' via cython-users < cython-users(a)googlegroups.com> wrote: > BTW, > > Please don't spam so many lists with the same question. cython-users is > the right one for this. > > -CHB > > > > On Mon, Jul 30, 2018 at 10:50 AM, Chris Barker <chris.barker(a)noaa.gov> > wrote: > >> On Sun, Jul 29, 2018 at 6:17 PM, Eleftherios Garyfallidis < >> garyfallidis(a)gmail.com> wrote: >> >>> For a project in DIPY (http://dipy.org) we need to be calling 1D and 2D >>> FFTs and iFFTs very often. For this reason the most efficient way seems to >>> be through Cython. >>> >>> It would be great if we could call scipy's fft or numpy's fft without >>> the overhead. >>> >> >> have you profiled the overhead? Is it really significant? I"d expect the >> answer would only be yes if you were doing a lot of really small arrays -- >> is that the case? >> >> -CHB >> >> >> -- >> >> Christopher Barker, Ph.D. >> Oceanographer >> >> Emergency Response Division >> NOAA/NOS/OR&R (206) 526-6959 voice >> 7600 Sand Point Way NE >> <https://maps.google.com/?q=7600+Sand+Point+Way+NE&entry=gmail&source=g> >> (206) 526-6329 fax >> Seattle, WA 98115 (206) 526-6317 main reception >> >> Chris.Barker(a)noaa.gov >> > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE > <https://maps.google.com/?q=7600+Sand+Point+Way+NE&entry=gmail&source=g> > (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker(a)noaa.gov > > -- > > --- > You received this message because you are subscribed to the Google Groups > "cython-users" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to cython-users+unsubscribe(a)googlegroups.com. > For more options, visit https://groups.google.com/d/optout. >

1 0

Google Optimization Tools
by Robert Lucente July 30, 2018

July 30, 2018

1 0

[Cython] [Scipy-Dev] FFT from Cython
by Eleftherios Garyfallidis July 30, 2018

July 30, 2018

Hello all, For a project in DIPY (http://dipy.org) we need to be calling 1D and 2D FFTs and iFFTs very often. For this reason the most efficient way seems to be through Cython. It would be great if we could call scipy's fft or numpy's fft without the overhead. I would like to ask if someone has worked on this and can share some of hers/his experience or code. What we are trying to do is avoid increasing our dependencies. For example we prefer not using FFTW if this is not available in scipy/numpy. Finally, many thanks to scipy devs and cython devs for making available BLAS and LAPACK functions from Cython. This has been super useful for our project. I hope we can do something similar for FFT. Let me know if you have any ideas. Can we for example access the FFT functions of scipy directly from Cython. Has someone done this? Best regards, Eleftherios

4 4

Proposed Modifications to Nelder-Mead Simplex Algorithm
by dickson41152 July 24, 2018

July 24, 2018

Hi all, I have been working on optimisation problems with expensive black-box functions. As part of this work, I have made two modifications to the Nelder-Mead Simplex solver implementation provided by Scipy. These are: 1. Allowing bound constraints in the same manner as NLopt's Nelder-Mead implementation which is itself originally based upon the method of satisfying bound constraints in Box's Complex solver [1]. 2. Allowing the user to specify initial objective function values for any or all points in a user's initial simplex. I am also considering adding a further modification: 3. Checking the objective function evaluation count against the user-specified evaluation budget at each and every function evaluation in the solver, then terminating if the budget has been reached. Modification 1 moves a candidate point to within user specified bounds by simply "clipping" the infeasible variable value to lie on the bound, e.g if a candidate variable value of 10 is given for a problem with an upper bound constraint of 5 on that variable, then the candidate variable value is "clipped" to 5. This is how I believe NLopt's Nelder-Mead implementation works, and I believe that this is similar to the method in [1] which moves the value slightly within the bound by a very small magnitude. Modification 2 simply allows the solver to avoid having to evaluate all the points in the user-specified initial simplex if the user already has that information at hand. In my own work, I am performing experimental designs within the problem bound constraints before handing the best points from the design evaluations to the Nelder-Mead as an initial simplex. Since I already had the function evaluations, it made sense to give these the the solver rather than repeating the evaluations. In my use case this saves a lot of computational time since I am looking at expensive black-boxes. Modification 3 would ensure that the Nelder-Mead algorithm can only run one objective function evaluation before a check is made to see if the count of function evaluations has matched an evaluation budget. Currently, more than one function evaluation can be performed during a single solver iteration; further, no evaluation checks are made before the first iteration i.e. when evaluating the user-specified initial simplex. My use case demands that the Nelder-Mead solver tightly conforms to a user specified evaluation budget and no more. This is a time saving measure to avoid excess expensive black-box evaluations which will be rejected anyway since my other work demands tight conformity with the evaluation budget. I am a software development amateur so contributing to Scipy would be a brand new experience. However I am prepared to try out adding these modifications to Scipy, if people believe that these modifications may be useful to others. Please share your thoughts on these proposals! I look forward to your insight. Many thanks, colvin4993 [1] M. J. Box; A New Method of Constrained Optimization and a Comparison With Other Methods, The Computer Journal, Volume 8, Issue 1, 1 April 1965, Pages 42–52, https://doi.org/10.1093/comjnl/8.1.42

4 6

NumPy 1.15.0 released.
by Charles R Harris July 23, 2018

July 23, 2018

Hi All, On behalf of the NumPy team I'm pleased to announce the release of NumPy 1.15.0rc2. This release has an unusual number of cleanups, many deprecations of old functions, and improvements to many existing functions. A total of 438 pull reguests were merged for this release, please look at the release notes <https://github.com/numpy/numpy/releases/tag/v1.15.0>for details. Some highlights are: - NumPy has switched to pytest for testing. - A new `numpy.printoptions` context manager. - Many improvements to the histogram functions. - Support for unicode field names in python 2.7. - Improved support for PyPy. - Fixes and improvements to `numpy.einsum`. The Python versions supported by this release are 2.7, 3.4-3.7. The wheels are linked with OpenBLAS v0.3.0, which should fix some of the linalg problems reported for NumPy 1.14. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.15.0/>, source archives are available from Github <https://github.com/numpy/numpy/releases/tag/v1.15.0>. *Contributors* A total of 133 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Aaron Critchley + * Aarthi + * Aarthi Agurusa + * Alex Thomas + * Alexander Belopolsky * Allan Haldane * Anas Khan + * Andras Deak * Andrey Portnoy + * Anna Chiara * Aurelien Jarno + * Baurzhan Muftakhidinov * Berend Kapelle + * Bernhard M. Wiedemann * Bjoern Thiel + * Bob Eldering * Cenny Wenner + * Charles Harris * ChloeColeongco + * Chris Billington + * Christopher + * Chun-Wei Yuan + * Claudio Freire + * Daniel Smith * Darcy Meyer + * David Abdurachmanov + * David Freese * Deepak Kumar Gouda + * Dennis Weyland + * Derrick Williams + * Dmitriy Shalyga + * Eric Cousineau + * Eric Larson * Eric Wieser * Evgeni Burovski * Frederick Lefebvre + * Gaspar Karm + * Geoffrey Irving * Gerhard Hobler + * Gerrit Holl * Guo Ci + * Hameer Abbasi + * Han Shen * Hiroyuki V. Yamazaki + * Hong Xu * Ihor Melnyk + * Jaime Fernandez * Jake VanderPlas + * James Tocknell + * Jarrod Millman * Jeff VanOss + * John Kirkham * Jonas Rauber + * Jonathan March + * Joseph Fox-Rabinovitz * Julian Taylor * Junjie Bai + * Juris Bogusevs + * Jörg Döpfert * Kenichi Maehashi + * Kevin Sheppard * Kimikazu Kato + * Kirit Thadaka + * Kritika Jalan + * Kyle Sunden + * Lakshay Garg + * Lars G + * Licht Takeuchi * Louis Potok + * Luke Zoltan Kelley * MSeifert04 + * Mads R. B. Kristensen + * Malcolm Smith + * Mark Harfouche + * Marten H. van Kerkwijk + * Marten van Kerkwijk * Matheus Vieira Portela + * Mathieu Lamarre * Mathieu Sornay + * Matthew Brett * Matthew Rocklin + * Matthias Bussonnier * Matti Picus * Michael Droettboom * Miguel Sánchez de León Peque + * Mike Toews + * Milo + * Nathaniel J. Smith * Nelle Varoquaux * Nicholas Nadeau, P.Eng., AVS + * Nick Minkyu Lee + * Nikita + * Nikita Kartashov + * Nils Becker + * Oleg Zabluda * Orestis Floros + * Pat Gunn + * Paul van Mulbregt + * Pauli Virtanen * Pierre Chanial + * Ralf Gommers * Raunak Shah + * Robert Kern * Russell Keith-Magee + * Ryan Soklaski + * Samuel Jackson + * Sebastian Berg * Siavash Eliasi + * Simon Conseil * Simon Gibbons * Stefan Krah + * Stefan van der Walt * Stephan Hoyer * Subhendu + * Subhendu Ranjan Mishra + * Tai-Lin Wu + * Tobias Fischer + * Toshiki Kataoka + * Tyler Reddy + * Unknown + * Varun Nayyar * Victor Rodriguez + * Warren Weckesser * William D. Irons + * Zane Bradley + * cclauss + * fo40225 + * lapack_lite code generator + * lumbric + * luzpaz + * mamrehn + * tynn + * xoviat Cheers, Charles Harris

1 1

Link error on the website
by Matthieu Brucher July 19, 2018

July 19, 2018

Hi all, It seems that the link http://www.scipy.org/Tentative_NumPy_Tutorial is redirecting to a wrong page. I would suppose it's supposed to be https://docs.scipy.org/doc/numpy/user/quickstart.html instead of https://docs.scipy.org/doc/numpy-dev/user/quickstart.html Can someone fix it? Cheers, Matthieu -- Quantitative analyst, Ph.D. Blog: http://blog.audio-tk.com/ LinkedIn: http://www.linkedin.com/in/matthieubrucher

2 2