Mailman 3 August 2018 - SciPy-Dev

SciPy 1.0 paper writing proposal
by Ralf Gommers Jan. 29, 2019

Jan. 29, 2019

Hi all, TL;DR, let's write the long journal paper on SciPy that we've wanted for a while, let's form a small committee to coordinate, and get it out the door in 2-3 months. Motivation --------------- (credits for most of this text: Evgeni) Many scipy contributors' day jobs are in academia. Bibliometry -- papers in refereed journals and citations of papers by other papers -- is one of the main performance indicators in most academic establishments. Since we do not generate papers, scipy contributions are all but invisible for the purposes of a contributor's annual report. Of course, details vary wildly; in many cases a contributor manages to balance their time, or to argue common sense with their superiors, or get an approval for scipy work, or just ignores the issue altogether -- but sooner or later there is a form to be filled or boxes to be checked, and scipy contributions simply do not fit in. A peer-reviewed journal paper on scipy will help contributors get the academic credit they deserve. We can write *the* paper for SciPy 1.0, with overall project structure, goals, etc., and for specific features/modules a focus on say the last 3 years. History ---------- For SciPy 1.0 we had three targets on the publicity/credits front: an interesting release announcement, interesting blogs/stories (NumFOCUS blog, Hacker News, etc.) and a paper. We didn't have the bandwidth for a paper in the end, the rest was successful. [1] is a previous announcement on this list about writing (a) paper(s) on SciPy. We wanted both "short papers" to cover one or two releases (target journal JOSS) and a full paper as the authoritative reference for SciPy. We had an earlier attempt for a "short paper", it's mostly written but has stalled (see [2]). We ran out of steam on that one. To avoid that this time around, it would be good to have a clear public plan, target dates, and a small committee rather than one person to drive things forward. Proposal ------------ Here's a proposal for all aspects of this exercise that I can think about now. Some parts stolen from the AstroPy paper [3] (because their process worked quite well). Form a small coordination committee of 3-5 people that set up the paper structure, move things along when parts stall, propose/take decisions as needed, invite co-authors, and organise paper submission/rework. Paper writing to be done by whoever volunteers for a section, not just the coordination committee. First outline/structure to be done by committee, which then asks for review of structure and volunteers for section writing. Scope: a 6-10 page paper, covering history, package scope and structure, community/organisational aspects, key features and recent enhancements per module, and roadmap. Authorship: anyone who made a substantial contribution in the history of the project. Here "substantial" is interpreted as anything beyond a one-line doc fix. Rationale: better to be too inclusive than exclusive. Sign-up via a web form, we send the link to that form to all email addresses in the commit history till v1.0. Author order (details tbd by committee): 1. The SciPy Developers 2. Maintainers, paper writers, other key contributors - in order of contribution level 3. All other authors - alphabetically ordered Submission target: mid-April, to either PeerJ Computer Science or Journal of Open Research Software (tbd by committee). Comments? Volunteers for committee? References ---------------- [1] https://mail.python.org/pipermail/scipy-dev/2016-August/021474.html [2] https://github.com/scipy/scipy-articles/pull/4 [3] https://github.com/astropy/astropy-v2.0-paper Cheers, Ralf

12 19

Re: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions
by Nikolay Mayorov Sept. 17, 2018

Sept. 17, 2018

Hi, Aditya! Glad that you set up the blog and good job on setting up the documentation build as well. Curious, what is this blogging platform like? How do you create posts in it? As for your first post: while not strictly necessary I think it would be nice to see a more thorough introductory post with a brief overview, motivation and/or an example. Do you want to work on it? Best, Nikolay

4 15

General discussion on parallelisation
by Andrew Nelson Sept. 8, 2018

Sept. 8, 2018

As part of https://github.com/scipy/scipy/pull/8259 I'm proposing that a `workers` keyword is added to optimize.differential_evolution to parallelise some computation. The proposal is that: 1. the workers keyword accepts either an integer or an object with a map-like method. 2. If an integer is supplied then the parallelisation is taken care of by scipy (more on that later), with -1 signifying that all processors are to be used. 3. If an object with a map-like method is supplied, e.g. `multiprocessing.Pool.map`, `mpi4py.futures.MPIPoolExecutor.map`, etc, then the parallelisation is taken care of by that object. This allows the user to specify the parallelisation configuration for their problem. 4. If workers=1, then computation will be done by the builtin `map` function. Now we come to the under the hood part. I've written something called PoolWrapper ( https://github.com/andyfaff/scipy/blob/b14bb513c0ffb9807a67663d39b9ab399375…) which wraps `multiprocessing.Pool` to achieve the behaviour outlined above. It can be used as a context manager, or the user of the object can decide when to close the resources opened by PoolWrapper. I've looked at using joblib instead of PoolWrapper and it seems useful but it doesn't have a couple of bits of functionality that are needed for this specific problem: viz: 5. joblib.Parallel doesn't have a map method (desirable to allow 3) so a small wrapper would have to be created anyway. 6. joblib.Parallel creates/destroys a multiprocessing.Pool each time the Parallel object is `__call__`ed. This leads to significant overhead. One can use the Parallel object with a context manager, which allows reuse of the Pool, but I don't think that's do-able in the context of using the DifferentialEvolutionSolver (DES) object as an iterator: >>> solver = DifferentialEvolutionSolver(func, bounds) >>> # use DES object as an iterator >>> for it in solver: ... res = next(solver) >>> print(res) >>> # use the DES.solve method >>> res = solver.solve() Whilst the DES object is not currently public (it's called by the differential_evolution function) it would be nice to expose it in the future, and people will want to use both approaches. Unfortunately with the first approach if we used joblib.Parallel we'd have to use Parallel.__call__ in DES.next() which has the overhead of creating/destroying Pools. For efficient use of resources the Pool should persist for the lifetime of the DES object. I also looked into `concurrent.futures.ProcessPoolExecutor`, but it's not available for Python 2.7. The purpose of this email is to elicit feedback for developing parallelisation strategy for scipy - what does the public interface look like, what does scipy do under the hood? Under the hood I think a mixture of PoolWrapper and joblib.Parallel could be used (with scipy vendoring joblib). A. -- _____________________________________ Dr. Andrew Nelson _____________________________________

5 18

1D parabolic PDE solver
by Saixoz Sept. 7, 2018

Sept. 7, 2018

Dear all, A recent task that I had to perform involved rewriting a large number of scripts and files from MATLAB to Python. In the process I noticed that Scipy lacks a PDE solver. When I tried to find alternatives, the options found were all unusable for various reasons. I wrote a solver which solves slab symmetric problems with a similar PDE form to that which MATLAB’s pdepe solves, namely c(x,u,t,du/dx)(du/dt) = (d/dx)(f(x,u,t,du/dx) + s(x,u,t,du/dx) using the method of lines approach, finite differences, and the BDF method provided by solve_ivp. I was wondering whether there would be some interest in adding the solver under scipy.integrate, to begin building a part of a collection. I feel that for very simple problems such as the ones that my solver can solve, users shouldn’t need to turn to overly complex packages. If there is interest, I would need to clean up the code and add tests and stability checks to it, help/advice would be appreciated. I believe BDF already has stability in the time grid, so I would only need to check stability in space grid? Can someone confirm? I’d also like to move a large chunk of the code to Cython since I wrote it in native Python and used Numba to speed it up, which still makes it rather slow. I’ve worked in C and Python before but never in Cython, could someone provide me with a link to get started? As far as legal issues go, there should be no concern – I explicitly avoided looking at and using the MATLAB source code as a reference since I wanted to leave publishing the solver under Scipy an option. I did use their solver to check some of the solutions that my solver produced though. The solver was written while working for the ETH Zurich, where my supervisor responded very enthusiastically to the possibility of providing the solver to the public under Scipy, so there should be no issues there either. All the best, Nicolas

2 1

Cython code generation with Jinja2 templates
by Mark Alexander Mikofski Aug. 31, 2018

Aug. 31, 2018

Hi All, I would like to use Jinja2 templates to generate the Cython pyx and pxd files necessary for the Cython Optimize Zeros API proposed in #8431, maybe in a follow up PR https://github.com/scipy/scipy/pull/8431 Does anyone have any experience with code generation in SciPy? Is it best to generate the files offline and then commit them to the repository, or is it better to add some scripting to either setup.py or to the CI yaml config files? I put something together in a branch in my scipy fork, and it recreates the existing code verbatim. See the templates folder and _generate_zeros_type.py script. https://github.com/mikofski/scipy/tree/cython_optimize_generate_code/scipy/… Any comments would be greatly appreciated. thanks! Mark -- Mark Mikofski, PhD (2005) *Fiat Lux*

5 8

Initializing error number in optimize zeros
by Mark Alexander Mikofski Aug. 30, 2018

Aug. 30, 2018

Hi, I have a question about the scalar function root finders in optimize/Zeros like brentq, etc, that are written in C, the error number is never explicitly initialized, so it could be previously set to -2 or -1, which could be misinterpreted as a failure. Any reason not to set the error number to zero explicitly at the start of the function? I've created an issue to track this : ENH: initialize params->error_num to zero in optimize/Zeros/*.c · Issue #9189 · scipy/scipy https://github.com/scipy/scipy/issues/9189 and implemented it in the cython_optimize_struct branch of my fork, eg bisect.c: https://github.com/mikofski/scipy/blob/cython_optimize_struct/scipy/optimiz… Please, any comment? Thanks! Mark

1 0

Plans for PyPy3 wheels?
by Juan Luis Cano Aug. 28, 2018

Aug. 28, 2018

Hi all, [I'm sending this to scipy-dev because I'm already subscribed, but I guess someone could bring this discussion to numpy-discussion as well] Lately PyPy3 (I'm only interested in Python 3) has matured a lot and it's becoming more and more interesting with time. The developers invested a lot of effort in improving the compatibility with CPython C extensions, and I'm happy to see that SciPy in particular passes all tests in PyPy3: https://circleci.com/gh/scipy/scipy/8318 There was an effort by Antonio Cuni, a PyPy developer, to produce wheels of popular scientific packages. However, it's still a non official thing, and I tried to link them to OpenBLAS but I'm not skilled (or patient) enough: https://github.com/antocuni/pypy-wheels/pull/6 Are there any plans to officially support PyPy3 (and perhaps PyPy) and upload official wheels to pypi.org? Lack of binary packages is holding me back for playing more with PyPy, and there are still no clear plans on what to do with conda packages: https://github.com/conda-forge/pypy2.7-feedstock/issues/1 Did other SciPy or NumPy developers play with PyPy3? Do you think this would be an interesting thing to have? What needs to be done to get there? Best, -- Juan Luis Cano

3 3

please review cython optimize zeros api
by Mark Alexander Mikofski Aug. 24, 2018

Aug. 24, 2018

Hi, PR 8431 is ready for review, IMHO. https://github.com/scipy/scipy/pull/8431 This PR adds a Cython API for root-finding of scalar functions in Optimize. It exposes brentq, brenth, ridder, bisect, newton, secant, and halley with 3 different callback signatures, two which are safe with Cython `nogil` which allows them to be called with Cython `prange`. this PR is a follow on to #8357, and allows efficient looping from C of large arrays with the same callback using any of the scalar-function root finders. I would appreciate your comments. Best, Mark -- Mark Mikofski, PhD (2005) *Fiat Lux*

1 0

Re: [SciPy-Dev] Proposal for some new fitting routines
by Robert Lucente Aug. 23, 2018

Aug. 23, 2018

2 1

Re: [SciPy-Dev] Proposal for some new fitting routines
by Christoph Baumgarten Aug. 23, 2018

Aug. 23, 2018

I have never used these methods, but they look useful and given the links to stackoverflow etc, there seems to be a certain "demand" for them. In stats, we have linregress as well as theilslopes (robust regression), i recently created PR #9101 to add another robust regression technique. so it could fit there. On Thu, Aug 23, 2018 at 6:02 PM <scipy-dev-request(a)python.org> wrote: > Send SciPy-Dev mailing list submissions to > scipy-dev(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request(a)python.org > > You can reach the person managing the list at > scipy-dev-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. Re: Proposal for some new fitting routines (Joseph Fox-Rabinovitz) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Wed, 22 Aug 2018 13:32:04 -0400 > From: Joseph Fox-Rabinovitz <jfoxrabinovitz(a)gmail.com> > To: SciPy Developers List <scipy-dev(a)python.org> > Subject: Re: [SciPy-Dev] Proposal for some new fitting routines > Message-ID: > < > CAAa1KPYoS__fKgb3o+4xt0s+Xn-1DivQSZsD0x-DqraXLFhj3Q(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > On Tue, Aug 21, 2018 at 11:02 PM <josef.pktd(a)gmail.com> wrote: > > > > > > > On Tue, Aug 21, 2018 at 10:12 PM, Joseph Fox-Rabinovitz < > > jfoxrabinovitz(a)gmail.com> wrote: > > > >> I had sent out an email about a PR proposing an exponential and a power > >> fit function, but I did not specify the PR number. The PR is #9158. > >> > >> Regards, > >> > >> - Joe > >> > >> > >> On Mon, Aug 20, 2018, 3:40 PM Joseph Fox-Rabinovitz < > >> jfoxrabinovitz(a)gmail.com> wrote: > >> > >>> I have added a function for fitting exponential and power curves. This > >>> seems to > >>> be a fairly common problem (e.g., see the list of selections from stack > >>> exchange > >>> below). The algorithm I have implemented here is objectively better > than > >>> the > >>> common solution of fitting to log(y) because it requires no up-front > >>> knowledge > >>> of the intercept, or any other estimations. > >>> > >>> Assuming that this PR is acceptable, I would like to get some criticism > >>> on where > >>> I chose to put it in the directory structure. My thought was to create > a > >>> separate > >>> module for such common non-iterative optimizations. > >>> > >>> I am also unsure of whether my citation of the original work is > formatted > >>> properly. I would like to make sure that the author gets the credit and > >>> publicity they > >>> deserve. The paper is mentioned (by the author) in answers to [1] and > >>> [2] below. > >>> > >>> List of relevant stack exchange questions: > >>> > >>> [1]: > >>> > https://stackoverflow.com/questions/3938042/fitting-exponential-decay-with-… > >>> [2]: > >>> > https://math.stackexchange.com/questions/1337601/fit-exponential-with-const… > >>> [3]: > >>> > https://math.stackexchange.com/questions/350754/fitting-exponential-curve-t… > >>> [4]: > >>> > https://math.stackexchange.com/questions/1999069/fit-curve-exponential-mayb… > >>> [5]: > >>> > https://mathematica.stackexchange.com/questions/87645/exponential-fit-to-da… > >>> [6]: > >>> > https://stackoverflow.com/questions/48099026/exponential-curve-fit-will-not… > >>> [7]: > >>> > https://stackoverflow.com/questions/50683768/exponent-curve-fitting-in-pyth… > >>> [8]: > >>> > https://stackoverflow.com/questions/49565152/curve-fit-an-exponential-decay… > >>> [9]: > >>> > https://stackoverflow.com/questions/21420792/exponential-curve-fitting-in-s… > >>> > >>> Regards, > >>> > >>> - Joe > >>> > >> > > > > The main question in my opinion is whether scipy should add functions in > > this area, this area essentially doesn't exist in scipy and is covered, > for > > example, by lmfit. > > > > Except for the scipy.stats.distribution and the simple linear model, > there > > are no fit or estimation methods for specific functions, AFAIK. ODR is > > related but also doesn't provide specific nonlinear functions > > scipy.optimize has the optimization algorithm but not specific nonlinear > > functions that are estimated. > > > > lmfit has built in functions for nonlinear estimation > > https://lmfit.github.io/lmfit-py/builtin_models.html > > and there some other less well known packages, AFAIR. > > > > (There was also a discussion about adding similar predefined functions to > > statsmodels, but they were never added and for those cases I usually > refer > > to lmfit.) > > > > lmfit definitely looks like a good choice to put the functions. On the > other hand, it offers much more advanced features than just a simple > regression. > > My guess is that, unless there is a clear plan to expand in this area in > > scipy, then pointing to lmfit would be the better approach. > > Nevertheless, there could also be demand for having something simple in > > scipy. E.g. scipy.stats is a bit "eclectic" and has some overlap with > > statsmodels, or the other way around. scipy.stats has simple linear > > regression but we never added a full OLS, > > > > On the other hand, the functions I propose are extremely simple, and would > do well alongside linregress if they were renamed expregress and > powregress. > > > > multiple linear regression to scipy.stats because there would be too much > > code duplication especially if there is feature creep and users want > > similar extended results as statsmodels provides. > > > > numpy has polynomial fitting as the only fitting function, AFAIK. > > > > Definitely out of scope for numpy and borderline for scipy. > > > > > > Josef > > > > > > > > > > > >> > >> _______________________________________________ > >> SciPy-Dev mailing list > >> SciPy-Dev(a)python.org > >> https://mail.python.org/mailman/listinfo/scipy-dev > >> > >> > > _______________________________________________ > > SciPy-Dev mailing list > > SciPy-Dev(a)python.org > > https://mail.python.org/mailman/listinfo/scipy-dev > > >

1 0