Mailman 3 June 2018 - SciPy-Dev

SciPy 1.0 paper writing proposal
by Ralf Gommers 29 Jan '19

29 Jan '19

Hi all, TL;DR, let's write the long journal paper on SciPy that we've wanted for a while, let's form a small committee to coordinate, and get it out the door in 2-3 months. Motivation --------------- (credits for most of this text: Evgeni) Many scipy contributors' day jobs are in academia. Bibliometry -- papers in refereed journals and citations of papers by other papers -- is one of the main performance indicators in most academic establishments. Since we do not generate papers, scipy contributions are all but invisible for the purposes of a contributor's annual report. Of course, details vary wildly; in many cases a contributor manages to balance their time, or to argue common sense with their superiors, or get an approval for scipy work, or just ignores the issue altogether -- but sooner or later there is a form to be filled or boxes to be checked, and scipy contributions simply do not fit in. A peer-reviewed journal paper on scipy will help contributors get the academic credit they deserve. We can write *the* paper for SciPy 1.0, with overall project structure, goals, etc., and for specific features/modules a focus on say the last 3 years. History ---------- For SciPy 1.0 we had three targets on the publicity/credits front: an interesting release announcement, interesting blogs/stories (NumFOCUS blog, Hacker News, etc.) and a paper. We didn't have the bandwidth for a paper in the end, the rest was successful. [1] is a previous announcement on this list about writing (a) paper(s) on SciPy. We wanted both "short papers" to cover one or two releases (target journal JOSS) and a full paper as the authoritative reference for SciPy. We had an earlier attempt for a "short paper", it's mostly written but has stalled (see [2]). We ran out of steam on that one. To avoid that this time around, it would be good to have a clear public plan, target dates, and a small committee rather than one person to drive things forward. Proposal ------------ Here's a proposal for all aspects of this exercise that I can think about now. Some parts stolen from the AstroPy paper [3] (because their process worked quite well). Form a small coordination committee of 3-5 people that set up the paper structure, move things along when parts stall, propose/take decisions as needed, invite co-authors, and organise paper submission/rework. Paper writing to be done by whoever volunteers for a section, not just the coordination committee. First outline/structure to be done by committee, which then asks for review of structure and volunteers for section writing. Scope: a 6-10 page paper, covering history, package scope and structure, community/organisational aspects, key features and recent enhancements per module, and roadmap. Authorship: anyone who made a substantial contribution in the history of the project. Here "substantial" is interpreted as anything beyond a one-line doc fix. Rationale: better to be too inclusive than exclusive. Sign-up via a web form, we send the link to that form to all email addresses in the commit history till v1.0. Author order (details tbd by committee): 1. The SciPy Developers 2. Maintainers, paper writers, other key contributors - in order of contribution level 3. All other authors - alphabetically ordered Submission target: mid-April, to either PeerJ Computer Science or Journal of Open Research Software (tbd by committee). Comments? Volunteers for committee? References ---------------- [1] https://mail.python.org/pipermail/scipy-dev/2016-August/021474.html [2] https://github.com/scipy/scipy-articles/pull/4 [3] https://github.com/astropy/astropy-v2.0-paper Cheers, Ralf

12 19

Re: [SciPy-Dev] [GSOC 2018 Project Thread]: Rotation formalism in 3 dimensions
by Nikolay Mayorov 17 Sep '18

17 Sep '18

Hi, Aditya! Glad that you set up the blog and good job on setting up the documentation build as well. Curious, what is this blogging platform like? How do you create posts in it? As for your first post: while not strictly necessary I think it would be nice to see a more thorough introductory post with a brief overview, motivation and/or an example. Do you want to work on it? Best, Nikolay

4 15

General discussion on parallelisation
by Andrew Nelson 08 Sep '18

08 Sep '18

As part of https://github.com/scipy/scipy/pull/8259 I'm proposing that a `workers` keyword is added to optimize.differential_evolution to parallelise some computation. The proposal is that: 1. the workers keyword accepts either an integer or an object with a map-like method. 2. If an integer is supplied then the parallelisation is taken care of by scipy (more on that later), with -1 signifying that all processors are to be used. 3. If an object with a map-like method is supplied, e.g. `multiprocessing.Pool.map`, `mpi4py.futures.MPIPoolExecutor.map`, etc, then the parallelisation is taken care of by that object. This allows the user to specify the parallelisation configuration for their problem. 4. If workers=1, then computation will be done by the builtin `map` function. Now we come to the under the hood part. I've written something called PoolWrapper ( https://github.com/andyfaff/scipy/blob/b14bb513c0ffb9807a67663d39b9ab399375…) which wraps `multiprocessing.Pool` to achieve the behaviour outlined above. It can be used as a context manager, or the user of the object can decide when to close the resources opened by PoolWrapper. I've looked at using joblib instead of PoolWrapper and it seems useful but it doesn't have a couple of bits of functionality that are needed for this specific problem: viz: 5. joblib.Parallel doesn't have a map method (desirable to allow 3) so a small wrapper would have to be created anyway. 6. joblib.Parallel creates/destroys a multiprocessing.Pool each time the Parallel object is `__call__`ed. This leads to significant overhead. One can use the Parallel object with a context manager, which allows reuse of the Pool, but I don't think that's do-able in the context of using the DifferentialEvolutionSolver (DES) object as an iterator: >>> solver = DifferentialEvolutionSolver(func, bounds) >>> # use DES object as an iterator >>> for it in solver: ... res = next(solver) >>> print(res) >>> # use the DES.solve method >>> res = solver.solve() Whilst the DES object is not currently public (it's called by the differential_evolution function) it would be nice to expose it in the future, and people will want to use both approaches. Unfortunately with the first approach if we used joblib.Parallel we'd have to use Parallel.__call__ in DES.next() which has the overhead of creating/destroying Pools. For efficient use of resources the Pool should persist for the lifetime of the DES object. I also looked into `concurrent.futures.ProcessPoolExecutor`, but it's not available for Python 2.7. The purpose of this email is to elicit feedback for developing parallelisation strategy for scipy - what does the public interface look like, what does scipy do under the hood? Under the hood I think a mixture of PoolWrapper and joblib.Parallel could be used (with scipy vendoring joblib). A. -- _____________________________________ Dr. Andrew Nelson _____________________________________

5 18

Are you using Bento to build scipy?
by CJ Carey 16 Jul '18

16 Jul '18

With the 1.0 release coming up, I figured this is a good time to take the pulse of the community regarding Bento. The project seems to have been abandoned before reaching stability, and it seems that the core problems it was trying to address have been at least partially fixed by the more official build systems. If you use Bento and would like to keep using it post 1.0, or if my previous sentence is wrong, please let me know!

10 15

Return residual and derivative in gradient descent methods like Newton
by Mark Alexander Mikofski 26 Jun '18

26 Jun '18

Hi, Would anyone disagree or would anyone be interested in a proposal to allow the derivative to be returned from the user supplied function as an optional second argument in gradient search method like Newton? EG >>> lambda x,a: (x**3-a, 3*x*"2) >>> newton(f, x0, fprime='f2', args=(a,)) Some simple tests show that this may have a 2X speed in cases where the derivative expression requires the value of the original function call. See this issue in which I proposed this idea and wore a sample test script https://github.com/scipy/scipy/issues/8354 I proposed a way to keep the existing API and add this new feature. * If `fprime` is a callable, then same as before * If it's a string like "f2" the get fprime from second output Thanks, Mark

4 4

Interest in improvements to the cKDTree codebase?
by Jesse Livezey 25 Jun '18

25 Jun '18

Hi everyone, I'm using cKDTrees for a project and noticed two potential ways to improve the code and have written one additional count method that might be useful to others. I have written code and some tests for all three items and can contribute if there is interest. 1) Allowing an array of rs in cKDTree.query_ball_point(). I started a PR here <https://github.com/scipy/scipy/pull/8818>. In principle, this should speed up multiple queries with different rs, but see 2. 2) I noticed that for the cases when cKDTree.query_ball_point() returns large numbers of points (>100), it is faster to loop over queries in python. This is true both with and without my PR. This is largely because single point queries do not sort the return indices, but multi-point queries DO sort them (see details here <https://github.com/scipy/scipy/issues/8838>). Removing the sorted() leads to considerable speedups, but is not backwards compatible. However, the sorted behavior is not in the method description and is not even internally consistent, so maybe it could just be removed or made optional? 3) I have written a cKDTree.count_ball_point() method that behaves like query_ball_point() but just returns the number of points rather than their indices. This is much faster than calling len(query_ball_point()). Let me know if any of this is of interest. Jesse -- Jesse Livezey he/him/his

2 2

scipy.sparse.linalg.svds
by Robert Lucente 24 Jun '18

24 Jun '18

https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.sv… states "This is a naive implementation using ARPACK as an eigensolver on A.H * A or A * A.H, depending on which one is more efficient." One of my co-workers is struggling to find an implementation for SVD for a sparse matrix. I was thinking of suggesting that he look at SciPy. The phrase "naive implementation" has made me think twice about making the suggestion. I just don't want the first intro of Python/SciPy into the organization to be a negative one. I poked around and found the following 2 articles http://fa.bianp.net/blog/2012/singular-value-decomposition-in-scipy/ https://simplyml.com/benchmarking-the-singular-value-decomposition/ I realize that I should be suspicious of bench marks and that you have to run your own. I would like to provide my co-worker the link to SciPy and then to 2 articles which can help guide him in creating his own bench mark. Any suggestions would be appreciated. Please note that I am not an expert in SVD.

3 2

NumPy 1.15.0rc1 released
by Charles R Harris 21 Jun '18

21 Jun '18

Hi All, On behalf of the NumPy team I'm pleased to announce the release of NumPy 1.15.0rc1. This release has an unusual number of cleanups, many deprecations of old functions, and improvements to many existing functions. A total of 423 pull reguests were merged for this release, please look at the release notes <https://github.com/numpy/numpy/releases/tag/v1.15.0rc1>for details. Some highlights are: - NumPy has switched to pytest for testing. - A new `numpy.printoptions` context manager. - Many improvements to the histogram functions. - Support for unicode field names in python 2.7. - Improved support for PyPy. The Python versions supported by this release are 2.7, 3.4-3.6. The wheels are linked with OpenBLAS 3.0, which should fix some of the linalg problems reported for NumPy 1.14, and the source archives were created using Cython 0.28.2 and should work with the upcoming Python 3.7. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/#description>, source archives are available from Github <https://github.com/numpy/numpy/releases/tag/v1.15.0rc1>. A total of 128 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Aaron Critchley + - Aarthi + - Aarthi Agurusa + - Alex Thomas + - Alexander Belopolsky - Allan Haldane - Anas Khan + - Andras Deak - Andrey Portnoy + - Anna Chiara - Aurelien Jarno + - Baurzhan Muftakhidinov - Berend Kapelle + - Bernhard M. Wiedemann - Bjoern Thiel + - Bob Eldering - Cenny Wenner + - Charles Harris - ChloeColeongco + - Chris Billington + - Christopher + - Chun-Wei Yuan + - Claudio Freire + - Daniel Smith - Darcy Meyer + - David Abdurachmanov + - David Freese - Deepak Kumar Gouda + - Dennis Weyland + - Derrick Williams + - Dmitriy Shalyga + - Eric Cousineau + - Eric Larson - Eric Wieser - Evgeni Burovski - Frederick Lefebvre + - Gaspar Karm + - Geoffrey Irving - Gerhard Hobler + - Gerrit Holl - Guo Ci + - Hameer Abbasi + - Han Shen - Hiroyuki V. Yamazaki + - Hong Xu - Ihor Melnyk + - Jaime Fernandez - Jake VanderPlas + - James Tocknell + - Jarrod Millman - Jeff VanOss + - John Kirkham - Jonas Rauber + - Jonathan March + - Joseph Fox-Rabinovitz - Julian Taylor - Junjie Bai + - Juris Bogusevs + - Jörg Döpfert - Kenichi Maehashi + - Kevin Sheppard - Kimikazu Kato + - Kirit Thadaka + - Kritika Jalan + - Lakshay Garg + - Lars G + - Licht Takeuchi - Louis Potok + - Luke Zoltan Kelley - MSeifert04 + - Mads R. B. Kristensen + - Malcolm Smith + - Mark Harfouche + - Marten H. van Kerkwijk + - Marten van Kerkwijk - Matheus Vieira Portela + - Mathieu Lamarre - Mathieu Sornay + - Matthew Brett - Matthew Rocklin + - Matthias Bussonnier - Matti Picus - Michael Droettboom - Miguel Sánchez de León Peque + - Mike Toews + - Milo + - Nathaniel J. Smith - Nelle Varoquaux - Nicholas Nadeau + - Nick Minkyu Lee + - Nikita + - Nikita Kartashov + - Nils Becker + - Oleg Zabluda - Orestis Floros + - Pat Gunn + - Paul van Mulbregt + - Pauli Virtanen - Pierre Chanial + - Ralf Gommers - Raunak Shah + - Robert Kern - Russell Keith-Magee + - Ryan Soklaski + - Samuel Jackson + - Sebastian Berg - Siavash Eliasi + - Simon Conseil - Simon Gibbons - Stefan Krah + - Stefan van der Walt - Stephan Hoyer - Subhendu + - Subhendu Ranjan Mishra + - Tai-Lin Wu + - Tobias Fischer + - Toshiki Kataoka + - Tyler Reddy + - Varun Nayyar - Victor Rodriguez + - Warren Weckesser - Zane Bradley + - fo40225 - lumbric + - luzpaz + - mamrehn + - tynn + - xoviat Cheers Chuck

1 0

ANN: SfePy 2018.2
by Robert Cimrman 19 Jun '18

19 Jun '18

I am pleased to announce release 2018.2 of SfePy. Description ----------- SfePy (simple finite elements in Python) is a software for solving systems of coupled partial differential equations by the finite element method or by the isogeometric analysis (limited support). It is distributed under the new BSD license. Home page: http://sfepy.org Mailing list: https://mail.python.org/mm3/mailman3/lists/sfepy.python.org/ Git (source) repository, issue tracker: https://github.com/sfepy/sfepy Highlights of this release -------------------------- - generalized-alpha and velocity Verlet elastodynamics solvers - terms for dispersion in fluids - caching of reference coordinates for faster repeated use of probes - new wrapper of MUMPS linear solver for parallel runs For full release notes see http://docs.sfepy.org/doc/release_notes.html#id1 (rather long and technical). Cheers, Robert Cimrman --- Contributors to this release in alphabetical order: Robert Cimrman Lubos Kejzlar Vladimir Lukes Matyas Novak

1 0

Re: [SciPy-Dev] [Numpy-discussion] rackspace ssl certificates
by Charles R Harris 18 Jun '18

18 Jun '18

On Mon, Jun 18, 2018 at 2:22 PM, Nathan Goldbaum <nathan12343(a)gmail.com> wrote: > I think Matthew Brett needs to fix this. > That would be nice, but I'm not convinced it is helpful :) I note that latest `apache-libcloud` does not install directly on windows, there seem to be some missing dependencies. <snip> Chuck

1 0