Mailman 3 September 2018 - SciPy-Dev

SciPy 1.0 paper writing proposal
by Ralf Gommers Jan. 29, 2019

Jan. 29, 2019

Hi all, TL;DR, let's write the long journal paper on SciPy that we've wanted for a while, let's form a small committee to coordinate, and get it out the door in 2-3 months. Motivation --------------- (credits for most of this text: Evgeni) Many scipy contributors' day jobs are in academia. Bibliometry -- papers in refereed journals and citations of papers by other papers -- is one of the main performance indicators in most academic establishments. Since we do not generate papers, scipy contributions are all but invisible for the purposes of a contributor's annual report. Of course, details vary wildly; in many cases a contributor manages to balance their time, or to argue common sense with their superiors, or get an approval for scipy work, or just ignores the issue altogether -- but sooner or later there is a form to be filled or boxes to be checked, and scipy contributions simply do not fit in. A peer-reviewed journal paper on scipy will help contributors get the academic credit they deserve. We can write *the* paper for SciPy 1.0, with overall project structure, goals, etc., and for specific features/modules a focus on say the last 3 years. History ---------- For SciPy 1.0 we had three targets on the publicity/credits front: an interesting release announcement, interesting blogs/stories (NumFOCUS blog, Hacker News, etc.) and a paper. We didn't have the bandwidth for a paper in the end, the rest was successful. [1] is a previous announcement on this list about writing (a) paper(s) on SciPy. We wanted both "short papers" to cover one or two releases (target journal JOSS) and a full paper as the authoritative reference for SciPy. We had an earlier attempt for a "short paper", it's mostly written but has stalled (see [2]). We ran out of steam on that one. To avoid that this time around, it would be good to have a clear public plan, target dates, and a small committee rather than one person to drive things forward. Proposal ------------ Here's a proposal for all aspects of this exercise that I can think about now. Some parts stolen from the AstroPy paper [3] (because their process worked quite well). Form a small coordination committee of 3-5 people that set up the paper structure, move things along when parts stall, propose/take decisions as needed, invite co-authors, and organise paper submission/rework. Paper writing to be done by whoever volunteers for a section, not just the coordination committee. First outline/structure to be done by committee, which then asks for review of structure and volunteers for section writing. Scope: a 6-10 page paper, covering history, package scope and structure, community/organisational aspects, key features and recent enhancements per module, and roadmap. Authorship: anyone who made a substantial contribution in the history of the project. Here "substantial" is interpreted as anything beyond a one-line doc fix. Rationale: better to be too inclusive than exclusive. Sign-up via a web form, we send the link to that form to all email addresses in the commit history till v1.0. Author order (details tbd by committee): 1. The SciPy Developers 2. Maintainers, paper writers, other key contributors - in order of contribution level 3. All other authors - alphabetically ordered Submission target: mid-April, to either PeerJ Computer Science or Journal of Open Research Software (tbd by committee). Comments? Volunteers for committee? References ---------------- [1] https://mail.python.org/pipermail/scipy-dev/2016-August/021474.html [2] https://github.com/scipy/scipy-articles/pull/4 [3] https://github.com/astropy/astropy-v2.0-paper Cheers, Ralf

12 19

Improving spatial.distance code and performance
by Kane & Anna O'Donnell Oct. 9, 2018

Oct. 9, 2018

Hi all, TLDR; I'm wondering about a) porting the spatial.distance code to Cython, and b) adding some performance optimizations, and I'd like the dev community's input/feedback. For context (though I don't really need you to read these to give the feedback I'm looking for), - original issue/proposal: https://github.com/scipy/scipy/issues/9205 - PR: https://github.com/scipy/scipy/pull/9218 Before submitting the PR, I naively thought it was going to be nice Cython or similar. Turns out it's some pretty old code, that I found pretty hard to wrap my head around and understand. I eventually figured it out after spending ages debugging a nastily hidden 'bug', and demonstrated the performance optimizations, but it prompted the discussion about whether it was best to port everything to Cython first. *Existing stuff to Cython* Doing so shouldn't be too hard, and it shouldn't change any functionality, except to replace the distance functions with their Cython ones (instead of the current approach, where the distance functions are actually numpy things, and there's not supported access to the underlying C stuff). A few possible 'bugs' (as above) should hopefully become non-issues too. So, it'd be a win for performance (e.g. all the distance functions will be much faster), and code quality, and future maintainability and development. However, things to think about: - should I just replace like-for-like, or consider some nicer OOP stuff like e.g. sklearn's approach (which is already Cython)? https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/neighbors/… (I'm guessing the reason they rolled their own was because they found scipy lacking, as above.) In fact, we could largely just copy that file over. Not sure about the interplay between scipy and scikit learn though. - what's the best way to structure a PR? One metric at a time, or the whole caboodle? *Adding performance optimizations and future functionality* As per the links, this was initially about providing a nifty performance hack. It should still be pretty easy to implement. Personally, I think it makes sense to implement after the Cythonization - unless the community are against that. However, there are other possibilities: - using 'indices' within calculations. E.g. when doing pdist, it might pay to use a tree of some description. I also proposed another 'index' to optimize the 'bail early' approach further (which would, I think, actually work well with trees too). This would involve more API changes, possibly significant. - using approximate searches (e.g. Faiss). My understanding is that depending on other libraries probably isn't really an option, so I'm not sure what means. - maybe other cool approaches like https://github.com/droyed/eucl_dist - providing a way to 'tune' distance computations to be optimal to your particular dataset and constraints (e.g. my 'bail early' optimization might be a lot faster or a bit slower, depending on your data ... or you might be OK with approximate matching with a 'low' error rate, etc.) I guess what I'd like to see is a single place where users can get access to everything related to distance metrics and their uses, including all sorts of optimizations etc. (possibly for different hardware, and possibly distributed). To do that well is a pretty big undertaking, and I don't know whether it's suited to scipy - e.g. maybe scipy doesn't really care about distance stuff, or only wants to stick with 'simple' distance metric cases (e.g. a few thousand vectors, etc.). So, maybe it'd be better to start a completely new python package - which would probably be a lot easier to develop as I'd have a lot more flexibility (e.g. to depend on other packages, and not have to worry about breaking the scipy API etc.). On the other hand (as discussed in the latest comment on the PR), that might not be best - it might never get used/maintained etc. So, what does the community think is the best approach? I've got too little context of what scipy is and what it's aiming for, and I don't want to head off on the wrong tack. Comments on any of the other implied questions would also be appreciated. Thanks, kodonnell

4 5

planning 1.2.0 release
by Ralf Gommers Sept. 27, 2018

Sept. 27, 2018

Hi all, We released 1.1.0 in early May, so aiming for a 1.2.0 in the first half of November sounds about right. A couple of questions: - is anyone interested in being the release manager for this release? (I can do it, but volunteers *very* welcome) - are there any PRs or issues you are aware of that you want for the next release and are not under https://github.com/scipy/scipy/milestone/36? If so, please add them or suggest them here Cheers, Ralf

7 10

Proposal for a new scipy.special function: Voigt
by Muhammad Firmansyah Kasim Sept. 23, 2018

Sept. 23, 2018

I am proposing to implement a new special function under `scipy.special` module: Voigt function. Voigt function/profile is a very common line profile in atomic transitions and widely used in the spectroscopy field. It is a convolution between Gaussian and Cauchy distribution, thus it is not trivial to calculate it efficiently. There are several approximations to calculate it efficiently, but I am proposing to use the one in the "Stellar atmosphere" by Mihalas (1978). It is the one used in FLYCHK by NIST and IAEA (I work closely with the author of the code). Please let me know if you have any suggestions or objections to the proposal. Thanks! Muhammad

3 2

Numpy 1.14.6 release
by Charles R Harris Sept. 23, 2018

Sept. 23, 2018

Hi All, On behalf of the NumPy team, I am pleased to announce the release of NumPy 1.14.6. This is a bug fix release for bugs and regressions reported following the 1.14.5 release. The significant fixes are: - Fix for behavior change in ``ma.masked_values(shrink=True)`` - Fix the new cached allocations machinery to be thread safe. The Python versions supported by this release are 2.7, 3.4-3.7. The wheels are linked with OpenBLAS v0.3.0, which should fix some of the linalg problems reported for earlier releases of NumPy 1.14. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.14.6/>, source archives are available from Github <https://github.com/numpy/numpy/releases/tag/v1.14.6>. Compatibility Note ================== The NumPy 1.14.6 OS X wheels released on PyPI no longer contain 32-bit binaries. See #11625 <https://github.com/numpy/numpy/issues/11625> for the related discussion. Those needing 32-bit support should look elsewhere or build from source. Contributors ============ A total of 4 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Charles Harris - Eric Wieser - Julian Taylor - Matti Picus Pull requests merged ==================== A total of 4 pull requests were merged for this release. - #11985: BUG: fix cached allocations without the GIL - #11986: BUG: Undo behavior change in ma.masked_values(shrink=True) - #11987: BUG: fix refcount leak in PyArray_AdaptFlexibleDType - #11995: TST: Add Python 3.7 testing to NumPy 1.14. Cheers, Charles Harris

1 0

NumPy 1.15.2 released
by Charles R Harris Sept. 23, 2018

Sept. 23, 2018

Hi All, On behalf of the NumPy team, I am pleased to announce the release of NumPy 1.15.2. This is a bug fix release for bugs and regressions reported following the 1.15.2 release. The fixes are: - the matrix PendingDeprecationWarning is now suppressed in pytest 3.8, - the new cached allocations machinery has been fixed to be thread safe, - the boolean indexing of subclasses now works correctly, - a small memory leak in PyArray_AdaptFlexibleDType has been fixed. The Python versions supported by this release are 2.7, 3.4-3.7. The wheels are linked with OpenBLAS v0.3.0, which should fix some of the linalg problems reported for NumPy 1.14. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.15.2/>, source archives are available from Github <https://github.com/numpy/numpy/releases/tag/v1.15.2> Compatibility Note ================== The NumPy 1.15.x OS X wheels released on PyPI no longer contain 32-bit binaries. That will also be the case in future releases. See `#11625 <https://github.com/numpy/numpy/issues/11625>`__ for the related discussion. Those needing 32-bit support should look elsewhere or build from source. Contributors ============ A total of 4 people contributed to this release. People with a "+" by their names contributed a patch for the first time. * Charles Harris * Julian Taylor * Marten van Kerkwijk * Matti Picus Pull requests merged ==================== A total of 4 pull requests were merged for this release. * `#11902 <https://github.com/numpy/numpy/pull/11902>`__: BUG: Fix matrix PendingDeprecationWarning suppression for pytest... * `#11981 <https://github.com/numpy/numpy/pull/11981>`__: BUG: fix cached allocations without the GIL for 1.15.x * `#11982 <https://github.com/numpy/numpy/pull/11982>`__: BUG: fix refcount leak in PyArray_AdaptFlexibleDType * `#11992 <https://github.com/numpy/numpy/pull/11992>`__: BUG: Ensure boolean indexing of subclasses sets base correctly. Cheers, Charles Harris

1 0

Dormand-Prince 8(5,3) in new SciPy IVP integrators
by Juan Luis Cano Sept. 20, 2018

Sept. 20, 2018

Hi all, We wanted to use the new initial value problem solvers present in SciPy but the DOP853 method was not present there, so we implemented it ourselves: https://github.com/poliastro/poliastro/blob/v0.11b1/src/poliastro/integrato… We have done some soft validation (i.e. it has worked well so far for our specific use case) and we think it would be great to have it in SciPy, given that it was present in the "Old API" and that it's a widely used method in Celestial Mechanics. If I understood correctly, in-depth quantitative validation was not written for the current solvers: https://github.com/scipy/scipy/pull/6326#issuecomment-250830082 Would you be interested in a PR? For testing, would it be enough to add the new method here? https://github.com/scipy/scipy/blob/4d16aea1267898e8532f962f0104d485578ab67… Regards, -- Juan Luis Cano

3 3

welcome Kai to the core team
by Ralf Gommers Sept. 18, 2018

Sept. 18, 2018

Hi all, On behalf of the SciPy developers I'd like to welcome Kai Striega as a member of the core dev team. Kai has been contributing to scipy.optimize since June, and has so far made a number of improvements to the simplex method of linprog that fixed all of the many known bugs - https://github.com/scipy/scipy/pulls/Kai-Striega I'm looking forward to his continued contributions! Cheers, Ralf

7 6

Regarding presolve methods used in scipy's linear programming implementation
by Touqir Sajed Sept. 18, 2018

Sept. 18, 2018

This question is regarding how well the scipy's linear programming's presolve implementation method works for sparse matrices using scipy.sparse.linalg.spsolve compared to scipy.linalg.lu_solve. For reference : line 322 and 201 in https://github.com/scipy/scipy/blob/master/scipy/optimize/_remove_redundanc… <https://ml-trckr.com/link/https%3A%2F%2Fgithub.com%2Fscipy%2Fscipy%2Fblob%2…> . scipy.sparse.linalg.spsolve uses the sparse linear system solver from scikit-umfpack (at the moment I am not interested in the SuperLU library) and have looked into several academic papers regarding the approaches used in scikit-umfpack which are Unsymmetric MultiFrontal methods but none of them has analyzed the runtime complexity. Hence it would be useful to know its empirical performance in comparison to an existing non-sparse linear system solver implementation. I asked Matt and wanted to know if anyone else has experimented with these. Thanks! -- Computing Science Master's student at University of Alberta, Canada, specializing in Machine Learning. Website : https://ca.linkedin.com/in/touqir-sajed-6a95b1126 <https://ml-trckr.com/link/https%3A%2F%2Fca.linkedin.com%2Fin%2Ftouqir-sajed…>

2 1

Intel's Math Kernel Library (MKL) - MKL DNN
by rlucente＠pipeline.com Sept. 17, 2018

Sept. 17, 2018

I think that most people on this mailing list are aware of Intel’s Math Kernel Library (MKL) https://software.intel.com/en-us/mkl Since neural networks are now all the rage, they now have MKL-DNN https://software.intel.com/en-us/articles/intel-mkl-dnn-part-1-library-over… They actually did a presentation at O’Reilly’s Strata https://www.oreilly.com/ideas/accelerating-ai-on-xeon-through-sw-optimizati… Thought that people on this mailing list might be interested in it

1 0