Mailman 3 June 2019 - SciPy-Dev

NumPy 1.17.0rc1 released
by Charles R Harris 27 Jul '19

27 Jul '19

Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.17.0rc1. The 1.17 release contains a number of new features that should substantially improve its performance and usefulness. The Python versions supported are 3.5-3.7, note that Python 2.7 has been dropped. Python 3.8b1 should work with the released source packages, but there are no guarantees about future releases. Highlights of this release are: - A new extensible random module along with four selectable random numbe5 generators and improved seeding designed for use in parallel processes has been added. The currently available bit generators are MT19937, PCG64, Philox, and SFC64. - NumPy's FFT implementation was changed from fftpack to pocketfft, resulting in faster, more accurate transforms and better handling of datasets of prime length. - New radix sort and timsort sorting methods. It is currently not possible to choose which will be used, but they are hardwired to the datatype and used when either ``stable`` or ``mergesort`` is passed as the method. - Overriding numpy functions is now possible by default Downstream developers should use Cython >= 0.29.10 for Python 3.8 support and OpenBLAS >= 3.7 (not currently out) to avoid problems on the Skylake architecture. The NumPy wheels on PyPI are built from the OpenBLAS development branch in order to avoid those problems. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.17.0rc1/>, source archives and release notes are available from Github <https://github.com/numpy/numpy/releases/tag/v1.17.0rc1>. *Contributors* A total of 142 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Aaron Voelker + - Abdur Rehman + - Abdur-Rahmaan Janhangeer + - Abhinav Sagar + - Adam J. Stewart + - Adam Orr + - Albert Thomas + - Alex Watt + - Alexander Blinne + - Alexander Shadchin - Allan Haldane - Ander Ustarroz + - Andras Deak - Andreas Schwab - Andrew Naguib + - Andy Scholand + - Ankit Shukla + - Anthony Sottile - Antoine Pitrou - Antony Lee - Arcesio Castaneda Medina + - Assem + - Bernardt Duvenhage + - Bharat Raghunathan + - Bharat123rox + - Bran + - Bruce Merry + - Charles Harris - Chirag Nighut + - Christoph Gohlke - Christopher Whelan + - Chuanzhu Xu + - Daniel Hrisca - Daniel Lawrence + - Debsankha Manik + - Dennis Zollo + - Dieter Werthmüller + - Dominic Jack + - EelcoPeacs + - Eric Larson - Eric Wieser - Fabrice Fontaine + - Gary Gurlaskie + - Gregory Lee + - Gregory R. Lee - Hameer Abbasi - Haoyu Sun + - He Jia + - Hunter Damron + - Ian Sanders + - Ilja + - Isaac Virshup + - Isaiah Norton + - Jaime Fernandez - Jakub Wilk - Jan S. (Milania1) + - Jarrod Millman - Javier Dehesa + - Jeremy Lay + - Jim Turner + - Jingbei Li + - Joachim Hereth + - John Belmonte + - John Kirkham - John Law + - Jonas Jensen - Joseph Fox-Rabinovitz - Joseph Martinot-Lagarde - Josh Wilson - Juan Luis Cano Rodríguez - Julian Taylor - Jérémie du Boisberranger + - Kai Striega + - Katharine Hyatt + - Kevin Sheppard - Kexuan Sun - Kiko Correoso + - Kriti Singh + - Lars Grueter + - Maksim Shabunin + - Manvi07 + - Mark Harfouche - Marten van Kerkwijk - Martin Reinecke + - Matthew Brett - Matthias Bussonnier - Matti Picus - Michel Fruchart + - Mike Lui + - Mike Taves + - Min ho Kim + - Mircea Akos Bruma - Nick Minkyu Lee - Nick Papior - Nick R. Papior + - Nicola Soranzo + - Nimish Telang + - OBATA Akio + - Oleksandr Pavlyk - Ori Broda + - Paul Ivanov - Pauli Virtanen - Peter Andreas Entschev + - Peter Bell + - Pierre de Buyl - Piyush Jaipuriayar + - Prithvi MK + - Raghuveer Devulapalli + - Ralf Gommers - Richard Harris + - Rishabh Chakrabarti + - Riya Sharma + - Robert Kern - Roman Yurchak - Ryan Levy + - Sebastian Berg - Sergei Lebedev + - Shekhar Prasad Rajak + - Stefan van der Walt - Stephan Hoyer - SuryaChand P + - Søren Rasmussen + - Thibault Hallouin + - Thomas A Caswell - Tobias Uelwer + - Tony LaTorre + - Toshiki Kataoka - Tyler Moncur + - Tyler Reddy - Valentin Haenel - Vrinda Narayan + - Warren Weckesser - Weitang Li - Wojtek Ruszczewski - Yu Feng - Yu Kobayashi + - Yury Kirienko + - @aashuli + - @euronion + - @luzpaz - @parul + - @spacescientist + Cheers, Charles Harris

1 2

Combinatorial optimization in scipy.optimize
by Søren Fuglede Jørgensen 23 Jul '19

23 Jul '19

Hi SciPy-Dev This question is motivated by the recent improvements to scipy.optimize.linear_sum_assignment. [0] Here, we get a much improved algorithm for solving minimum weight perfect matching in bipartite graphs (which is excellent). Now, this problem is one in a large family of combinatorial optimization problems on packing and covering, and what I'm wondering is if SciPy should be the home for a general class of efficient algorithms for solving those? Most of these can be phrased in terms of graph theory, and indeed networkx already contains implementations of many classical algorithms (and I was surprised to find that it didn't have anything matching linear_sum_assignment directly; it does have an implementation of the Blossom algorithm). So there's some overlap of scope here, but I guess that's not a problem in and by itself, but it had me wondering where the split should be? Just to phrase this in terms of (a very incomplete list of) examples (some of which have polynomial solutions, some of which don't), would there be appetite for including into scipy.optimize algorithms for solving e.g. 1. other covering/packing problems (e.g. min set/vertex/edge cover, maximum matching, ...), 2. maximum flow, 3. shortest path, 4. travelling salesman, 5. circuit minimization? Søren [0]: https://github.com/scipy/scipy/pull/10296

5 6

Enabling use of scipy.special from Numba
by Joshua Wilson 08 Jul '19

08 Jul '19

Hey everyone, For a while I've been thinking about how we can improve the compatibility of `scipy.special` with Numba. There is a way to do that currently by using `cython_special`: https://github.com/numba/numba/issues/3086 but that has a few drawbacks: (1) it's not very user friendly (2) it relies on the (I think?) private implementation details of Cython's `__pyx_capi__` (including the name mangling scheme used for fused functions) (3) it's clumsy for complex functions I think (3) requires changes on the Numba side, e.g. https://github.com/numba/numba/issues/3860 For (1) you could imagine us implementing another Numba specific API, but that runs into some immediate issues: - We'd need Numba as a dependency (which has been discussed: https://mail.python.org/pipermail/scipy-dev/2018-March/022576.html) - We don't want to just keep implementing new APIs For those reasons I'd prefer to find another solution. In an ideal world I'd like to solve (2) in such a way that it's easy for a third-party library to wrap `cython_special` in Numba jitted functions. There is a solution to that already that doesn't involve `__pyx_capi__`: a third-party library writes some Cython glue that exports PyCapsules of function pointers to the specializations of all the `cython_special` functions and then you grab those using ctypes to construct the jitted functions. For the Cython side of that see the docs on specialization: http://docs.cython.org/en/latest/src/userguide/fusedtypes.html#selecting-sp… But that feels rather convoluted. In an ideal world it would be nice to have an official way to grab the PyCapsules out of `__pyx_capi__` instead of writing a new set of capsules yourself. Note that we run into a similar problem with our own `LowLevelCallable`: https://docs.scipy.org/doc/scipy/reference/generated/scipy.LowLevelCallable… Getting the required name of the exported function for a fused function requires searching through the mangled names for the one you want. Out of all that, my main questions are: - Maybe I'm missing something obvious? Do people have better ideas for improving compatibility? - If not, would it be crazy to talk to the Cython folks about an official method for getting the PyCapsules out of `__pyx_capi__`? - Josh

6 16

scipy.fft submodule
by Peter Bell 08 Jul '19

08 Jul '19

Following on from some of the discussion surrounding my backend proposal and the discussion on gh-10175<https://github.com/scipy/scipy/issues/10175>; I've been working on adding a new scipy.fft submodule. This is now working, tested, documented and ready for review in gh-10238<https://github.com/scipy/scipy/pull/10238>. The new submodule is almost a drop-in replacement for numpy.fft and scipy.fftpack, with a few exceptions: * Does not include NumPy's Hermitian transforms (hfft, ihfft) (but would be simple to add) * Uses numpy's conventions for rfft (complex array) instead of fftpack's (complex values packed into a real array) * Convolutions and pseudo-differential operators from fftpack are not included The new submodule adds pocketfft to implement the normal FFTs (not yet DCT and DST) which adds several new features beyond scipy.fftpack: * long double transforms * multi-dimensional real FFTs (rfftn) * Orthonormal transforms (norm='ortho' argument) * Bluestein's algorithm to avoid the worst case O(n^2) complexity of FFTPACK A few implementation details worth noting: * Unlike NumPy's version of pocketfft, this is C++ and uses templates to supporting the different floating point types * pocketfft also uses pybind11 which both adds a new build-time dependency and requires C++11. I believe this is the first use of C++11 in SciPy. * pocketfft has optional support for multithreading using OpenMP but this is currently disabled (and not compiled in at all). And finally, there is the rather awkward issue that scipy.fft already exists and is an alias for the numpy.fft.fft function. Currently I've had to add a workaround to allow the module scipy.fft to be callable, as discussed in gh-10253<https://github.com/scipy/scipy/issues/10253>. This is only relevant to code that imports scipy.fft so I don't expect it to be used often, if at all. Hopefully this is a temporary solution and the NumPy functions can be removed from the scipy namespace in some future release, or at the very least the FFT functions can be removed. Peter

4 5

Sprint at SciPy Conference?
by Matt Haberland 05 Jul '19

05 Jul '19

The SciPy Conference <https://www.scipy2019.scipy.org/> approaches! Who would be interested in a sprint <https://www.scipy2019.scipy.org/sprints>? Can you participate in-person or virtually? We don't necessarily need a theme, but would you like to recruit help for something in particular? If you already registered a sprint on behalf of the SciPy library, please let me know. Otherwise, I can register (and I'll ask to be located near NumPy so people don't have to choose one or the other). Matt

3 5

Fwd: 2-factor auth on PyPI & NumPy projects
by Charles R Harris 28 Jun '19

28 Jun '19

Forwarding a message from PyPI. I looked at the two factor login a while ago and decided it wasn't for me, but others might be interested. ---------- Forwarded message --------- From: Sumana Harihareswara <sh(a)changeset.nyc> Date: Thu, Jun 27, 2019 at 4:11 PM Subject: 2-factor auth on PyPI & NumPy projects To: Charles R Harris <charlesr.harris(a)gmail.com> Dear Mr. Harris: Hi! I found you via the NumPy discussion mailing list, where I saw you are part of the release team. I'm the project manager for PyPI. Right now we're beta testing a new security feature on PyPI: https://pyfound.blogspot.com/2019/06/pypi-now-supports-two-factor-login-via… 2-factor auth for website login with WebAuthn (hardware devices like Yubikeys). And during this beta, we'd love more testing from package maintainers who use a variety of operating systems, browsers, and browser plugins, including on mobile. There's more info at https://wiki.python.org/psf/WarehousePackageMaintainerTesting . And within the next few months we'll be adding support for API keys for uploading releases to PyPI https://github.com/pypa/warehouse/issues/994 , which should make release automation easier. Could you possibly pass this information on to the NumPy developers' list or another relevant group? Thanks, Sumana -- Sumana Harihareswara PyPI project manager Changeset Consulting https://changeset.nyc

2 1

Re: [SciPy-Dev] Contributing to SciPy's Documentation [Google Season Of Docs '19]
by Harivallabha Rangarajan 28 Jun '19

28 Jun '19

Hi Christoph, Thanks for the reply! I've attached the proposal for the Google Season of Docs below. I'd be glad to incorporate your suggestions, and I'll submit it if this looks good to you :) Thanks! Yours sincerely, Hari

1 0

Re: [SciPy-Dev] Contributing to SciPy's Documentation [Google Season Of Docs '19]
by Christoph Baumgarten 26 Jun '19

26 Jun '19

Hi Harivallabha, thanks for your interest to contribute to the documentation of scipy.stats. I think that writing a comprehensive tutorial is a good idea. You probably need to describe the proposal in a more concrete way. Do you have a tutorial in mind what you would be interested to work on? The application deadline is already on June 28th: https://developers.google.com/season-of-docs/docs/timeline Further information on the application process: https://developers.google.com/season-of-docs/docs/tech-writer-application-h… Christoph On Sun, Jun 23, 2019 at 6:03 PM <scipy-dev-request(a)python.org> wrote: > Send SciPy-Dev mailing list submissions to > scipy-dev(a)python.org > > To subscribe or unsubscribe via the World Wide Web, visit > https://mail.python.org/mailman/listinfo/scipy-dev > or, via email, send a message with subject or body 'help' to > scipy-dev-request(a)python.org > > You can reach the person managing the list at > scipy-dev-owner(a)python.org > > When replying, please edit your Subject line so it is more specific > than "Re: Contents of SciPy-Dev digest..." > > > Today's Topics: > > 1. Sprint at SciPy Conference? (Matt Haberland) > 2. Contributing to SciPy's Documentation [Google Season Of Docs > '19] (Harivallabha Rangarajan) > > > ---------------------------------------------------------------------- > > Message: 1 > Date: Sat, 22 Jun 2019 22:40:47 -0700 > From: Matt Haberland <haberland(a)ucla.edu> > To: SciPy Developers List <scipy-dev(a)python.org> > Subject: [SciPy-Dev] Sprint at SciPy Conference? > Message-ID: > <CADuxUixr=xqb18-fRduN7Hw40ct1yv4UsxLLm+= > yET5AEkUyHQ(a)mail.gmail.com> > Content-Type: text/plain; charset="utf-8" > > The SciPy Conference <https://www.scipy2019.scipy.org/> approaches! Who > would be interested in a sprint <https://www.scipy2019.scipy.org/sprints>? > Can you participate in-person or virtually? > > We don't necessarily need a theme, but would you like to recruit help for > something in particular? > > If you already registered a sprint on behalf of the SciPy library, please > let me know. Otherwise, I can register (and I'll ask to be located near > NumPy so people don't have to choose one or the other). > > Matt >

1 0

Overlap-add convolution implementation
by Todd 25 Jun '19

25 Jun '19

Hi all, I am currently working an an implementation of overlap-add convolution [0]. I have a 1D implementation working and I am going to start expanding it to ND. Overlap-add convolution can provide significant performance benefits over FFT-based convolution. Does this make sense for scipy? Assuming it is, I have a couple questions. First is the name. We have "convolve" and "fftconvolve" already. A few options: overall_add_conv oaconvolve oadconvolve Second is how to calculate the optimal chunk size for the convolution. The equation for the number of complex convolutions, according to wikipedia, is: (Nx/(N-M+1))*N*(log2(N)+1) Where Nx is the signal length, M is the filter length, and N is the chunk size. I need to find the minimum of that equation. Nx is a constant scaling factor so I get rid of that. But I still can't find a closed-form solution to the equation. Assuming there is no closed-form solution, I need another approach. What I am doing now is calculating the minimum across all possible power of 2 indices (I calculated the log of the ends, do a linear range, then convert that back to linear steps to avoid doing logarithms on every step). However, this means I miss the fast powers of 3 and 5. I could do the minimum for all powers, but that would take longer. What would be a good trade-off? [0] https://en.wikipedia.org/wiki/Overlap%E2%80%93add_method

4 4

Contributing to SciPy's Documentation [Google Season Of Docs '19]
by Harivallabha Rangarajan 23 Jun '19

23 Jun '19

Hi, I’m Harivallabha Rangarajan, a 3rd year undergraduate student pursuing *B.E. Computer Science* and *M.Sc. Mathematics*, with a *minor* in* Data Science*, at the *Birla Institute of Technology and Science (BITS) Pilani, Hyderabad, *one of India's premier research institutes. I am *deeply interested* in contributing to the *GSoD, 2019* project on Improving the documentation of the scipy.stats module*.* Right now, I'm participating in the *Google Summer of Code, '19 *with *PyMC3* under the NumFOCUS umbrella wherein my project is on implementing the Dirichlet Process module. [Proposal: Dirichlet Processes - GSoC, 2019 <https://drive.google.com/open?id=1voujFrb26W6OuvW7fs5qqE0VAAqWSw9R>] Therefore, I'm quite familiar with the open-source workflow. I'd be writing tests, documentation, blog posts and a final report for the module implemented - so by the end of the GSoC period, I'd be comfortable with technical writing as well. I have extensively used the scipy.stats module (and am using it for some writing some checks for distributions in PyMC3), so I hope I'd be able to contribute effectively to improving its documentation :) The API references and the documentation of the scipy.stats module, as is, is quite useful and straightforward. However, the module's documentation could be augmented with more extensive tutorials, and links to existing implementations using scipy.stats. By extensive tutorials, I mean notebooks such as these: https://docs.pymc.io/nb_examples/index.html As for adding links to existing implementations: The Dirichlet distribution documentation could direct users to something like https://dp.tdhopper.com/dirichlet-distribution/ In addition (as an ambitious goal, or as a nice-to-have) we could add some programming crestomathies and comparisons: Implementation of, say, a hierarchical regression model in scipy and pymc3 or stan, and compare them. I trust that I have the necessary statistics background to both "improve the tutorial site of the stats module" and "improve the documentation of the reference guide" - the two points mentioned in the SciPy - GSoD project ideas page. The GSoC coding period ends on Aug, 26. After that, I would be able to devote 5-6 hours a day for GSoD. I see this as an opportunity to both contribute to the SciPy docs as well as familiarize myself quite well with the scipy.stats codebase, which would help me contribute to SciPy beyond the GSoD period (in the same manner GSoC enabled me to acquaint myself with the PyMC3 codebase. I will be contributing to PyMC3 well beyond the GSoC period). Besides that, I'm currently working on a thesis project titled *Investigation of Learning Techniques for Regression Problems* with *Dr. Emre Ozkaya*, *Chair of Scientific Computing, TU - Kaiserslautern, *and* Dr. Anil Nemili, Department of Mathematics, BITS Pilani - Hyderabad. * Previously, I had worked on: *Performance Analysis of Algorithmic Differentiation based Adjoint Solvers* under *Dr. Anil Nemili.* I had designed and developed a *Scalable and Robust Journal Recommender System based on Stochastic Gradient Descent (SGD) and Singular Value Decomposition (SVD)*, implemented in Python, during my summer internship (2018) at* Indira Gandhi Centre for Atomic Research (IGCAR)*, Kalpakkam, India under the guidance of *Dr. N. Madurai Meenachi.* My coursework includes Partial Differential Equations, Functional Analysis, Numerical Methods, Optimization, Linear Algebra, Differential Geometry, Ordinary Differential Equations, Multivariable Calculus, Probability and Statistics, Complex Numbers, Real Analysis, Introduction to Topology, Graphs and Networks, Mathematical Methods, Discrete Mathematics, Operations Research, Data Structures and Algorithms, Database Management Systems, Object-Oriented Programming, Microprocessors and Interfacing, Digital Design, and Logic in Computer Science. I have a strong grasp of Data Structures and Algorithms, having implemented them in C++, for Competitive Coding. I’m interested in developing efficient and scalable computational methods for space and time optimizations in Machine Learning. *I would be extremely grateful if you could guide me towards getting started with the contributions, and continuing onward to participate in GSoD '19 with SciPy.* Looking forward to a positive response! Thank you for your time :) Yours sincerely, Harivallabha Rangarajan, BITS Pilani | Hyderabad Campus

1 0