Mailman 3 August 2017 - SciPy-Dev

Are you using Bento to build scipy?
by CJ Carey 16 Jul '18

16 Jul '18

With the 1.0 release coming up, I figured this is a good time to take the pulse of the community regarding Bento. The project seems to have been abandoned before reaching stability, and it seems that the core problems it was trying to address have been at least partially fixed by the more official build systems. If you use Bento and would like to keep using it post 1.0, or if my previous sentence is wrong, please let me know!

10 15

establishing a Code of Conduct for SciPy
by Ralf Gommers 24 Oct '17

24 Oct '17

Hi all, I propose that we as SciPy developers and community adopt a Code of Conduct. As you probably know, Code of Conduct (CoC) documents are becoming more common every year for open source projects, and there are a number of good reasons to adopt a CoC: 1. It gives us the opportunity to explicitly express the values and behaviors we'd like to see in our community. 2. It is designed to make everyone feel welcome (and while I think we're a welcoming community anyway, not having a CoC may look explicitly unwelcoming to some potential contributors nowadays). 3. It gives us a tool to address a set of problems if and when they occur, as well as a way for anyone to report issues or behavior that is unacceptable to them (much better than having those people potentially leave the community). 4. SciPy is not yet a fiscally sponsored project of NumFOCUS, however I think we'd like to be in the near future. NumFOCUS has started to require having a CoC as a prerequisite for new projects joining it. The PSF has the same requirement for any sponsorship for events/projects that it gives. Also note that GitHub has starting checking the presence of a CoC fairly prominently (https://github.com/scipy/scipy/community), and has also produced a guide with things to think about when formulating a CoC: https://opensource.guide/code-of-conduct/. I recommend reading that guide (as well as others guides on that site), it's really good. To get to a CoC document, a good approach is to borrow text from a CoC that has been in use for a while and has proven to be valuable, and then modify where needed (similar to a software license - don't invent your own). I considered three existing CoC's: - The Contributor Covenant (http://contributor-covenant.org/version/1/2/0/): simple, concise, the most widely used one. The NumFOCUS recommended one is based on it as well (https://www.numfocus.org/about/code-of-conduct/). - The Python Community Code of Conduct ( https://www.python.org/psf/codeofconduct/): also simple, addresses mostly the spirit in which the Python community is operating / should operate. - The Jupyter Code of Conduct ( https://github.com/jupyter/governance/tree/master/conduct): much more detailed, in part derived from the Speak up! and Django ones, more appropriate for large communities. I think the Python Community CoC isn't a good basis, it's more a statement of intent for a wider community while it's missing too many elements that a project CoC should have. That leaves the choice between the short and simple Contributor Covenant, and the more extensive Jupyter one. Personally I like the tone of the Jupyter one *much* more, so I have started from that one and made the following modifications to make it fit SciPy better: 1. Removed the part about reporting during events (we don't organize those) and removed the language about events from the faq. 2. Changes to reporting options: removed the form (email should be enough), and added reporting to NumFOCUS as a second option. 3. Changes to enforcement manual: reply within 72 hours rather than 24 hours (we don't have paid work on SciPy, so 24 hours is not very realistic). Changed the committee from 5 to 3 members (5 is a lot, and we're significantly smaller than Jupyter/Django). Here is a WIP PR with the CoC content: https://github.com/scipy/scipy/pull/7764. I suggest to bring up any larger questions/issues here, and detailed textual comments on the PR. Once the content is agreed upon I will change it to reST and integrate it with the rest of the docs. Thoughts? Volunteers for the committee? Cheers, Ralf

12 46

SciPy 1.0 release schedule proposal
by Ralf Gommers 16 Sep '17

16 Sep '17

Hi all, We're now at 5 months after the 0.19.0 release, and well behind our planned 1.0 release date [1]. So it's high time for a more concrete plan - here it is: Sep 16: Branch 1.0.x Sep 17: beta1 Sep 27: rc1 Oct 7: rc2 Oct 17: final release This gives us one month to finish and merge everything that we really would like to see in 1.0 - should be enough time for things that are already in progress. The list of things now marked for 1.0 [2] is not too long, and not all of it is critical. It would be useful if everyone could add PRs/issues that they think are critical to the 1.0 milestone. Adding yourself as an "assignee" on PRs/issues you plan to tackle would also be helpful. Besides some really nice new features, we made major strides in infrastructure (CI, testing) and project organisation for this release. Fom my perspective there are three critical things left: 1. Windows wheels. Looks like we're pretty much there, but we need to ensure that 1.0 is the first release that has Windows wheels available. 2. Adding a code of conduct. This is the last thing we imho need from an "open source project maturity" perspective (an FSA would be nice, but can wait). 3. Merging a lot of PRs. There's ~150 PRs open now, I hope in the next month we can focus on reducing that number and getting some nice improvements in that have been waiting for quite some time. Finally, we've discussed before writing a paper about SciPy to coincide with this release. I'll follow up on that separately. Thoughts? Cheers, Ralf [1] https://mail.python.org/pipermail/scipy-dev/2016-September/021485.html [2] https://github.com/scipy/scipy/milestones/1.0

5 10

Clarkson Woodruff Transform implementation (Randomized Numerical Linear Algebra)
by Jordi Montes 13 Sep '17

13 Sep '17

Hello, I just made a pull request to the official repository with the implementation of the Clarkson Woodruff Transform (Count Min Sketch) for dense matrices. You can find more details of the pull request here <https://github.com/scipy/scipy/pull/7809>. I would not like to made it badly (this is my pull request to the project) so any advice is welcome (Main implementation/tests/documentation). Jordi Montes.

2 1

Proposed API change for some functions in scipy.signal
by Warren Weckesser 02 Sep '17

02 Sep '17

I've been using quite a few of the functions in scipy.signal recently, and I've been reminded of some of the quirks of the API. I have submitted a pull request to clean up one of those quirks. In the new pull request, I have added the argument 'fs' (the sampling frequency) to the following functions: firwin, firwin2, firls, and remez. For firwin, firwin2 and firls, the new argument replaces 'nyq', and for remez, it replaces 'Hz'. This makes these functions consistent with the other functions in which the sampling frequency is specified using 'fs': periodogram, welch, csd, coherence, spectrogram, stft, and istft. 'fs', or in LaTeX, $f_s$, is common notation for the sampling frequency in the DSP literature. In the pull request, the old argument is given a "soft" deprecation. That means the docstring says the argument is deprecated, but code to actually generate a DeprecationWarning has not been added yet. I'm fine with adding that now, but some might prefer a relatively long and soft transition for these changes. (Personally, I don't mind if they hang around for a while, but they should be gone by 2.0. :) I haven't changed the default value of the sampling frequency. For the FIR filter design functions firls, firwin and firwin2, the default is nyq=1.0 (equivalent to fs=2), while for remez the default is Hz=1 (i.e. fs=1). The functions that currently already use 'fs' all have the default fs=1. Changing the default for the FIR design functions would be a much more disruptive change. Comments here or on the pull request are welcome. P.S. I can see future pull requests in which 'fs' is added to functions that currently don't have an argument to specify the sampling frequency. I'm looking at you, freqz.

5 6

N dimensional Gaussian quadrature
by oss 02 Sep '17

02 Sep '17

Hi scipy-dev, We ve implemented and tested n dimensional fixed_quad function with choice of orders for each dimension separately (useful when known singular points need to be avoided). See the diff below. We re open for any request for improvement. https://github.com/scipy/scipy/compare/master...multiwave:n_fixed_quad <https://github.com/scipy/scipy/compare/master...multiwave:n_fixed_quad> Thanks tryfon@multiwave

1 1

scikit-procrustes
by Melissa Weber Mendonça 30 Aug '17

30 Aug '17

Hello everybody, My name is Melissa, I'm a professor and researcher at the Federal University of Santa Catarina, in Brazil. I'm an applied mathematician working with Numerical Optimization and I've been working for some time with Procrustes Problems. So I realized there was no implementation of methods to solve the unbalanced Procrustes problem in python, and I decided to build a scikit. Here it is: - https://github.com/melissawm/skprocrustes http://scikits.scipy.org/scikit-procrustes All are welcome to criticize and to send pull requests if you feel you have something to contribute, I'd appreciate it. I hope this is useful for someone. Cheers! Melissa p.s. I'm not an expert in packaging and so it's highly possible that my code is not optimal, if you have any suggestions I'd appreciate as I'm trying to get better at this :)

3 3

Re: [SciPy-Dev] Improving performance of sparse matrix multiplication
by Paul Nation 29 Aug '17

29 Aug '17

It is possible to write your own openmp implimentation, performing the row multiplications in parallel, using Cython or whatever low language you like. Since sparse matrix dense vector multiplication is memory bandwidth limited, you will see good performance if the problem fits into CPU cache. Otherwise, you will see only marginal improvements. If your problem has structure, then that should be exploited to make things faster. Also permutations like reverse Cuthill-Mckee, in SciPy, can give a modest boost. Best, Paul On Aug 29, 2017 04:26, "Michele Martone" < michelemartone(a)users.sourceforge.net> wrote: Dear Marc, did you try the PyRSB prototype: "librsb is a high performance sparse matrix library implementing the Recursive Sparse Blocks format, which is especially well suited for multiplications in iterative methods on huge sparse matrices. PyRSB is a Cython-based Python interface to librsb." https://github.com/michelemartone/pyrsb ? How large are your matrices ? Are they symmetric ? If your matrices are large you might get quite of a speedup; if symmetric, even better. Best regards, Michele p.s.: PyRSB (a thin interface) is a prototype, but librsb itself http://librsb.sourceforge.net/ is in a mature state and usable also from Fortran, and OpenMP based. On 20170828@18:14, marc wrote: > Dear Scipy developers, > > We are developing a program that perform a large number of sparse matrix > multiplications. We recently wrote a Python version of this program for > several reasons (the original code is in Fortran). > > We are trying now to improve the performance of the Python version and we > noticed that one of the bottlenecks are the sparse matrix multiplications, > as example, > > import numpy as np > from scipy.sparse import csr_matrix > > row = np.array([0, 0, 1, 2, 2, 2]) > col = np.array([0, 2, 2, 0, 1, 2]) > data = np.array([1, 2, 3, 4, 5, 6], dtype=np.float32) > > csr = csr_matrix((data, (row, col)), shape=(3, 3)) > print(csr.toarray()) > > A = np.array([1, 2, 3], dtype=np.float32) > > print(csr*A) > > > I started to look at the Scipy code to see how this functions were > implemented, and realized that there is no openmp parallelization over the > for loops. Like in function csr_matvec in sparse/sparsetools/csr.h (line > 1120). Is it possible to parallelize this loops with openmp? Do you have > maybe better ideas to improve the performances for this kind of operations? > > Best regards, > Marc Barbry > _______________________________________________ > SciPy-Dev mailing list > SciPy-Dev(a)python.org > https://mail.python.org/mailman/listinfo/scipy-dev _______________________________________________ SciPy-Dev mailing list SciPy-Dev(a)python.org https://mail.python.org/mailman/listinfo/scipy-dev

1 0

GSoC 2017 - Large-scale Constrained Optimization
by Antonio Ribeiro 29 Aug '17

29 Aug '17

Hi all, During the last few months I have worked on my Google Summer of Code (GSoC) project, that consists of implementing a large-scale optimization algorithm to be integrated to Scipy. The algorithm implemented was an interior point method described in <https://antonior92.github.io/posts/2017/07/interior-point-method/ <https://antonior92.github.io/posts/2017/07/interior-point-method/>>. A series of blog post describe different aspects of the algorithm and its use <https://antonior92.github.io/tags/#gsoc-2017 <https://antonior92.github.io/tags/#gsoc-2017>>. The implementation can be found on the separate repository: <https://github.com/antonior92/ip-nonlinear-solver <https://github.com/antonior92/ip-nonlinear-solver>> and is being integrated to SciPy through the pull request #7729 <https://github.com/scipy/scipy/pull/7729 <https://github.com/scipy/scipy/pull/7729>> During my GSoC I have implemented the optimization algorithm, tested it on almost one hundred examples <https://antonior92.github.io/posts/2017/00/NumericalResults/ <https://antonior92.github.io/posts/2017/00/NumericalResults/>> and created an interface for using it. While the optimization solver is ready to be used and tested, there are still a features I want to include, namely quasi-Newton approximations to the Hessian matrix including: SR1 approximation; BFGS approximation; and, L-BFGS approximation. I included those as optional items on my GSoC proposal and, unfortunately, they will not be ready for the GSoC submission. Furthermore, there are still some questions about how to best integrate my implementation to the optimization SciPy library that are still under discussion. However, I will tend to those final points in the weeks following the end of the program. Finally, I would like to thanks SciPy community and my mentors: Nikolay, Matt and Ralf, with whom I have greatly enjoyed the opportunity to work with during these three months. Antônio H. Ribeiro

4 3

Improving performance of sparse matrix multiplication
by marc 29 Aug '17

29 Aug '17

Dear Scipy developers, We are developing a program that perform a large number of sparse matrix multiplications. We recently wrote a Python version of this program for several reasons (the original code is in Fortran). We are trying now to improve the performance of the Python version and we noticed that one of the bottlenecks are the sparse matrix multiplications, as example, import numpy as np from scipy.sparse import csr_matrix row = np.array([0, 0, 1, 2, 2, 2]) col = np.array([0, 2, 2, 0, 1, 2]) data = np.array([1, 2, 3, 4, 5, 6], dtype=np.float32) csr = csr_matrix((data, (row, col)), shape=(3, 3)) print(csr.toarray()) A = np.array([1, 2, 3], dtype=np.float32) print(csr*A) I started to look at the Scipy code to see how this functions were implemented, and realized that there is no openmp parallelization over the for loops. Like in function csr_matvec in sparse/sparsetools/csr.h (line 1120). Is it possible to parallelize this loops with openmp? Do you have maybe better ideas to improve the performances for this kind of operations? Best regards, Marc Barbry

3 2