Mailman 3 April 2015 - SciPy-Dev

GSoC'15 accepted students for Scipy/Numpy
by Ralf Gommers 24 May '15

24 May '15

Hi all, Google has just announced which students got accepted for this year's GSoC. For Scipy these are: - Nikolay Mayorov, "Improve nonlinear least squares minimization functionality in SciPy" mentors: Chuck & Evgeni - Abraham Escalante, "SciPy: scipy.stats improvements" mentor: Ralf (Evgeni is backup mentor) Furthermore, this proposal was accepted for Scikit-image: - Aman Singh, "Scikit-Image: rewriting scipy.ndimage to cython" mentors: Jaime, Ralf & the scikit-image devs Congratulations to all of you! We had a lot of interest this year, which is great to see. GSoC applications are competitive, and unfortunately there are students who didn't make it. To those students I would say: please stay involved, and you're very welcome to apply again next year! Today is also the start of the "community bonding period", where the students aren't yet expected to start working on their project but do get time to further figure out how things work, interact with the community and ensure that they can hit the ground running on day 1 of the coding period: http://googlesummerofcode.blogspot.nl/2007/04/so-what-is-this-community-bon…. It looks like it'll be an interesting and productive summer! Cheers, Ralf P.S. all proposals are linked on https://github.com/scipy/scipy/wiki/GSoC-project-ideas#student-applications… for who's interested in the details. P.P.S. some students have asked to get some feedback about why they were/weren't accepted, in order to learn from it for a next time. Until today we weren't allowed to say much, but now that Google has announced the results I'd be happy to give some feedback - please contact me in private if you want.

1 1

0.16.0 release
by Ralf Gommers 23 May '15

23 May '15

Hi all, It's about time to start preparing for the 0.16.0 release. Unusually, there are zero blocking issues it looks like. A lot of PRs to merge of course, but nothing very large that still needs a lot of work. Most important question: does anyone want to take the release manager role for this release? It would be good to start rotating this role for every minor release. This will help spread the knowledge on how to do releases and also spread the workload (it's not that much for a single release, but it adds up). Cheers, Ralf

6 18

Growth and sustainability of Scipy development
by Ralf Gommers 23 Apr '15

23 Apr '15

Hi, Here's a long story about something I've been thinking about for a while. Some individual parts have been discussed with a few people, but the complete story is only how I look at things at the moment. I make some concrete proposals that could be agreed on in this thread if they turn out to be uncontroversial; we can start separate threads as needed though. Summary -------------- *Problem Statement* The growth of core developer and project admin capacity, while pretty strong over the past few years, is not quite keeping up with the growth in contributions we're seeing. *Solution* Identify this as an important issue and start addressing it more actively. *Main Proposals* - recruitment: a "become a core dev" mentoring program - docs: - write "how to become a core developer" documentation - improve the developer guide further - finally merge the Scipy 1.0 roadmap - communication: set up a private core dev mailing list - funding: - we got some donations (excellent, very useful) - have an agreed upon strategy (written down) of what to do with funds - then aim at getting more donations and/or getting grants - governance: we have very little formalized, add something lightweight that covers most bases Long Version ------------------- *Problem Statement* The number of open PRs is very slowly increasing, and stands now at ~110. Some of those PRs are stalled, but many simply need reviewing. The number of active core developers who review and merge PRs is also increasing, but evidently not at the rate that is necessary for PRs to be reviewed quickly. The sometimes long delays in getting a PR merged have likely also discouraged further contributions from some new contributors. Note that we did have a lot of growth in active core developers over the past few years - I remember Fernando Perez pointing out to me at EuroSciPy'11 that the Scipy bus factor was exactly 2, now it's more like 8. *Recruitment* Therefore I'd like to look for new core developers in a more pro-active way. That starts by very clearly saying: We need your help! To be more precise, we need both domain specialists that help review PRs for a single module and core developers that have a broad understanding of how things are done in Scipy and that take responsibility for signing off on the quality of PRs and are merging them independently. A "core dev mentoring program". This can go quite quickly for people that are experienced Python programmers and are familiar with the Scipy community; it may be as little as a few email conversations or Skype calls on how to get started. On the other end of the scale, I'd also like to help people who are new to the Scipy community or even open source development get started - as long as they have the right level of coding/communication skills of course. Then the road to commit rights will of course be longer, but as long as the ambition to get there is present then it's worth the effort (I know I certainly wasn't very experienced in contributing to open source projects at the time I became Scipy release manager, and could definitely have used some help). *Docs* What developer docs are missing or need significant improvements? Let's add those (I volunteer for writing duty). We have a reasonable set (links at end of email), but it's quite scattered. And what we don't have is "how to become a core developer" documentation, let's add that. Plus finally merge the 1.0 roadmap: https://github.com/scipy/scipy/pull/2908. *Communication* I'd like to set up a new core dev closed mailing list. Not with the purpose to do less in public, but simply for organizing the communication that we already have off-list in a very ad-hoc manner now. Typical topics: deciding on commit rights, discussions on ranking GSoC students, potential funding sources and what to do with such funding. I'm not yet sure what to propose for people with access to that list. We currently have 20 people with commit rights, of which about half are actually active. Currently decisions are mostly made between the active devs. *Funding* I'm starting to see more opportunities to use funding in effective ways to sustain or accelerate Scipy development. Main ones are organizing developer meetups / sprints, and targeted funding to talented individuals (students mainly) and for work on particularly hairy bottlenecks (example: Windows test&build). We have a donation page on scipy.org and have received a number of donations, but have not yet actively solicited donations because of (a) no fiscal sponsorship agreement with NumFOCUS signed yet, and (b) while we can use the money now, I'm uncomfortable to spend large amounts without some written down strategy on how and why we spend funds. *Governance* Scipy activity is currently very much centered around code, and other project aspects get relatively little attention. Big email debates about governance are often not very productive (more high-bandwidth communication is necessary here), but it would be helpful to keep making steps in this area. Documenting processes, how decisions are made, who decides on things like commit rights, resolving API discussions, etc. is important. That goes hand in hand with the developer docs mentioned above. The HACKING and MAINTAINERS documents together with the Numpy developer guide (which also applies to Scipy) do answer quite a few of those questions, but they're not exactly comprehensive. I think we can liberally borrow from other projects on this front, and draft something reasonable that covers at least basic aspects of how decisions are made and about things like licensing, (non-)ownership of trademarks and domain names. This will likely be much easier to reach consensus on than starting from scratch. Request: I'd like to keep detailed discussion on the content of such a governance doc draft for later, and here just discuss/decide whether or not we want to take this on now and who wants to be involved in putting the first draft together. Ralf Developer docs: https://github.com/scipy/scipy/blob/master/HACKING.rst.txt https://github.com/rgommers/scipy/blob/master/doc/MAINTAINERS.rst.txt http://docs.scipy.org/doc/numpy-dev/dev/ https://github.com/numpy/numpy/blob/master/doc/source/dev/development_envir…

7 16

Improvements (scipy.integrate._ode) -- accessing the dense output of DOPRI5 and DOP853.
by Jim Martin 18 Apr '15

18 Apr '15

** Summary ** The wrappers around the differential equation solvers DOPRI5 and DOP853 in "scipy/integrate/_ode.py" do not currently allow access to their "dense output" option (iout=2). This option provides the coefficients of an interpolating polynomial for the solution between the last and current integration step. Dense output interpolation is commonly used for event location and improved plotting. The "_ode.py" code can be modified to optionally allow access to the dense output of DOPRI5/DOP853. The proposed changes are here: https://github.com/scipy/scipy/compare/master...jddmartin:dense_output_from… with example usage here: https://github.com/jddmartin/dense_output_example_usage Existing code using "_ode.py" would not be impacted by the changes. ** Background ** The efficient numerical solution of differential equations requires adaptive stepsizes. Modern high order Runge-Kutta code --- such as DOPRI5 and DOP853 --- can take large variable stepsizes, so that plots made simply by connecting lines between steps are of poor visual quality. As discussed by [Shampine, 1997, see Section 5 and Figure 1, and Shampine et al., 2003, pages 54, 55] this issue is addressed in the Matlab solvers by interpolation between integration steps, with a default of 4 output solutions provided per step (but can be modified using the "Refine" option). Interpolation of the solution output between steps is also desirable for event location (see discussion in Shampine et al., 2003, Section 2.3.1). The Fortran codes DOPRI5/DOP853 have a dense output option (iout=2), as discussed in Hairer et al.'s book [Hairer, 1993]. This dense output option is illustrated in the example usage code for these routines: dr_dopri5.f and dr_dop853.f (see http://www.unige.ch/~hairer/software.html ). However, without implementation of the iout=2 option, these examples cannot be replicated using scipy. Some popular sources such as Press et al.'s "Numerical Recipes" and Boost's odeint have solvers based on DOPRI5/DOP853 and provide access to dense output. ** Details ** The original DOPRI5/DOP853 codes provide the option for a user-specified callback function to be called after each internal integration step. Currently this callback function is only passed the solution at the current step. With the proposed change, the callback function will be provided (optionally) with the coefficients of an interpolating polynomial for the solution between the last and current step by using the "dense output" option (iout=2). As far as the user of scipy.integrate is concerned, the main change is the addition of an optional "dense_components" keyword argument to the ".set_solout" method of classes "ode" and "complex_ode". Behavior should be unchanged in the absence of this keyword. In addition, I have put in some additional error handling code for the bug reported here: https://github.com/scipy/scipy/issues/4118 and made some minor changes (see git commit messages) that should not impact existing code. The new "dense_components" keyword argument is used to specify the components for which dense output is desired. If this keyword option is present, the callback function will now be passed the additional information required for interpolation. A convenience function "dense_dop" has been provided in _ode.py for interpolation using this information. The introduction of the "dense_dop" function seems a bit awkward --- suggestions for alternative approaches are welcome. Most users will simply want to evaluate the interpolation function to determine the solution at points between the last and current step position. My motivation for development of access to the dense output is an application in which I will use the coefficients of the interpolating polynomial directly, rather than the interpolated solution. Access to the interpolating polynomial may also be useful for event location. I've updated the relevant docstrings and added test code for this new option (I have only tested this code on my Linux system with Python 2.7). Comments are welcome. Thanks -- Jim. References: [Shampine, 1997]: Lawrence F. Shampine and Mark W. Reichelt, The MATLAB ODE Suite, SIAM J. Sci. Comput., 18(1), 1–22. 1997 (22 pages) http://dx.doi.org/10.1137/S1064827594276424 [Shampine, 2003], L. F. Shampine, I. Gladwell, S. Thompson, "Solving ODEs with Matlab", 2003, Cambridge University Press. [Hairer, 1993] Hairer et al., "Solving Ordinary Differential Equations, Nonstiff Problems", Second Revised Edition, Springer, 1993.

1 0

Mass 1D interpolation, advice
by Matthew Brett 15 Apr '15

15 Apr '15

Hi, In brain imaging (my world) - we need to do 1D interpolation of large datasets, specifically, we have data in the order of (X, Y) = (200, 4000), where we are interpolating at the same ~200 X locations for each of the 4000 rows. We need to do this with extrapolation at the boundaries, so I think we can't use scipy.interpolate.interp1d which is otherwise well suited for the task. scipy.interpolate.InterpolatedUnivariateSpline does deal with extrapolation, but only works with vectors. I think, in order to make speed reasonable, I will need to loop over the rows in compiled code. My questions are: a) how easy would it be to try and link some external Cython code against the fitpack code in scipy? b) if that is not easy, what can I do that will also be of most use to other scipy users? Cheers, Matthew

4 12

Split signal.lti class into subclasses
by Felix Berkenkamp 12 Apr '15

12 Apr '15

Hi everyone, I started looking into improving the signal.lti class following the issue discussed at https://github.com/scipy/scipy/issues/2912 The pull request can be found here: https://github.com/scipy/scipy/pull/4576 The main idea is to split the lti class into ss, tf, and zpk subclasses. Calling the lti class itself returns instances of these three subclasses. Advantages * No redundant information (lti class currently holds the information of all 3 classes) * Reduce overhead (creating 3 system representations) * Switching between the different subclasses is more explicit: obj.ss(), obj.tf(), obj.zpk() * Avoids one huge class for everything * Is fully backwards compatible (as far as I can tell) * Similar to what Octave / Matlab does (easier to convert code from there to scipy) Disadvantages: * Accessing properties that are not part of the subclass is more expensive (e.g.: sys = ss(1,1,1,1); sys.num --> this now returns sys.tf().num). Any suggestions / comments / things I've broken? Best, Felix

3 3

Adding routines for updating QR decompositions
by Eric Moore 06 Apr '15

06 Apr '15

Hi, I've just submitted PR #4249 adding qr_update, qr_insert and qr_delete for performing rank-k updates, adding rows or columns and removing rows or colums. https://github.com/scipy/scipy/pull/4249 Comments welcome. Eric.

5 6

Re: [SciPy-Dev] [Numpy-discussion] GSoC students: please read
by Ralf Gommers 05 Apr '15

05 Apr '15

On Mon, Mar 23, 2015 at 10:29 PM, Stephan Hoyer <shoyer(a)gmail.com> wrote: > On Mon, Mar 23, 2015 at 2:21 PM, Ralf Gommers <ralf.gommers(a)gmail.com> > wrote: > >> It's great to see that this year there are a lot of students interested >> in doing a GSoC project with Numpy or Scipy. So far five proposals have >> been submitted, and it looks like several more are being prepared now. >> > > Hi Ralf, > > Is there a centralized place for non-mentors to view proposals and give > feedback? > Hi Stephan, there isn't really. All students post their drafts to the mailing list, where they can get feedback. They're free to keep that draft wherever they want - blogs, Github, StackEdit, ftp sites and more are all being used. The central overview is in Melange (the official GSoC tool), but that's not publicly accessible. Note that an overview of project ideas can be found at https://github.com/scipy/scipy/wiki/GSoC-project-ideas. If you're particularly interested in one or more of those, it should be easy to find back in the mailing list archive what students sent draft proposals for feedback. Your comments on individual proposals will be much appreciated. Cheers, Ralf

1 1

Pull request #4648
by Andreea Georgescu 05 Apr '15

05 Apr '15

Hi, I was wondering if there's anything else I could improve on for the pull request #4688 <https://github.com/scipy/scipy/pull/4648> (Fixes #4408: Vector-valued constraints in minimize() et al). It's still marked as "needs-work", although I followed all the things that Pauli Virtanen suggested. I'm a new contributor to SciPy so I'm not sure what the process is at this stage :). Should I just wait for it to be reviewed and merged, or is there anything else I should do? Thanks! Andreea

2 2

`histogram2` and `signaltonoise` deprecation from `scipy.stats`
by Abraham Escalante 04 Apr '15

04 Apr '15

Hello all, As part of the StatistisCleanup milestone (which I aim to complete by late August) `scipy.stats.histogram2` and `scipy.stats.signaltonoise` are to be deprecated but of course, we would like to get opinions from the community. In short: - `histogram2` is not well tested and is unnecessary since `np.histogram2d` can be used instead. - `signaltonoise` doesn't really belong in `scipy.stats` and it is rarely used. For more details, please refer to issues #602 and #609. If you have an objection or any opinion regarding this please let me know to take it into account. Regards, Abraham Escalante.

2 2