Hi, On Sat, Apr 11, 2015 at 5:09 AM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Thu, Apr 9, 2015 at 8:46 PM, Matthew Brett <matthew.brett@gmail.com> wrote:
Hi,
On Sun, Apr 5, 2015 at 3:31 PM, Ralf Gommers <ralf.gommers@gmail.com> wrote:
Hi,
Here's a long story about something I've been thinking about for a while. Some individual parts have been discussed with a few people, but the complete story is only how I look at things at the moment. I make some concrete proposals that could be agreed on in this thread if they turn out to be uncontroversial; we can start separate threads as needed though.
Summary --------------
*Problem Statement* The growth of core developer and project admin capacity, while pretty strong over the past few years, is not quite keeping up with the growth in contributions we're seeing.
*Solution* Identify this as an important issue and start addressing it more actively.
*Main Proposals* - recruitment: a "become a core dev" mentoring program - docs: - write "how to become a core developer" documentation - improve the developer guide further - finally merge the Scipy 1.0 roadmap - communication: set up a private core dev mailing list - funding: - we got some donations (excellent, very useful) - have an agreed upon strategy (written down) of what to do with funds - then aim at getting more donations and/or getting grants - governance: we have very little formalized, add something lightweight that covers most bases
Long Version -------------------
*Problem Statement* The number of open PRs is very slowly increasing, and stands now at ~110. Some of those PRs are stalled, but many simply need reviewing. The number of active core developers who review and merge PRs is also increasing, but evidently not at the rate that is necessary for PRs to be reviewed quickly. The sometimes long delays in getting a PR merged have likely also discouraged further contributions from some new contributors.
Note that we did have a lot of growth in active core developers over the past few years - I remember Fernando Perez pointing out to me at EuroSciPy'11 that the Scipy bus factor was exactly 2, now it's more like 8.
*Recruitment* Therefore I'd like to look for new core developers in a more pro-active way. That starts by very clearly saying:
We need your help!
To be more precise, we need both domain specialists that help review PRs for a single module and core developers that have a broad understanding of how things are done in Scipy and that take responsibility for signing off on the quality of PRs and are merging them independently.
A "core dev mentoring program". This can go quite quickly for people that are experienced Python programmers and are familiar with the Scipy community; it may be as little as a few email conversations or Skype calls on how to get started. On the other end of the scale, I'd also like to help people who are new to the Scipy community or even open source development get started - as long as they have the right level of coding/communication skills of course. Then the road to commit rights will of course be longer, but as long as the ambition to get there is present then it's worth the effort (I know I certainly wasn't very experienced in contributing to open source projects at the time I became Scipy release manager, and could definitely have used some help).
*Docs* What developer docs are missing or need significant improvements? Let's add those (I volunteer for writing duty). We have a reasonable set (links at end of email), but it's quite scattered. And what we don't have is "how to become a core developer" documentation, let's add that. Plus finally merge the 1.0 roadmap: https://github.com/scipy/scipy/pull/2908.
*Communication* I'd like to set up a new core dev closed mailing list. Not with the purpose to do less in public, but simply for organizing the communication that we already have off-list in a very ad-hoc manner now. Typical topics: deciding on commit rights, discussions on ranking GSoC students, potential funding sources and what to do with such funding. I'm not yet sure what to propose for people with access to that list. We currently have 20 people with commit rights, of which about half are actually active. Currently decisions are mostly made between the active devs.
*Funding* I'm starting to see more opportunities to use funding in effective ways to sustain or accelerate Scipy development. Main ones are organizing developer meetups / sprints, and targeted funding to talented individuals (students mainly) and for work on particularly hairy bottlenecks (example: Windows test&build). We have a donation page on scipy.org and have received a number of donations, but have not yet actively solicited donations because of (a) no fiscal sponsorship agreement with NumFOCUS signed yet, and (b) while we can use the money now, I'm uncomfortable to spend large amounts without some written down strategy on how and why we spend funds.
*Governance* Scipy activity is currently very much centered around code, and other project aspects get relatively little attention. Big email debates about governance are often not very productive (more high-bandwidth communication is necessary here), but it would be helpful to keep making steps in this area. Documenting processes, how decisions are made, who decides on things like commit rights, resolving API discussions, etc. is important. That goes hand in hand with the developer docs mentioned above. The HACKING and MAINTAINERS documents together with the Numpy developer guide (which also applies to Scipy) do answer quite a few of those questions, but they're not exactly comprehensive.
I think we can liberally borrow from other projects on this front, and draft something reasonable that covers at least basic aspects of how decisions are made and about things like licensing, (non-)ownership of trademarks and domain names. This will likely be much easier to reach consensus on than starting from scratch. Request: I'd like to keep detailed discussion on the content of such a governance doc draft for later, and here just discuss/decide whether or not we want to take this on now and who wants to be involved in putting the first draft together.
Maybe a developer hangout would be the best way forward for all of these?
Thanks Matthew, that's a good idea.
Probably a Google hangout at some time that's evening in Europe and daytime in the US would work well. Here's a poll, please fill it out if you want to participate: http://doodle.com/ibqnemrvuhkvexm2
Done - thanks.
Would be good to make an effort to keep the discussion as public as possible. So let's make sure we write a summary of what gets discussed and post it to this list. And I propose that people who are regular contributors but happen to not have a commit bit are also very welcome to join the hangout.
Thanks - that is a good idea.
The discussions on governance in the past have been so miserable and tendentious that it might be better to separate that into a separate track.
I think for Scipy previous discussions (there weren't many) were fine, but yes I agree that for Numpy they weren't always pleasant. Let's discuss on the hangout what the best approach is.
Yes, right, it was the numpy discussions, but there is so much overlap between the lists, I am sure the atmosphere over there has had an effect. Cheers, Matthew