On Thu, Apr 18, 2019 at 5:27 PM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:

Hi Ralf,

Thanks for the feedback Marten, it's very valuable. We've gotten some more feedback from people experienced with applying for or reviewing NSF (and DOE, NASA, NIH) grants, it helps a lot in figuring out what to do next.

I'm sorry to hear the proposal did not pass the first round, but, having looked at it briefly (about as much time as I would have spent had I been on the panel), I have to admit I am not surprised: it is nice but nice is not enough for a competition like this.

Compared to what will have included some really exciting, novel proposals, most damning will likely have been the modest, incremental goals (for a large sum of money):

I'm very aware that's how reviewers will look at it. I don't agree it's true though (not sure if you think that) - the impact on the science NSF supports of spending on the order of 10 million dollars on the SciPy ecosystem will be way higher than of building some new facility, or supporting one more supercomputer, or whatever else will have been proposed. Another thought, and this does make our job harder, is that it's very difficult to claim to do really novel things. Because whatever we do must in the end pass review from and be accepted by the core teams of each project and the community. Proposing really novel things will mean starting new projects; which is just a different kind of proposal - more easy to sell, likely a lot less impactful. performance improvements, but without any actual sense of what would now

become solvable (how does it beat throwing more computers at a problem, which is cheap?); better implementations of things that exist (arrays with units, sparse arrays); better GPU support (feels like something everybody and their brother was excited about a decade ago); etc. I also think any panel would expect some concrete examples of facilities that would now be helped: e.g., how is this going to help LSST analyze its 20TB/night of data?

You're completely right - we should have focused more on this, trying to be more concrete. Note that it's very hard to come up with provable statements like "this is what we can do after this proposal that we can't do now", but we must do better here. If anyone has references that we can use that would be very helpful; things like the Decadal Survey in astronomy that state something about the SciPy ecosystem). What we can also do is better elaborate the impact of not maintaining/evolving our projects.

Going forward, best may be to explicitly involve the facilities that use python - within astronomy, that would include LIGO and LSST,

We did, both of those. We also had a senior LIGO person as co-PI, and includes a quote from a LIGO spokesperson about this being critical infrastructure for them. but certainly also STScI (and other NASA institutes), which actually

supports SPE already. It would be good especially to show how much money it would save them when this is implemented, so that it becomes clear this is a net win.

I like that idea. Indeed, for any future proposal, I'd suggest to involve (or at least ask

for advice) some more senior people who have been successful before (within astronomy, the likes of Steve Kahn, the LSST director; he was at Columbia for most of his career, so there is a connection).

We also did that, got advice and a full draft proposal review several times from a former NSF program manager. As well as from the likes of Fernando Perez and Brian Granger at the start and Ryan Abernathy at the end. Your advice about the kinds of people to get involved is all true, but I don't think this was our main issue because we did all that. Also talking to the PMs, we know that's critical and did that (the government shutdown didn't help here though). Cheers, Ralf

On Thu, Apr 18, 2019 at 5:27 PM Marten van Kerkwijk < m.h.vankerkwijk@gmail.com> wrote:

Hi Ralf,

Thanks for the feedback Marten, it's very valuable. We've gotten some more feedback from people experienced with applying for or reviewing NSF (and DOE, NASA, NIH) grants, it helps a lot in figuring out what to do next.

I'm sorry to hear the proposal did not pass the first round, but, having looked at it briefly (about as much time as I would have spent had I been on the panel), I have to admit I am not surprised: it is nice but nice is not enough for a competition like this.

Compared to what will have included some really exciting, novel proposals, most damning will likely have been the modest, incremental goals (for a large sum of money):

I'm very aware that's how reviewers will look at it. I don't agree it's true though (not sure if you think that) - the impact on the science NSF supports of spending on the order of 10 million dollars on the SciPy ecosystem will be way higher than of building some new facility, or supporting one more supercomputer, or whatever else will have been proposed. Another thought, and this does make our job harder, is that it's very difficult to claim to do really novel things. Because whatever we do must in the end pass review from and be accepted by the core teams of each project and the community. Proposing really novel things will mean starting new projects; which is just a different kind of proposal - more easy to sell, likely a lot less impactful. performance improvements, but without any actual sense of what would now

become solvable (how does it beat throwing more computers at a problem, which is cheap?); better implementations of things that exist (arrays with units, sparse arrays); better GPU support (feels like something everybody and their brother was excited about a decade ago); etc. I also think any panel would expect some concrete examples of facilities that would now be helped: e.g., how is this going to help LSST analyze its 20TB/night of data?

You're completely right - we should have focused more on this, trying to be more concrete. Note that it's very hard to come up with provable statements like "this is what we can do after this proposal that we can't do now", but we must do better here. If anyone has references that we can use that would be very helpful; things like the Decadal Survey in astronomy that state something about the SciPy ecosystem). What we can also do is better elaborate the impact of not maintaining/evolving our projects.

Going forward, best may be to explicitly involve the facilities that use python - within astronomy, that would include LIGO and LSST,

We did, both of those. We also had a senior LIGO person as co-PI, and includes a quote from a LIGO spokesperson about this being critical infrastructure for them. but certainly also STScI (and other NASA institutes), which actually

supports SPE already. It would be good especially to show how much money it would save them when this is implemented, so that it becomes clear this is a net win.

I like that idea. Indeed, for any future proposal, I'd suggest to involve (or at least ask

for advice) some more senior people who have been successful before (within astronomy, the likes of Steve Kahn, the LSST director; he was at Columbia for most of his career, so there is a connection).

We also did that, got advice and a full draft proposal review several times from a former NSF program manager. As well as from the likes of Fernando Perez and Brian Granger at the start and Ryan Abernathy at the end. Your advice about the kinds of people to get involved is all true, but I don't think this was our main issue because we did all that. Also talking to the PMs, we know that's critical and did that (the government shutdown didn't help here though). Cheers, Ralf