Mailman 3 May 2019 - NumPy-Discussion

Style guide for numpy code?
by Chris Barker 09 May '19

09 May '19

Hey all, Do any of you know of a style guide for computational / numpy code? I don't mean code that will go into numpy itself, but rather, users code that uses numpy (and scipy, and...) I know about (am a proponent -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

2 1

Preparation for 1.17.x branch
by Charles R Harris 08 May '19

08 May '19

Hi All, It's time to look forward to branching 1.17.x and the first rc. Please note of any issues that you think fixing or PRs that should be merged before the first release. It would also help if some folks would review the release notes for completeness. I figure the branch should take place in a week or so. Chuck

3 3

NumPy Community Meeting May 8/ 2019
by Tyler Reddy 07 May '19

07 May '19

Hi, There will be a NumPy Community meeting at 12 pm Pacific Time on May 8/ 2019. Anyone is welcome to join and edit the work in progress meeting document: https://hackmd.io/M-ef_Fu5QOOitACnyoO0kQ?view Best wishes, Tyler

1 0

Contributing: NumPy Documentation
by Bhanu Bhandari 07 May '19

07 May '19

Hello, I’m Bhanu, a third year student of Electronics and Computers Engineering. I’d like to get started with contributing into the NumPy documentation. I’m a writer who is actively machine learning research as well, and have used NumPy extensively for my coding assignments and research work. Thanks, Bhanu Bhandari

2 1

GSoD'19 project discussion
by Oishika Pradhan 06 May '19

06 May '19

Hi, I am Oishika Pradhan, a research student at IIIT Hyderabad, India. I am interested in machine learning and neural networks and hence have used numpy very regularly in my projects and assignments. This is why I'm interested in becoming a contributor to this organization. I have some prior experience in technical documentation. I am interested in working on the following project: Improving the structure and content of https://numpy.org/ Please provide pointers on how to get started with this project. Thanks, Oishika Pradhan

2 1

type and kind for custom dtypes
by Alex Samuel 06 May '19

06 May '19

Hi, I'm working on building a number of related custom dtypes, and I'm not sure how to set the type and kind fields in PyArray_Descr. I tried using type='V' and choosing a single unused kind for all my dtypes; this mostly worked, except I found that coercions would sometimes treat values of two different dtypes as if they were the same. But not always... sometimes my registered cast functions would be called. Through trial and error, I've found that if I choose an unused type code for each dtype, coercion seems to work as I expect it to (no coercion unless I've provided a cast). The kind doesn't seem to matter. I couldn't find any guidance in the docs for how to choose these values. Apologies if I've overlooked something. Could someone please advise me? More widely, is there some global registry of these codes? Is the number of NumPy dtypes limited to the number of (UTF-8-encodable) chars? It seems like common practice to use dtype.kind in user code. If I use one or more for my custom dtypes, is there any mechanism to ensure they do not collide with others'? Are there any other semantics for either field I should take into account? Thanks, Alex

2 5

Re: [Numpy-discussion] grant proposal for core scientific Python projects (rejected)
by Chris Barker - NOAA Federal 05 May '19

05 May '19

Thanks for the update — this is great stuff! -CHB On May 3, 2019, at 3:13 PM, Joe Harrington <jh(a)physics.ucf.edu> wrote: Just to keep people in the loop, Ralf and I are in discussion with people at NASA HQ about a funding stream for core development.ï¿½ Ralf has put together a short description of the development and funding model (5 core projects, 10-20 core developers each, nearly all volunteer now, how NumFOCUS fits in, what we hope to establish from NASA vs. from other agencies, industry, other countries' science entities, etc.).ï¿½ That will circulate within the agency, to see what can be scraped together.ï¿½ Program managers in NASA's Science Mission Directorate (SMD) gave quite-positive feedback on how vital the Python ecosystem is to NASA's mission.ï¿½ We're emphasizing the need for both new functionality and maintenance (e.g., docs, web site, bug fixing).ï¿½ If this is ultimately successful, it can be a model for approaching other agencies in the US and elsewhere. To Steve's point, regarding how hard it is for Civil Servants to contribute to OSS (due to NASA's lengthy internal review process for releasing software), this problem was clearly called out in the Academies report.ï¿½ We proposed some solutions to streamline things.ï¿½ What's needed now is for NASA Civil Servants to take that report and the relevant white papers (cited in the report and posted online) to their center's senior management, and to NASA HQ, and similarly for others in government agencies.ï¿½ You may wish to start from NASA's (or your agency's) mission, which includes sharing technology openly to boost the economy, and how you are encountering unreasonable barriers to that goal.ï¿½ This is mandated by the National Air and Space Act of 1958. For example, there is little reason to conduct an export-control review with lawyers looking at code emerging from a group that has nothing to do with anything near an export-controlled topic.ï¿½ Universities and contractors are subject to the same export-control laws as NASA, and they have not routinely conducted similar reviews of every line of code released.ï¿½ This has not led to a pattern of export violations.ï¿½ (Whether there is any benefit at all to the export control laws as applied to software is debatable, since it's usually easy for coders elsewhere to write the same codes, but the law is the law.) --jh-- On 5/3/19 12:48 PM, numpy-discussion-request(a)python.org wrote: Subject: Re: [Numpy-discussion] grant proposal for core scientific Python projects (rejected) From: Mark Mikofski <mikofski(a)berkeley.edu> <mikofski(a)berkeley.edu> Date: 5/3/19, 12:47 PM To: Discussion of Numerical Python <numpy-discussion(a)python.org> <numpy-discussion(a)python.org> Sorry, that last attachment was just a slide show of the topic 3 recording, here is the full funding opportunity announcement - letter with 200 word abstract are due May 7th On Fri, May 3, 2019 at 8:40 AM Mark Mikofski <mikofski(a)berkeley.edu> wrote: > Hi Ralf, and others, > > Sorry for the late notice, but there is are several funding opportunities > in solar, including one for $350,000 to develop open source software to > lower soft costs of solar. > https://eere-exchange.energy.gov/#FoaId45eda43a-e826-4481-ae7a-cc6e8ed4fdae > ï¿½ > see topic 3.4 specifically in attached PDF - also note to view the > recording the password is "*Setofoa2019"*ï¿½it's about 30 minutes long. > > I know that this is a extremely niche, but as a few others have said, [the > DOE] grants tend to be very specific, but perhaps we can creatively think > of ways to channel funds to NumPy and SciPy. > > Also there is a cost share that is typically 20%, which would be a > non-starter for volunteer projects. > > But here's an idea, perhaps partnering with a company, like mine (DNV GL) > who is applying for the grant, and who uses NumPy,and could pay the cost > share, and then we collaborate on something that is required to complete > theï¿½project, which is contributed to NumPy (or SciPy) - but we would have > to figure what we could align on. > > Seems like NumFOCUS, Quantsight, or some other company in the OSS space > could figure out ways to help connect companies, OSS projects, and funding > opportunities like these, where there's a possibility of alignment and > mutual benefit? > > The full list of funding opportunities is here: > https://eere-exchange.energy.gov/ï¿½ > > Best Regards, > Markï¿½ > ï¿½ > > On Thu, May 2, 2019 at 11:52 PM Ralf Gommers <ralf.gommers(a)gmail.com> > wrote: > >> >> >> On Fri, May 3, 2019 at 3:49 AM Stephen Waterbury <waterbug(a)pangalactic.us> >> wrote: >> >>> P.S.ï¿½ If anyone wants to continue this discussion at SciPy 2019, >>> I will be there (on my own nickel!ï¿½ ;) ... >>> >> >> Thanks for the input Stephen, and looking forward to see you at SciPy'19! >> >> Ralf >> >> >> Steve >>> >>> On 5/2/19 9:45 PM, Stephen Waterbury wrote: >>> >>> I am a NASA pythonista (for 20+ years ;), but you can now say you know >>> yet another person at NASA who has no idea this even exists ... :) >>> Not only do I not know of that, but I know of NASA policies that make >>> it very difficult for NASA civil servants to contribute to open source >>> projects -- quite hypocritical, given the amount of open source >>> code that NASA (like all other large organizations) depends critically >>> on, but it's a fact. >>> >>> Cheers, >>> Steve Waterbury >>> >>> (CLEARLY **NOT** SPEAKING IN ANY OFFICIAL CAPACITY FOR NASA OR >>> THE U.S. GOVERNMENT AS A WHOLE!ï¿½ Hence the personal email >>> address. :) >>> >>> On 5/2/19 9:31 PM, Chris Barker - NOAA Federal wrote: >>> >>> Sounds like this is a NASA specific thing, in which case, I guess >>> someone at NASA would need to step up. >>> >>> Iï¿½m afraid I know no pythonistas at NASA.ï¿½ >>> >>> But Iï¿½ll poke around NOAA to see if thereï¿½s anything similar. >>> >>> -CHB >>> >>> On Apr 25, 2019, at 1:04 PM, Ralf Gommers <ralf.gommers(a)gmail.com> >>> wrote: >>> >>> >>> >>> On Sat, Apr 20, 2019 at 12:41 PM Ralf Gommers <ralf.gommers(a)gmail.com> >>> wrote: >>> >>>> >>>> >>>> On Thu, Apr 18, 2019 at 10:03 PM Joe Harrington <jh(a)physics.ucf.edu> >>>> wrote: >>>> >>>> >>>>> 3. There's such a thing as a share-in-savings contract at NASA, in >>>>> which >>>>> you calculate a savings, such as from avoided costs of licensing IDL >>>>> or >>>>> Matlab, and say you'll develop a replacement for that product that >>>>> costs >>>>> less, in exchange for a portion of the savings.ï¿½ These are rare and >>>>> few >>>>> people know about them, but one presenter to the committee did discuss >>>>> them and thought they'd be appropriate.ï¿½ I've always felt that we >>>>> could >>>>> get a chunk of change this way, and was surprised to find that the >>>>> approach exists and has a name.ï¿½ About 3 of 4 people I talk to at >>>>> NASA >>>>> have no idea this even exists, though, and I haven't pursued it to its >>>>> logical end to see if it's viable. >>>>> >>>> >>>> I've heard of these. Definitely worth looking into. >>>> >>> >>> It seems to be hard to find any information about these share-in-savings >>> contracts. The closest thing I found is this: >>> https://www.federalregister.gov/documents/2018/06/22/2018-13463/nasa-federa… >>> >>> It is called "Shared Savings" there, and was replaced last year by >>> something called "Value Engineering Change Proposal". If anyone can comment >>> on whether that's the same thing as Joe meant and whether this is worth >>> following up on, that would be very helpful. >>> >>> Cheers, >>> Ralf >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion(a)python.org >>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >>> >>> _______________________________________________ >>> NumPy-Discussion mailing listNumPy-Discussion@python.orghttps://mail.python.org/mailman/listinfo/numpy-discussion >>> >>> >>> >>> _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion(a)python.org https://mail.python.org/mailman/listinfo/numpy-discussion

1 0

Re: [Numpy-discussion] grant proposal for core scientific Python projects (rejected)
by Joe Harrington 05 May '19

05 May '19

Hi Ralf, The rejection is disappointing, for sure. Some good ammo for next time might be the recommendations in this report from the US National Academies of Science, Engineering, and Medicine: http://sites.nationalacademies.org/SSB/CurrentProjects/SSB_178892 https://www.nap.edu/read/25217/chapter/1#ii You can download a free PDF if you click around and give them an email address. There is some code on the cover that might raise a smile. Lorena Barba, Kelle Cruz, and many other community members contributed, both as committee members and as white-paper authors. While it's a report for NASA, the conclusions are strong and there is explicit support of investment in community resources like numpy, scipy, astropy, matplotlib, etc. Other agencies are asking similar questions, so I expect the report to get somewhat of a look at NSF, etc. A note on the Academies' process: A consensus study report must only include statements that no single member of the committee objects to, and the committee generally only includes senior members or people with a lot of relevant experience. There were stakeholders from some large modeling shops for whom openness might be a threat to their business model, in their eyes, and some of the senior members had never experienced the open-source environment and had bad experiences sharing software. This made for an interesting social dynamic, and prevented an all-out recommendation to forcibly open everything immediately. Given all that, I was ultimately pleased that we got full agreement on the recommendations we did make. So, some general thoughts on fundraising, not specific to this proposal: 1. Try NASA. The Administrator for Space Science, Thomas Zurbuchen, is pushing "open" very hard, given the success of open data in NASA Earth Science, and its positive impact on the economy in fields like agriculture and weather forecasting. He paid for the study above. Many grant programs specifically solicit proposals for open-source tools. There are also technology development programs in other parts of NASA than the Science Mission Directorate. Try contacting Dr. Michael New, who is Zurbuchen's deputy, and could direct you to appropriate programs. (Please, let's be coordinated and not all deluge the guy.) 2. As suggested in another message, it's often easier to get support for a specific, targeted item as part of a big project or institute using that item, such as LSST or the black-hole group. There's a certain way to wend into those projects, usually originating from within. STScI has long devoted programmer resources, for example. 3. There's such a thing as a share-in-savings contract at NASA, in which you calculate a savings, such as from avoided costs of licensing IDL or Matlab, and say you'll develop a replacement for that product that costs less, in exchange for a portion of the savings. These are rare and few people know about them, but one presenter to the committee did discuss them and thought they'd be appropriate. I've always felt that we could get a chunk of change this way, and was surprised to find that the approach exists and has a name. About 3 of 4 people I talk to at NASA have no idea this even exists, though, and I haven't pursued it to its logical end to see if it's viable. 4. I mostly lurk here, since being more actively involved in the early days of numpy docs, so maybe this one has been tried already, or is in the works. My apologies if so. Think of development as a product to buy. You could put chunks of development up for sale, advertise them, and coordinate one or more groups buying them together. For something like an efficiency boost, you could price it according to the avoided cost of CPU resources for a project of a given size (e.g., somewhat below the net present value of avoided future AWS cycles for the projects buying it). It would be like buying a custom-built data pipeline, except that once you buy it, everyone gets it. This might mean scoping out a roadmap of improvements, packaging them into fundable projects with teams ready to go, pricing them, advertising them to specific customers and in trade media and shows, and making sales pitches. This sounds really weird to us scientists, but it would work just like a regular purchase for services, which the government and industry are much more used to doing than donations to open-source projects. Don't just sell what the customer is buying, sell in the manner that the customer likes to buy. 5. And, keep trying grant proposals to NSF! --jh-- On 4/18/19 6:36 AM, Ralf Gommers <ralf.gommers(a)gmail.com> wrote: > > A number of core projects (NumPy, SciPy, Matplotlib, Pandas, > scikit-learn) got together and put in a proposal to NSF for a large 5 > year grant, and it was unfortunately just rejected. We now published > the proposal, which may be of interest: > https://figshare.com/articles/Mid-Scale_Research_Infrastructure_-_The_Scien….

7 16

Re: [Numpy-discussion] grant proposal for core scientific Python projects (rejected)
by Joe Harrington 03 May '19

03 May '19

Just to keep people in the loop, Ralf and I are in discussion with people at NASA HQ about a funding stream for core development. Ralf has put together a short description of the development and funding model (5 core projects, 10-20 core developers each, nearly all volunteer now, how NumFOCUS fits in, what we hope to establish from NASA vs. from other agencies, industry, other countries' science entities, etc.). That will circulate within the agency, to see what can be scraped together. Program managers in NASA's Science Mission Directorate (SMD) gave quite-positive feedback on how vital the Python ecosystem is to NASA's mission. We're emphasizing the need for both new functionality and maintenance (e.g., docs, web site, bug fixing). If this is ultimately successful, it can be a model for approaching other agencies in the US and elsewhere. To Steve's point, regarding how hard it is for Civil Servants to contribute to OSS (due to NASA's lengthy internal review process for releasing software), this problem was clearly called out in the Academies report. We proposed some solutions to streamline things. What's needed now is for NASA Civil Servants to take that report and the relevant white papers (cited in the report and posted online) to their center's senior management, and to NASA HQ, and similarly for others in government agencies. You may wish to start from NASA's (or your agency's) mission, which includes sharing technology openly to boost the economy, and how you are encountering unreasonable barriers to that goal. This is mandated by the National Air and Space Act of 1958. For example, there is little reason to conduct an export-control review with lawyers looking at code emerging from a group that has nothing to do with anything near an export-controlled topic. Universities and contractors are subject to the same export-control laws as NASA, and they have not routinely conducted similar reviews of every line of code released. This has not led to a pattern of export violations. (Whether there is any benefit at all to the export control laws as applied to software is debatable, since it's usually easy for coders elsewhere to write the same codes, but the law is the law.) --jh-- On 5/3/19 12:48 PM, numpy-discussion-request(a)python.org wrote: > Subject: > Re: [Numpy-discussion] grant proposal for core scientific Python > projects (rejected) > From: > Mark Mikofski <mikofski(a)berkeley.edu> > Date: > 5/3/19, 12:47 PM > > To: > Discussion of Numerical Python <numpy-discussion(a)python.org> > > > Sorry, that last attachment was just a slide show of the topic 3 > recording, here is the full funding opportunity announcement - letter > with 200 word abstract are due May 7th > > On Fri, May 3, 2019 at 8:40 AM Mark Mikofski <mikofski(a)berkeley.edu > <mailto:mikofski@berkeley.edu>> wrote: > > Hi Ralf, and others, > > Sorry for the late notice, but there is are several funding > opportunities in solar, including one for $350,000 to develop open > source software to lower soft costs of solar. > https://eere-exchange.energy.gov/#FoaId45eda43a-e826-4481-ae7a-cc6e8ed4fdae > > see topic 3.4 specifically in attached PDF - also note to view the > recording the password is "*Setofoa2019"* it's about 30 minutes long. > > I know that this is a extremely niche, but as a few others have > said, [the DOE] grants tend to be very specific, but perhaps we > can creatively think of ways to channel funds to NumPy and SciPy. > > Also there is a cost share that is typically 20%, which would be a > non-starter for volunteer projects. > > But here's an idea, perhaps partnering with a company, like mine > (DNV GL) who is applying for the grant, and who uses NumPy,and > could pay the cost share, and then we collaborate on something > that is required to complete the project, which is contributed to > NumPy (or SciPy) - but we would have to figure what we could align on. > > Seems like NumFOCUS, Quantsight, or some other company in the OSS > space could figure out ways to help connect companies, OSS > projects, and funding opportunities like these, where there's a > possibility of alignment and mutual benefit? > > The full list of funding opportunities is here: > https://eere-exchange.energy.gov/ > > Best Regards, > Mark > > > On Thu, May 2, 2019 at 11:52 PM Ralf Gommers > <ralf.gommers(a)gmail.com <mailto:ralf.gommers@gmail.com>> wrote: > > > > On Fri, May 3, 2019 at 3:49 AM Stephen Waterbury > <waterbug(a)pangalactic.us <mailto:waterbug@pangalactic.us>> wrote: > > P.S. If anyone wants to continue this discussion at SciPy > 2019, > I will be there (on my own nickel! ;) ... > > > Thanks for the input Stephen, and looking forward to see you > at SciPy'19! > > Ralf > > > Steve > > On 5/2/19 9:45 PM, Stephen Waterbury wrote: > >> I am a NASA pythonista (for 20+ years ;), but you can now >> say you know >> yet another person at NASA who has no idea this even >> exists ... :) >> Not only do I not know of that, but I know of NASA >> policies that make >> it very difficult for NASA civil servants to contribute >> to open source >> projects -- quite hypocritical, given the amount of open >> source >> code that NASA (like all other large organizations) >> depends critically >> on, but it's a fact. >> >> Cheers, >> Steve Waterbury >> >> (CLEARLY **NOT** SPEAKING IN ANY OFFICIAL CAPACITY FOR >> NASA OR >> THE U.S. GOVERNMENT AS A WHOLE! Hence the personal email >> address. :) >> >> On 5/2/19 9:31 PM, Chris Barker - NOAA Federal wrote: >> >>> Sounds like this is a NASA specific thing, in which >>> case, I guess someone at NASA would need to step up. >>> >>> I’m afraid I know no pythonistas at NASA. >>> >>> But I’ll poke around NOAA to see if there’s anything >>> similar. >>> >>> -CHB >>> >>> On Apr 25, 2019, at 1:04 PM, Ralf Gommers >>> <ralf.gommers(a)gmail.com <mailto:ralf.gommers@gmail.com>> >>> wrote: >>> >>>> >>>> >>>> On Sat, Apr 20, 2019 at 12:41 PM Ralf Gommers >>>> <ralf.gommers(a)gmail.com >>>> <mailto:ralf.gommers@gmail.com>> wrote: >>>> >>>> >>>> >>>> On Thu, Apr 18, 2019 at 10:03 PM Joe Harrington >>>> <jh(a)physics.ucf.edu <mailto:jh@physics.ucf.edu>> wrote: >>>> >>>> >>>> 3. There's such a thing as a share-in-savings >>>> contract at NASA, in which >>>> you calculate a savings, such as from avoided >>>> costs of licensing IDL or >>>> Matlab, and say you'll develop a replacement >>>> for that product that costs >>>> less, in exchange for a portion of the savings. >>>> These are rare and few >>>> people know about them, but one presenter to >>>> the committee did discuss >>>> them and thought they'd be appropriate. I've >>>> always felt that we could >>>> get a chunk of change this way, and was >>>> surprised to find that the >>>> approach exists and has a name. About 3 of 4 >>>> people I talk to at NASA >>>> have no idea this even exists, though, and I >>>> haven't pursued it to its >>>> logical end to see if it's viable. >>>> >>>> >>>> I've heard of these. Definitely worth looking into. >>>> >>>> >>>> It seems to be hard to find any information about these >>>> share-in-savings contracts. The closest thing I found >>>> is this: >>>> https://www.federalregister.gov/documents/2018/06/22/2018-13463/nasa-federa… >>>> >>>> It is called "Shared Savings" there, and was replaced >>>> last year by something called "Value Engineering Change >>>> Proposal". If anyone can comment on whether that's the >>>> same thing as Joe meant and whether this is worth >>>> following up on, that would be very helpful. >>>> >>>> Cheers, >>>> Ralf >>>> >>>> _______________________________________________ >>>> NumPy-Discussion mailing list >>>> NumPy-Discussion(a)python.org >>>> <mailto:NumPy-Discussion@python.org> >>>> https://mail.python.org/mailman/listinfo/numpy-discussion >>> >>> _______________________________________________ >>> NumPy-Discussion mailing list >>> NumPy-Discussion(a)python.org <mailto:NumPy-Discussion@python.org> >>> https://mail.python.org/mailman/listinfo/numpy-discussion >> >> >

1 0

EuroSciPy 2019 - extension of deadline
by Pierre de Buyl 03 May '19

03 May '19

Hi all, The call for proposals for EuroSciPy 2019 is extended to 12 may 2019. https://pretalx.com/euroscipy-2019/cfp EuroSciPy 2019 takes place 2-6 September 2019 in Bilbao, Spain. There will be two days of tutorials, two days of conference, and one day of sprints! Find us on the web https://www.euroscipy.org/2019/ and on twitter https://twitter.com/EuroSciPy Regards, The EuroSciPy 2019 team

1 0

NumPy-Discussion May 2019