Mailman 3 April 2019 - NumPy-Discussion

Sebastian Berg at BIDS
by Stefan van der Walt 27 Apr '19

27 Apr '19

Hi everyone, We are happy to announce that Sebastian Berg has joined the team at BIDS, and will be working full-time on NumPy for the next two years. Sebastian has been a core contributor to the project for seven years already, and we are delighted that he will now be able to spend even more of his effort here. Sebastian will be working on the dtype refactoring and, as the rest of the team, be supporting the NumPy community in whichever ways we can. Please join the community call this Wednesday to welcome Sebastian on board. As always, your feedback on our work is much appreciated! Best regards, Stéfan

5 4

2019 John Hunter Excellence in Plotting Contest Reminder
by Madicken Munk 26 Apr '19

26 Apr '19

Hi everybody, I'd like to remind you all about the 2019 John Hunter Excellence in Plotting Contest. My apologies to those of you getting this on multiple lists. In memory of John Hunter, we are pleased to be reviving the SciPy John Hunter Excellence in Plotting Competition for 2019. This open competition aims to highlight the importance of data visualization to scientific progress and showcase the capabilities of open source software. Participants are invited to submit scientific plots to be judged by a panel. The winning entries will be announced and displayed at the conference. John Hunter’s family and NumFocus are graciously sponsoring cash prizes for the winners in the following amounts: - 1st prize: $1000 - 2nd prize: $750 - 3rd prize: $500 - Entries must be submitted by June, 8th to the form at https://goo.gl/forms/cFTB3FUBrMPfQ7Vz1 - Winners will be announced at Scipy 2019 in Austin, TX. - Participants do not need to attend the Scipy conference. - Entries may take the definition of “visualization” rather broadly. Entries may be, for example, a traditional printed plot, an interactive visualization for the web, or an animation. - Source code for the plot must be provided, in the form of Python code and/or a Jupyter notebook, along with a rendering of the plot in a widely used format. This may be, for example, PDF for print, standalone HTML and Javascript for an interactive plot, or MPEG-4 for a video. If the original data can not be shared for reasons of size or licensing, "fake" data may be substituted, along with an image of the plot using real data. - Each entry must include a 300-500 word abstract describing the plot and its importance for a general scientific audience. - Entries will be judged on their clarity, innovation and aesthetics, but most importantly for their effectiveness in communicating a real-world problem. Entrants are encouraged to submit plots that were used during the course of research or work, rather than merely being hypothetical. - SciPy reserves the right to display any and all entries, whether prize-winning or not, at the conference, use in any materials or on its website, with attribution to the original author(s). SciPy John Hunter Excellence in Plotting Competition Co-Chairs Hannah Aizenman Thomas Caswell Madicken Munk Nelle Varoquaux

1 0

bounty for PowerPC SIMD improvements in NumPy
by Ralf Gommers 23 Apr '19

23 Apr '19

Hi all, Now that I'm in write-long-emails-about-$$ mode, I'll continue with: IBM just posted at $5500 bounty on BountySource for adding VSX (= PowerPC equivalent of SSE/AVX SIMD instructions we have for x86) support to NumPy: https://github.com/numpy/numpy/issues/13393. This is an interesting experiment I think. There are some potential issues around this model that I see, because there's basically no "project management" (for lack of a better word) around bounties. Things that can potentially go wrong: - multiple people can show up with competing PRs - a PR stalls because of lack of reviewer time - a PR gets rejected because there is no consensus that the feature is wanted or the implementation is done the right way - etc However on the other hand, perhaps it's a good way to get things done that otherwise won't get done. I don't know yet..... Either way, we will learn something from this. I will try to keep track of what is going on to try and avoid the potential issues listed above. And for full disclosure: IBM did pre-discuss this with us, and it was brought up on the NumPy Steering Council list first, where there was some interest and no one raised a stop flag. We can evaluate the success and desirability of bounties later on. Cheers, Ralf

1 0

Tidelift - NumFOCUS agreement for NumPy
by Ralf Gommers 23 Apr '19

23 Apr '19

Hi all, On behalf of the NumPy Steering Council I'm happy to announce that we now have an agreement between Tidelift and NumFOCUS for NumPy. The summary of the agreement is: Tidelift will pay NumPy a minimum of $1000/month until Oct 2020, and NumPy will do the following: - provide a documented way to disclose security vulnerabilities, and respond to disclosures in a timely manner - deal with any licensing issues in a timely manner - write good release notes, and clarify our advice to users on what releases to use - some one-time things like getting our metadata into the Tidelift system, and acknowledging Tidelift as one of our funders on the website This blog gives a nice overview: https://blog.tidelift.com/how-to-start-earning-money-for-your-open-source-p… . Note that it seems to us that this is a quite modest amount of work that we will be able to do with volunteer resources. A lot of it we do anyway - this is a nice feature of Tidelift's business model, in a way they promise their customers that we will keep doing what we're doing, add some valuable things like unified dependency reporting around it, and pass on some of the benefits to the projects (or to individual maintainers for other projects). We haven't determined what to do with the funds yet, but there's lots of things that could be done (enable dev meetings, pay for a numpy.org redesign, perhaps fund some work on hairy problems that no one seems to want to solve for free, etc.) - to be determined in the future. The Tidelift model was discussed on this list back in September ( https://mail.python.org/pipermail/numpy-discussion/2018-September/078736.ht…) but at that point there was no "project wide" solution and the "pay some individuals" model had some issues. Letting all the funding flow into the NumPy account at NumFOCUS nicely solves this. Personally I plan to work with the core teams of SciPy and Pandas to get similar agreements in place, because there's also significant funding available for those two projects. Cheers, Ralf

1 0

NumPy community meeting April 24/ 2019
by Tyler Reddy 23 Apr '19

23 Apr '19

Hi, There will be a NumPy community meeting at 12 pm Pacific Time on April 24/ 2019. Anyone is welcome to join & edit the meeting notes available here: https://hackmd.io/a03zhF4TSsewJp_fXXXwOQ?view

1 0

numpydoc 0.9.0 released
by Ralf Gommers 23 Apr '19

23 Apr '19

Hi all, Numpydoc 0.9.0 was just released. Please see https://github.com/numpy/numpydoc/milestone/4?closed=1 for a list of included fixes and new features. Cheers, Ralf

1 2

Boolean arrays with nulls?
by Stuart Reynolds 22 Apr '19

22 Apr '19

Is there an efficient way to represent bool arrays with null entries? Tools like pandas push you very hard into 64 bit float representations - 64 bits where 3 will suffice is neither efficient for memory, nor (consequently), speed. What I’m hoping for is that there’s a structure that is ‘viewed’ as nan-able float data, but backed but a more efficient structures internally. Thanks - Stu

4 5

NumPy 1.16. 3 release
by Charles R Harris 21 Apr '19

21 Apr '19

Hi All, On behalf of the NumPy team I am pleased to announce the release of NumPy 1.16.3 which contains several fixes for newly reported bugs. The Python versions supported in this release are 2.7 and 3.5-3.7. There are several notable changes. - Unpickling object arrays when loading *.npy or *.npz files now requires an explicit opt-in. This backwards incompatible change was made in response to CVE-2019-6446 <https://nvd.nist.gov/vuln/detail/CVE-2019-6446>. - The covariance in 'random.mvnormal' is cast to double. This should make the tolerance more useful when checking the singular values of the covariance matrix. - The '__array_interface__' offset now works as documented. The offset value was being mistakenly ignored. Downstream developers building this release should use Cython >= 0.29.2 and, if using OpenBLAS, OpenBLAS > v0.3.4. Wheels for this release can be downloaded from PyPI <https://pypi.org/project/numpy/1.16.3/>, source archives and release notes are available from Github <https://github.com/numpy/numpy/releases/tag/v1.16.3>. If you are installing using pip, you may encounter a problem with older installed versions of NumPy that pip did not delete becoming mixed with the current version, resulting in an ``ImportError``. That problem is particularly common on Debian derived distributions due to a modified pip. The fix is to make sure all previous NumPy versions installed by pip have been removed. See #12736 <https://github.com/numpy/numpy/issues/12736> for discussion of the issue. *Contributors* A total of 16 people contributed to this release. People with a "+" by their names contributed a patch for the first time. - Andreas Schwab - Bharat Raghunathan + - Bran + - Charles Harris - Eric Wieser - Jakub Wilk - Kevin Sheppard - Marten van Kerkwijk - Matti Picus - Paul Ivanov - Ralf Gommers - Sebastian Berg - Tyler Reddy - Warren Weckesser - Yu Feng - adeak + *Pull requests merged* A total of 26 pull requests were merged for this release. - gh-13072 : BUG: Fixes to numpy.distutils.Configuration.get_version (#13056) - gh-13082 : BUG: Fix errors in string formatting while producing an error - gh-13083 : BUG: Convert fortran flags in environment variable - gh-13084 : BUG: Remove error-prone borrowed reference handling - gh-13085 : BUG: Add error checks when converting integers to datetime types - gh-13091 : BUG: Remove our patched version of `distutils.split_quoted` - gh-13141 : BUG: Fix testsuite failures on ppc and riscv - gh-13142 : BUG: Fix parameter validity checks in ``random.choice`` - gh-13143 : BUG: Ensure linspace works on object input. - gh-13144 : BLD: fix include list for sdist building. - gh-13145 : BUG: __array_interface__ offset was always ignored - gh-13274 : MAINT: f2py: Add a cast to avoid a compiler warning. - gh-13275 : BUG, MAINT: fix reference count error on invalid input to ndarray.flat - gh-13276 : ENH: Cast covariance to double in random mvnormal - gh-13278 : BUG: Fix null pointer dereference in PyArray_DTypeFromObjectHelper - gh-13339 : BUG: Use C call to sysctlbyname for AVX detection on MacOS. - gh-13340 : BUG: Fix crash when calling savetxt on a padded array - gh-13341 : BUG: ufunc.at iteration variable size fix - gh-13342 : DOC: Add as_ctypes_type to the documentation - gh-13350 : BUG: Return the coefficients array directly - gh-13351 : BUG/MAINT: Tidy typeinfo.h and .c - gh-13359 : BUG: Make allow_pickle=False the default for loading - gh-13360 : DOC: fix some doctest failures - gh-13363 : BUG/MAINT: Tidy typeinfo.h and .c - gh-13381 : BLD: address mingw-w64 issue. Follow-up to gh-9977 - gh-13382 : REL: Prepare for the NumPy release. Cheers, Charles Harris

1 0

grant proposal for core scientific Python projects (rejected)
by Ralf Gommers 20 Apr '19

20 Apr '19

Hi all, A number of core projects (NumPy, SciPy, Matplotlib, Pandas, scikit-learn) got together and put in a proposal to NSF for a large 5 year grant, and it was unfortunately just rejected. We now published the proposal, which may be of interest: https://figshare.com/articles/Mid-Scale_Research_Infrastructure_-_The_Scien… . Those of you who are on Twitter may already know about this. First mention of this rejection on Twitter with a lot of conversation following: https://twitter.com/amuellerml/status/1117455802598662144. Full quote from Andreas Mueller, replying to a tweet that the first ever image of a black hole was built on Matplotlib, SciPy, NumPy, Pandas, Jupyter, AstroPy: "Slightly ironic that in the same week @NSF rejects a grant to fund the scipy ecosystem saying that working on it is not impactful enough and hiring developers to work on it is too expensive." And a follow-up conversation on Twitter about the rejection: https://twitter.com/amuellerml/status/1118617331058475008 This proposal was led by Columbia, who submitted it together with NumFOCUS and Quansight. It was largely written by Andreas Mueller (scikit-learn, also the PI), Andy Terrel (NumFOCUS) and myself (NumPy/SciPy), with a lot of valuable input from Thomas Caswell (Matplotlib), Jeff Reback (Pandas), Gina Helfrich (NumFOCUS), the other co-PIs, the core teams of the projects, and many others who pitched in ideas and advice. This was the first time we tried a proposal of this scale and ambition (as far as I know), and while it's disappointing that the NSF doesn't seem to value software much (not really news, nor unique to NSF), rejections are a normal part of submitting grant proposals and we now have a much better idea of what it will take to submit further proposals in the future. Which we plan on doing. Cheers, Ralf

2 2

For broadcasting, can m by n by k matrix be multiplied with n by k matrix?
by C W 19 Apr '19

19 Apr '19

Hello all, Can an m x n x k matrix be multiplied with n x k matrix? Looking at the Numpy doc page 46 ( https://docs.scipy.org/doc/numpy-1.11.0/numpy-user-1.11.0.pdf), it should work. It says the following: A (3d array): 15 x 3 x 5 B (2d array): 3 x 5 Result (3d array): 15 x 3 x 5 But, the rule did not work for me. Here's my toy example: >>> a = np.arange(3*4*5).reshape(3,4,5) >>> b = np.arange(4*5).reshape(4,5) >>> np.dot(a, b) Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: shapes (3,4,5) and (3,5) not aligned: 5 (dim 2) != 3 (dim 0) Am I miss reading something? Thank you in advance!

3 5