Mailman 3 December 2020 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker 11 Jul '23

11 Jul '23

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem 18 Feb '23

18 Feb '23

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How do I avoid these messages to appear? I know that I could in principle use numpy.seterr. However, I do NOT want to remove these warnings for other potential divide/multiply/sqrt etc errors. Only when I am using a "where", to in fact avoid such warnings! Note that the warnings only happen once, but since I am going to release that code, I would like to avoid the user to get such messages which are irrelevant here (because I am testing, with the where, when NOT to divide by zero or take a sqrt of a negative number). thanks! Eric

5 4

Documentation Team meeting - Monday June 8th
by Melissa Mendonça 04 Dec '22

04 Dec '22

Hi all! A reminder that on Monday, June 8, we have another documentation team meeting at 3PM UTC**. If you wish to join on Zoom, you need to use this link https://zoom.us/j/420005230 Here's the permanent hackmd document with the meeting notes: https://hackmd.io/oB_boakvRqKR-_2jRV-Qjg <https://www.google.com/url?q=https%3A%2F%2Fhackmd.io%2FoB_boakvRqKR-_2jRV-Q…> Hope to see you around (especially if you want to introduce yourself or discuss ideas for Google Season of Docs). ** You can click this link to get the correct time at your timezone: https://www.timeanddate.com/worldclock/fixedtime.html?msg=NumPy+Documentati… - Melissa

6 74

[Feature Request] Add alias of np.concatenate as np.concat
by Iordanis Fostiropoulos 10 May '22

10 May '22

In regard to Feature Request: https://github.com/numpy/numpy/issues/16469 It was suggested to sent to the mailing list. I think I can make a strong point as to why the support for this naming convention would make sense. Such as it would follow other frameworks that often work alongside numpy such as tensorflow. For backward compatibility, it can simply be an alias to np.concatenate I often convert portions of code from tf to np, it is as simple as changing the base module from tf to np. e.g. np.expand_dims -> tf.expand_dims. This is done either in debugging (e.g. converting tf to np without eager execution to debug portion of the code), or during prototyping, e.g. develop in numpy and convert in tf. I find myself more than at one occasion to getting syntax errors because of this particular function np.concatenate. It is unnecessarily long. I imagine there are more people that also run into the same problems. Pandas uses concat (torch on the other extreme uses simply cat, which I don't think is as descriptive).

7 6

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell 22 Jul '21

22 Jul '21

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil

10 10

Addition of new distributions: Polya-gamma
by Zolisa Bleki 27 Jan '21

27 Jan '21

Hi All, I would like to know if Numpy accepts addition of new distributions since the implementation of the Generator interface. If so, what is the criteria for a particular distribution to be accepted? The reason why i'm asking is because I would like to propose adding the Polya-gamma distribution to numpy, for the following reasons: 1) Polya-gamma random variables are commonly used as auxiliary variables during data augmentation in Bayesian sampling algorithms, which have wide-spread usage in Statistics and recently, Machine learning. 2) Since this distribution is mostly useful for random sampling, it since appropriate to have it in numpy and not projects like scipy [1]. 3) The only python/C++ implementation of the sampler available is licensed under GPLv3 which I believe limits copying into packages that choose to use a different license [2]. 4) Numpy's random API makes adding the distribution painless. I have done preliminary work on this by implementing the distribution sampler as decribed in [3]; see: https://github.com/numpy/numpy/compare/master...zoj613:polyagamma . There is a more efficient sampling algorithm described in a later paper [4], but I chose not to start with that one unless I know it is worth investing time in. I would appreciate your thoughts on this proposal. Regards, Zolisa Refs: [1] https://github.com/scipy/scipy/issues/11009 [2] https://github.com/slinderman/pypolyagamma [3] https://arxiv.org/pdf/1205.0310v1.pdf [4] https://arxiv.org/pdf/1405.0506.pdf Disclaimer - University of Cape Town This email is subject to UCT policies and email disclaimer published on our website at http://www.uct.ac.za/main/email-disclaimer or obtainable from +27 21 650 9111. If this email is not related to the business of UCT, it is sent by the sender in an individual capacity. Please report security incidents or abuse via https://csirt.uct.ac.za/page/report-an-incident.php.

5 5

updated backwards compatibility and deprecation policy NEP
by Ralf Gommers 26 Jan '21

26 Jan '21

Hi all, Here is a long overdue update of the draft NEP about backwards compatibility and deprecation policy: https://github.com/numpy/numpy/pull/18097 - This is NEP 23: https://numpy.org/neps/nep-0023-backwards-compatibility.html - Link to the previous mailing list discussion: https://mail.python.org/pipermail/numpy-discussion/2018-July/078432.html It would be nice to get this NEP to Accepted status. Main changes are: - Removed all examples that people objected to - Removed all content regarding versioning - Restructured sections, and added "Strategies related to deprecations" (using suggestions by @njsmith and @shoyer). - Added concrete examples of deprecations, and a more thorough description of how to go about adding warnings incl. Sphinx directives, using `stacklevel`, etc. As always, feedback here or on the PR is very welcome! Cheers, Ralf Abstract -------- In this NEP we describe NumPy's approach to backwards compatibility, its deprecation and removal policy, and the trade-offs and decision processes for individual cases where breaking backwards compatibility is considered. Motivation and Scope -------------------- NumPy has a very large user base. Those users rely on NumPy being stable and the code they write that uses NumPy functionality to keep working. NumPy is also actively maintained and improved -- and sometimes improvements require, or are made much easier by, breaking backwards compatibility. Finally, there are trade-offs in stability for existing users vs. avoiding errors or having a better user experience for new users. These competing needs often give rise to long debates and to delays in accepting or rejecting contributions. This NEP tries to address that by providing a policy as well as examples and rationales for when it is or isn't a good idea to break backwards compatibility. In scope for this NEP are: - Principles of NumPy's approach to backwards compatibility. - How to deprecate functionality, and when to remove already deprecated functionality. - Decision making process for deprecations and removals. Out of scope are: - Making concrete decisions about deprecations of particular functionality. - NumPy's versioning scheme. General principles ------------------ When considering proposed changes that are backwards incompatible, the main principles the NumPy developers use when making a decision are: 1. Changes need to benefit users more than they harm them. 2. NumPy is widely used so breaking changes should by default be assumed to be fairly harmful. 3. Decisions should be based on data and actual effects on users and downstream packages rather than, e.g., appealing to the docs or for stylistic reasons. 4. Silently getting a wrong answer is much worse than getting a loud error. When assessing the costs of proposed changes, keep in mind that most users do not read the mailing list, do not look at deprecation warnings, and sometimes wait more than one or two years before upgrading from their old version. And that NumPy has millions of users, so "no one will do or use this" is very likely incorrect. Benefits include improved functionality, usability and performance, as well as lower maintenance cost and improved future extensibility. Fixes for clear bugs are exempt from this backwards compatibility policy. However in case of serious impact on users (e.g. a downstream library doesn't build anymore or would start giving incorrect results), even bug fixes may have to be delayed for one or more releases. Strategies related to deprecations ---------------------------------- Getting hard data on the impact of a deprecation of often difficult. Strategies that can be used to assess such impact include: - Use a code search engine ([1]_) or static ([2]_) or dynamic ([3]_) code analysis tools to determine where and how the functionality is used. - Testing prominent downstream libraries against a development build of NumPy containing the proposed change to get real-world data on its impact. - Making a change in master and reverting it, if needed, before a release. We do encourage other packages to test against NumPy's master branch, so this often turns up issues quickly. If the impact is unclear or significant, it is often good to consider alternatives to deprecations. For example discouraging use in documentation only, or moving the documentation for the functionality to a less prominent place or even removing it completely. Commenting on open issues related to it that they are low-prio or labeling them as "wontfix" will also be a signal to users, and reduce the maintenance effort needing to be spent. Implementing deprecations and removals -------------------------------------- Deprecation warnings are necessary in all cases where functionality will eventually be removed. If there is no intent to remove functionality, then it should not be deprecated either. A "please don't use this" in the documentation or other type of warning should be used instead. Deprecations: - shall include the version number of the release in which the functionality was deprecated. - shall include information on alternatives to the deprecated functionality, or a reason for the deprecation if no clear alternative is available. - shall use ``VisibleDeprecationWarning`` rather than ``DeprecationWarning`` for cases of relevance to end users. For cases only relevant to downstream libraries, a regular ``DeprecationWarning`` is fine. *Rationale: regular deprecation warnings are invisible by default; library authors should be aware how deprecations work and test for them, but we can't expect this from all users.* - shall be listed in the release notes of the release where the deprecation is first present. - shall set a ``stacklevel``, so the warning appears to come from the correct place. - shall be mentioned in the documentation for the functionality. A ``.. deprecated::`` directive can be used for this. Examples of good deprecation warnings: .. code-block:: python warnings.warn('np.asscalar(a) is deprecated since NumPy 1.16.0, use ' 'a.item() instead', DeprecationWarning, stacklevel=3) warnings.warn("Importing from numpy.testing.utils is deprecated " "since 1.15.0, import from numpy.testing instead.", DeprecationWarning, stacklevel=2) # A change in NumPy 1.14.0 for Python 3 loadtxt/genfromtext, slightly # tweaked in this NEP (original didn't have version number). warnings.warn( "Reading unicode strings without specifying the encoding " "argument is deprecated since NumPy 1.14.0. Set the encoding, " "use None for the system default.", np.VisibleDeprecationWarning, stacklevel=2) Removal of deprecated functionality: - shall be done after at least 2 releases (assuming the current 6-monthly release cycle; if that changes, there shall be at least 1 year between deprecation and removal). - shall be listed in the release notes of the release where the removal happened. - can be done in any minor (but not bugfix) release. For backwards incompatible changes that aren't "deprecate and remove" but for which code will start behaving differently, a ``FutureWarning`` should be used. Release notes, mentioning version number and using ``stacklevel`` should be done in the same way as for deprecation warnings. A ``.. versionchanged::`` directive can be used in the documentation to indicate when the behavior changed: .. code-block:: python def argsort(self, axis=np._NoValue, ...): """ Parameters ---------- axis : int, optional Axis along which to sort. If None, the default, the flattened array is used. .. versionchanged:: 1.13.0 Previously, the default was documented to be -1, but that was in error. At some future date, the default will change to -1, as originally intended. Until then, the axis should be given explicitly when ``arr.ndim > 1``, to avoid a FutureWarning. """ ... warnings.warn( "In the future the default for argsort will be axis=-1, not the " "current None, to match its documentation and np.argsort. " "Explicitly pass -1 or None to silence this warning.", MaskedArrayFutureWarning, stacklevel=3) Decision making ~~~~~~~~~~~~~~~ In concrete cases where this policy needs to be applied, decisions are made according to the `NumPy governance model <https://docs.scipy.org/doc/numpy/dev/governance/index.html>`_. All deprecations must be proposed on the mailing list, in order to give everyone with an interest in NumPy development to be able to comment. Removal of deprecated functionality does not need discussion on the mailing list. Functionality with more strict deprecation policies ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - ``numpy.random`` has its own backwards compatibility policy, see `NEP 19 <http://www.numpy.org/neps/nep-0019-rng-policy.html>`_. - The file format for ``.npy`` and ``.npz`` files must not be changed in a backwards incompatible way. Example cases ------------- We now discuss a few concrete examples from NumPy's history to illustrate typical issues and trade-offs. **Changing the behavior of a function** ``np.histogram`` is probably the most infamous example. First, a new keyword ``new=False`` was introduced, this was then switched over to None one release later, and finally it was removed again. Also, it has a ``normed`` keyword that had behavior that could be considered either suboptimal or broken (depending on ones opinion on the statistics). A new keyword ``density`` was introduced to replace it; ``normed`` started giving ``DeprecationWarning`` only in v.1.15.0. Evolution of ``histogram``:: def histogram(a, bins=10, range=None, normed=False): # v1.0.0 def histogram(a, bins=10, range=None, normed=False, weights=None, new=False): #v1.1.0 def histogram(a, bins=10, range=None, normed=False, weights=None, new=None): #v1.2.0 def histogram(a, bins=10, range=None, normed=False, weights=None): #v1.5.0 def histogram(a, bins=10, range=None, normed=False, weights=None, density=None): #v1.6.0 def histogram(a, bins=10, range=None, normed=None, weights=None, density=None): #v1.15.0 # v1.15.0 was the first release where `normed` started emitting # DeprecationWarnings The ``new`` keyword was planned from the start to be temporary. Such a plan forces users to change their code more than once, which is almost never the right thing to do. Instead, a better approach here would have been to deprecate ``histogram`` and introduce a new function ``hist`` in its place. **Disallowing indexing with floats** Indexing an array with floats is asking for something ambiguous, and can be a sign of a bug in user code. After some discussion, it was deemed a good idea to deprecate indexing with floats. This was first tried for the v1.8.0 release, however in pre-release testing it became clear that this would break many libraries that depend on NumPy. Therefore it was reverted before release, to give those libraries time to fix their code first. It was finally introduced for v1.11.0 and turned into a hard error for v1.12.0. This change was disruptive, however it did catch real bugs in, e.g., SciPy and scikit-learn. Overall the change was worth the cost, and introducing it in master first to allow testing, then removing it again before a release, is a useful strategy. Similar deprecations that also look like good examples of cleanups/improvements: - removing deprecated boolean indexing (in 2016, see `gh-8312 < https://github.com/numpy/numpy/pull/8312>`__) - deprecating truth testing on empty arrays (in 2017, see `gh-9718 < https://github.com/numpy/numpy/pull/9718>`__) **Removing the financial functions** The financial functions (e.g. ``np.pmt``) had short non-descriptive names, were present in the main NumPy namespace, and didn't really fit well within NumPy's scope. They were added in 2008 after `a discussion < https://mail.python.org/pipermail/numpy-discussion/2008-April/032353.html>`_ on the mailing list where opinion was divided (but a majority in favor). The financial functions didn't cause a lot of overhead, however there were still multiple issues and PRs a year for them which cost maintainer time to deal with. And they cluttered up the ``numpy`` namespace. Discussion on removing them happened in 2013 (gh-2880, rejected) and then again in 2019 (:ref:`NEP32`, accepted without significant complaints). Given that they were clearly outside of NumPy's scope, moving them to a separate ``numpy-financial`` package and removing them from NumPy after a deprecation period made sense. Alternatives ------------ **Being more aggressive with deprecations.** The goal of being more aggressive is to allow NumPy to move forward faster. This would avoid others inventing their own solutions (often in multiple places), as well as be a benefit to users without a legacy code base. We reject this alternative because of the place NumPy has in the scientific Python ecosystem - being fairly conservative is required in order to not increase the extra maintenance for downstream libraries and end users to an unacceptable level. Discussion ---------- - `Mailing list discussion on the first version of this NEP in 2018 < https://mail.python.org/pipermail/numpy-discussion/2018-July/078432.html>`__ References and Footnotes ------------------------ - `Issue requesting semantic versioning < https://github.com/numpy/numpy/issues/10156>`__ .. [1] https://searchcode.com/ .. [2] https://github.com/Quansight-Labs/python-api-inspect .. [3] https://github.com/data-apis/python-record-api

5 9

Comment published in Nature Astronomy about The ecological impact of computing with Python
by PIERRE AUGIER 26 Jan '21

26 Jan '21

Hi, I recently took a bit of time to study the comment "The ecological impact of high-performance computing in astrophysics" published in Nature Astronomy (Zwart, 2020, https://www.nature.com/articles/s41550-020-1208-y, https://arxiv.org/pdf/2009.11295.pdf), where it is stated that "Best however, for the environment is to abandon Python for a more environmentally friendly (compiled) programming language.". I wrote a simple Python-Numpy implementation of the problem used for this study (https://www.nbabel.org) and, accelerated by Transonic-Pythran, it's very efficient. Here are some numbers (elapsed times in s, smaller is better): | # particles | Py | C++ | Fortran | Julia | |-------------|-----|-----|---------|-------| | 1024 | 29 | 55 | 41 | 45 | | 2048 | 123 | 231 | 166 | 173 | The code and a modified figure are here: https://github.com/paugier/nbabel (There is no check on the results for https://www.nbabel.org, so one still has to be very careful.) I think that the Numpy community should spend a bit of energy to show what can be done with the existing tools to get very high performance (and low CO2 production) with Python. This work could be the basis of a serious reply to the comment by Zwart (2020). Unfortunately the Python solution in https://www.nbabel.org is very bad in terms of performance (and therefore CO2 production). It is also true for most of the Python solutions for the Computer Language Benchmarks Game in https://benchmarksgame-team.pages.debian.net/benchmarksgame/ (codes here https://salsa.debian.org/benchmarksgame-team/benchmarksgame#what-else). We could try to fix this so that people see that in many cases, it is not necessary to "abandon Python for a more environmentally friendly (compiled) programming language". One of the longest and hardest task would be to implement the different cases of the Computer Language Benchmarks Game in standard and modern Python-Numpy. Then, optimizing and accelerating such code should be doable and we should be able to get very good performance at least for some cases. Good news for this project, (i) the first point can be done by anyone with good knowledge in Python-Numpy (many potential workers), (ii) for some cases, there are already good Python implementations and (iii) the work can easily be parallelized. It is not a criticism, but the (beautiful and very nice) new Numpy website https://numpy.org/ is not very convincing in terms of performance. It's written "Performant The core of NumPy is well-optimized C code. Enjoy the flexibility of Python with the speed of compiled code." It's true that the core of Numpy is well-optimized C code but to seriously compete with C++, Fortran or Julia in terms of numerical performance, one needs to use other tools to move the compiled-interpreted boundary outside the hot loops. So it could be reasonable to mention such tools (in particular Numba, Pythran, Cython and Transonic). Is there already something planned to answer to Zwart (2020)? Any opinions or suggestions on this potential project? Pierre PS: Of course, alternative Python interpreters (PyPy, GraalPython, Pyjion, Pyston, etc.) could also be used, especially if HPy (https://github.com/hpyproject/hpy) is successful (C core of Numpy written in HPy, Cython able to produce HPy code, etc.). However, I tend to be a bit skeptical in the ability of such technologies to reach very high performance for low-level Numpy code (performance that can be reached by replacing whole Python functions with optimized compiled code). Of course, I hope I'm wrong! IMHO, it does not remove the need for a successful HPy! -- Pierre Augier - CR CNRS http://www.legi.grenoble-inp.fr LEGI (UMR 5519) Laboratoire des Ecoulements Geophysiques et Industriels BP53, 38041 Grenoble Cedex, France tel:+33.4.56.52.86.16

12 22

NumPy EOL Versions?
by Symphoni Bush - NOAA Affiliate 13 Jan '21

13 Jan '21

I am trying to find out if there are any end-of-life versions for NumPy, and if so, when do these versions typically become EOL/unsupported? Thanks.

4 5

proposal for NumPy sponsorship guidelines (NEP 46)
by Ralf Gommers 03 Jan '21

03 Jan '21

Hi all, I just opened https://github.com/numpy/numpy/pull/18084, "NumPy Sponsorship Guidelines". Below are the most important parts for review (for Related Work, References, etc. see the PR). Please bring up broader points here, and small/textual feedback on the PR. Cheers, Ralf Abstract -------- This NEP provides guidelines on how the NumPy project will acknowledge financial and in-kind support. Motivation and Scope -------------------- In the past few years the NumPy project has gotten significant financial support, as well as dedicated work time for maintainers to work on NumPy. There is a need to acknowledge that support - funders and organizations expect or require it, it's helpful when looking for new funding, and it's the right thing to do. Furthermore, having a clear policy for how NumPy acknowledges support is helpful when searching for new support. This NEP is aimed at both the NumPy community - who can use it when looking for support and acknowledging existing support - and at past, current and prospective sponsors, who often want or need to know what they get in return for their support (other than a healthier NumPy). The scope of this proposal includes: - direct financial support, employers providing paid time for NumPy maintainers and regular contributors, and in-kind support such as free hardware resources or services. - where and how NumPy acknowledges support (e.g., logo placement on the website). - the amount and duration of support which leads to acknowledgement. - who in the NumPy project is responsible for sponsorship related topics, and how to contact them. How NumPy will acknowledge support ---------------------------------- There will be two different ways to acknowledge financial and in-kind support, one to recognize significant active support, and another one to recognize support received in the past and smaller amounts of support. Entities who fall under "significant active supporter" we'll call Sponsor. The minimum level of support given to NumPy to be considered a Sponsor are: - $30,000/yr for unrestricted financial contributions - $60,000/yr for financial contributions for a particular purpose - $100,000/yr for in-kind contributions The rationale for the above levels is that unrestricted financial contributions are typically the most valuable for the project, and the hardest to obtain. The opposite is true for in-kind contributions. The dollar value of the levels also reflect that NumPy's needs have grown to the point where we need at least a few paid developers in order to effectively support our user base and continue to move the project forward. Financial support at or above these levels is needed to be able to make a significant difference. Sponsors will get acknowledged through: - a small logo displayed on the front page of the NumPy website - prominent logo placement on https://numpy.org/about/ - logos displayed in talks about NumPy by maintainers - announcements of the sponsorship on the NumPy mailing list and the numpy-team Twitter account In addition to Sponsors, we already have the concept of Institutional Partner (defined in NumPy's `governance document <https://numpy.org/devdocs/dev/governance/index.html >`__), for entities who employ a NumPy maintainer and let them work on NumPy as part of their official duties. The governance document doesn't currently define a minimum amount of paid maintainer time needed to be considered for partnership. Therefore we propose that level here, roughly in line with the sponsorship levels: - 6 person-months/yr of paid work time for one or more NumPy maintainers or regular contributors Institutional Partners get the same benefits as Sponsors, in addition to what is specified in the NumPy governance document. Finally, a new page on the website (https://numpy.org/funding/, linked from the About page) will be added to acknowledge all current and previous sponsors, partners, and any other entities and individuals who provided $5,000 or more of financial or in-kind support. This page will include relevant details of support (dates, amounts, names and purpose); no logos will be used on this page. The rationale for the $5,000 minimum level is to keep the amount of work maintaining the page reasonable; the level is the equivalent of, e.g., one GSoC or a person-week's worth of engineering time in a Western country, which seems like a reasonable lower limit. Implementation -------------- The following content changes need to be made: - Add a section with small logos towards the bottom of the `numpy.org <https://numpy.org/>`__ website. - Create a full list of historical and current support and deploy it to https://numpy.org/funding. - Update the NumPy governance document for changes to Institutional Partner eligibility requirements and benefits. - Update https://numpy.org/about with details on how to get in touch with the NumPy project about sponsorship related matters (see next section). A NumPy Funding Team ~~~~~~~~~~~~~~~~~~~~ At the moment NumPy has only one official body, the Steering Council, and no good way to get in touch with either that body or any person or group responsible for funding and sponsorship related matters. The way this is typically done now is to somehow find the personal email of a maintainer, and email them in private. There is a need to organize this more transparently - a potential sponsor isn't likely to inquire through the mailing list, nor is it easy for a potential sponsor to know if they're reaching out to the right person in private. https://numpy.org/about/ already says that NumPy has a "funding and grants" team, however that is not the case. We propose to organize this team, name team members on it, and add the names of those team members plus a dedicated email address for the team to the About page. Status before this proposal --------------------------- Acknowledgement of support ~~~~~~~~~~~~~~~~~~~~~~~~~~ At the time of writing (Dec 2020), the logos of the four largest financial sponsors and two institutional partners are displayed on https://numpy.org/about/. The `Nature paper about NumPy < https://www.nature.com/articles/s41586-020-2649-2>`__ mentions some early funding. No comprehensive list of received funding and in-kind support is published anywhere. Decisions on which logos to list on the website have been made mostly by the website team. Decisions on which entities to recognize as Institutional Partner have been made by the NumPy Steering Council. NumPy governance, decision-making and financial oversight ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ *This section is meant as context for the reader, to help put the rest of this NEP in perspective, and perhaps answer questions the reader has when reading this as a potential sponsor.* NumPy has a formal governance structure defined in `this governance document < https://numpy.org/devdocs/dev/governance/index.html>`__). Decisions are made by consensus among all active participants in a discussion (typically on the mailing list), and if consensus cannot be reached then the Steering Council takes the decision (also by consensus). NumPy is a sponsored project of NumFOCUS, a US-based 501(c)3 nonprofit. NumFOCUS administers NumPy funds, and ensures they are spent in accordance with its mission and nonprofit status. In practice, NumPy has a NumFOCUS subcommittee (with its members named in the NumPy governance document) who can authorize financial transactions. Those transactions, for example paying a contractor for a particular activity or deliverable, are decided on by the NumPy Steering Council. Alternatives ------------ *Tiered sponsorship levels.* We considered using tiered sponsorship levels, and rejected this alternative because it would be more complex, and not necessarily communicate the right intent - the minimum levels are for us to determine how to acknowledge support that we receive, not a commercial value proposition. Entities typically will support NumPy because they rely on the project or want to help advance it, and not to get brand awareness through logo placement. *Listing all donations*. Note that in the past we have received many smaller donations, mostly from individuals through NumFOCUS. It would be great to list all of those contributions, but given the way we receive information on those donations right now, that would be quite labor-intensive. If we manage to move to a more suitable platform, such as `Open Collective < https://opencollective.com/>`__, in the future, we should reconsider listing all individual donations.

2 2