Mailman 3 September 2019 - NumPy-Discussion

NEP 30 - Duck Typing for NumPy Arrays - Implementation
by Peter Andreas Entschev Sept. 17, 2019

Sept. 17, 2019

Hi, we have a new proposal for the implementation of NumPy array duck typing [1] [2], following the high-level overview described in NEP-22 [3]. Would be great to get some comments on that. [1] https://github.com/numpy/numpy/blob/master/doc/neps/nep-0030-duck-array-pro… [2] https://github.com/numpy/numpy/pull/14170 [3] https://numpy.org/neps/nep-0022-ndarray-duck-typing-overview.html Best, Peter

6 23

NumPy Community Meeting Wednesday, Sep. 18
by Sebastian Berg Sept. 17, 2019

Sept. 17, 2019

Hi all, There will be a NumPy Community meeting Wednesday September 18 at 11 am Pacific Time. Everyone is invited to join in and edit the work-in- progress meeting topics and notes: https://hackmd.io/76o-IxCjQX2mOXO_wwkcpg?both Best wishes Sebastian

1 0

Re: [Numpy-discussion] How to Capitalize numpy?
by Joe Harrington Sept. 17, 2019

Sept. 17, 2019

Here are my thoughts on textual capitalization (at first, I thought you wanted to raise money!): We all agree that in code, it is "numpy". If you don't use that, it throws an error. If, in text, we keep "numpy" with a forced lower-case letter at the start, it is just one more oddball to remember. It is even weirder in titles and the beginnings of sentences. I'd strongly like not to be weird that way. A few packages are, it's annoying, and it doesn't much earn them any goodwill. The default among people who are not "in the know" will be to do what they're used to. Let's give them what they're used to, a proper noun with initial (at least) capital. Likewise, I object to preferring a particular font. What fonts to use for the names of things like software packages is a decision for publications to make. A journal or manual might make fine distinctions and demand several different, specific fonts, while a popular publication might prefer not to do that. Leave the typesetting to the editors of the publications. We can certainly adopt a standard for our publications (docs, web pages, etc.), but we should state explicitly that others can do as they like. It's not an acronym, so that leaves the options of "Numpy" and "NumPy". It would be great, easy to remember, consistent for others, etc., if NumPy and SciPy were capitalized the same way and were pronounced the same (I still occasionally hear "numpee"). So, I would favor "NumPy" to go along with "SciPy", and let the context choose the font. --jh-- On 9/16/19 9:09 PM, Chris Barker <chris.barker(a)noaa.gov> wrote: Trivial note: On the subject of naming things (spelling things??) -- should it be: numpy or Numpy or NumPy ? All three are in the draft NEP 30 ( mostly "NumPy", I noticed this when reading/copy editing the NEP) . Is there an "official" capitalization? My preference, would be to use "numpy", and where practicable, use a "computer" font -- i.e. ``numpy`` in RST. But if there is consensus already for anything else, that's fine, I'd just like to know what it is. -CHB On Mon, Aug 12, 2019 at 4:02 AM Peter Andreas Entschev <peter(a)entschev.com <mailto:peter@entschev.com>> wrote: Apologies for the late reply. I've opened a new PR https://github.com/numpy/numpy/pull/14257 with the changes requested on clarifying the text. After reading the detailed description, I've decided to add a subsection "Scope" to clarify the scope where NEP-30 would be useful. I think the inclusion of this new subsection complements the "Detail description" forming a complete text w.r.t. motivation of the NEP, but feel free to point out disagreements with my suggestion. I've also added a new section "Usage" pointing out how one would use duck array in replacement to np.asarray where relevant. Regarding the naming discussion, I must say I like the idea of keeping the __array_ prefix, but it seems like that is going to be difficult given that none of the existing ideas so far play very nicely with that. So if the general consensus is to go with __numpy_like__, I would also update the NEP to reflect that changes. FWIW, I particularly neither like nor dislike __numpy_like__, but I don't have any better suggestions than that or keeping the current naming. Best, Peter On Thu, Aug 8, 2019 at 3:40 AM Stephan Hoyer <shoyer(a)gmail.com <mailto:shoyer@gmail.com>> wrote: > > > > On Wed, Aug 7, 2019 at 6:18 PM Charles R Harris <charlesr.harris(a)gmail.com <mailto:charlesr.harris@gmail.com>> wrote: >> >> >> >> On Wed, Aug 7, 2019 at 7:10 PM Stephan Hoyer <shoyer(a)gmail.com <mailto:shoyer@gmail.com>> wrote: >>> >>> On Wed, Aug 7, 2019 at 5:11 PM Ralf Gommers <ralf.gommers(a)gmail.com <mailto:ralf.gommers@gmail.com>> wrote: >>>> >>>> >>>> On Mon, Aug 5, 2019 at 6:18 PM Stephan Hoyer <shoyer(a)gmail.com <mailto:shoyer@gmail.com>> wrote: >>>>> >>>>> On Mon, Aug 5, 2019 at 2:48 PM Ralf Gommers <ralf.gommers(a)gmail.com <mailto:ralf.gommers@gmail.com>> wrote: >>>>> >>>>>> >>>>>> The NEP currently does not say who this is meant for. Would you expect libraries like SciPy to adopt it for example? >>>>>> >>>>>> The NEP also (understandably) punts on the question of when something is a valid duck array. If you want this to be widely used, that will need an answer or at least some rough guidance though. For example, we would expect a duck array to have a mean() method, but probably not a ptp() method. A library author who wants to use np.duckarray() needs to know, because she can't test with all existing and future duck array implementations. >>>>> >>>>> >>>>> I think this is covered in NEP-22 already. >>>> >>>> >>>> It's not really. We discussed this briefly in the community call today, Peter said he will try to add some text. >>>> >>>> We should not add new functions to NumPy without indicating who is supposed to use this, and what need it fills / problem it solves. It seems pretty clear to me that it's mostly aimed at library authors rather than end users. And also that mature libraries like SciPy may not immediately adopt it, because it's too fuzzy - so it's new libraries first, mature libraries after the dust has settled a bit (I think). >>> >>> >>> I totally agree -- we definitely should clarify this in the docstring and elsewhere in the docs. An example in the new doc page on "Writing custom array containers" (https://numpy.org/devdocs/user/basics.dispatch.html) would also probably be appropriate. >>> >>>>> >>>>> As discussed there, I don't think NumPy is in a good position to pronounce decisive APIs at this time. I would welcome efforts to try, but I don't think that's essential for now. >>>> >>>> >>>> There's no need to pronounce a decisive API that fully covers duck array. Note that RNumPy is an attempt in that direction (not a full one, but way better than nothing). In the NEP/docs, at least saying something along the lines of "if you implement this, we recommend the following strategy: check if a function is present in Dask, CuPy and Sparse. If so, it's reasonable to expect any duck array to work here. If not, we suggest you indicate in your docstring what kinds of duck arrays are accepted, or what properties they need to have". That's a spec by implementation, which is less than ideal but better than saying nothing. >>> >>> >>> OK, I agree here as well -- some guidance is better than nothing. >>> >>> Two other minor notes on this NEP, concerning naming: >>> 1. We should have a brief note on why we settled on the name "duck array". Namely, as discussed in NEP-22, we don't love the "duck" jargon, but we couldn't come up with anything better since NumPy already uses "array like" and "any array" for different purposes. >>> 2. The protocol should use *something* more clearly namespaced as NumPy specific than __duckarray__. All the other special protocols NumPy defines start with "__array_". That suggests either __array_duckarray__ (sounds a little redundant) or __numpy_duckarray__ (which I like the look of, but is a different from the existing protocols). >>> >> >> `__numpy_like__` ? > > > > This could work, but I think we would also want to rename the NumPy function itself to either np.like or np.numpy_like. The later is a little redundant but definitely more self-descriptive than "duck array". > >> >> Chuck >> _______________________________________________ >> NumPy-Discussion mailing list >> NumPy-Discussion(a)python.org <mailto:NumPy-Discussion@python.org> >> https://mail.python.org/mailman/listinfo/numpy-discussion > > _______________________________________________ > NumPy-Discussion mailing list > NumPy-Discussion(a)python.org <mailto:NumPy-Discussion@python.org> > https://mail.python.org/mailman/listinfo/numpy-discussion _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion(a)python.org <mailto:NumPy-Discussion@python.org> https://mail.python.org/mailman/listinfo/numpy-discussion -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov <mailto:Chris.Barker@noaa.gov>

3 2

Add a total_seconds() method to timedelta64?
by Chris Barker Sept. 16, 2019

Sept. 16, 2019

I just noticed that there is no obvious way to convert a timedelta64 to seconds (or some other easy unit) as a number. The stdlib datetime.datetime has a .total_seconds() method for doing that. I think it's a handy thing to have. Looking at StackOverflow (and others), I see people suggesting things like: a_timedelta.astype(np.float) / 1e6 This seems a really bad idea, as it's assuming the timedelta is storing milliseconds. The "proper" way to do it also suggested: a_timedelta / np.timedelta64(1, 's') This is, in fact, a much better way to do it, and allows you to specify other units if you like: "ms"., "us", etc. There was semi-recently a discussion thread on python-ideas about adding other methods to datetime: (e.g. .total_hours, .total_minutes). That was pretty much rejected (or petered out anyway), and some argued that dividing by a timedelta of the unit you want is the "right" way to do it anyway (some argued that .total_seconds() never should have been added. Personally I understand the "correctness" of using united-division, but "practicality beats purity", and the discoverability of a method or two really makes it easier on folks. That being said, of folks don't want to add .total_seconds and the like -- we should add a bit to the docs about this, suggesting using the division approach. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker(a)noaa.gov

1 0

round / set_printoptions discrepancy
by Irvin Probst Sept. 13, 2019

Sept. 13, 2019

Hi, Is it expected/documented that np.round and np.set_printoptions do not output the same result on screen ? I tumbled into this running this code: import numpy as np mes = np.array([ [16.06, 16.13, 16.06, 16.00, 16.06, 16.00, 16.13, 16.00] ]) avg = np.mean(mes, axis=1) print(np.round(avg, 2)) np.set_printoptions(precision=2) print(avg) Which outputs: [16.06] [16.05] Is that worth a bug report or did I miss something ? I've been able to reproduce this on many windows/linux PCs with python/numpy releases from 2017 up to last week. Thanks.

5 8

Code review for adding axis argument to permutation and shuffle function
by Kexuan Sun Sept. 13, 2019

Sept. 13, 2019

Hi, I would like to request a code review. The random.permutation and random.shuffle functions now can only shuffle along the first axis of a multi-dimensional array. I propose to add an axis argument for the functions and allow them to shuffle along a given axis. Here is the link to the PR (https://github.com/numpy/numpy/pull/13829). Thanks!

5 4

Re: [Numpy-discussion] round / set_printoptions discrepancy
by Stefano Miccoli Sept. 13, 2019

Sept. 13, 2019

In my opinion the problem is that numpy floats break the Liskov substitution principle, >>> pyfloat = 16.055 >>> npfloat = np.float64(pyfloat) >>> isinstance(npfloat, float) True >>> round(pyfloat, 2) 16.05 >>> round(npfloat, 2) 16.06 Since numpy.float64 is a subclass of builtins.float I would expect that >>> round(x, j) == round(np.float64(x), j) is an invariant, but unfortunately this is not the case. Moreover the python3 semantics of the round function require that when the number of digits is None, the return value should be of integral type: >>> round(pyfloat) 16 >>> round(pyfloat, None) 16 >>> round(pyfloat, 0) 16.0 >>> round(npfloat) 16.0 >>> round(npfloat, None) 16.0 >>> round(npfloat, 0) 16.0 see also https://github.com/numpy/numpy/issues/11810 Stefano

1 0

NumPy Community Survey
by Inessa Pawson Sept. 12, 2019

Sept. 12, 2019

You know that NumPy is essential to the Python community. The NumPy team wants you to know that YOU, our user and developer community, are essential to us. That’s why we are putting together a team to create the inaugural NumPy Community Survey. We hope feedback will provide insights that will help us to guide better decision-making about the development of NumPy as software and community. For more information about the proposed survey please refer to github.com/numpy/numpy-surveys . *Call for Contributions* We are looking for volunteers experienced in survey design and translating English into Spanish, Portuguese, Russian, Hindi, Chinese and other languages. If you’d like to learn more about these volunteer opportunities, or additional ways to support NumPy, feel free to reach out to our community coordinators at numpy-team(a)googlegroups.com or join us on Slack numpy-team.slack.com (email to numpy-team(a)googlegroups.com for an invite first). -- Every good wish, Inessa Pawson NumPy Team

3 2

Using hypothesis in testing
by Matti Picus Sept. 11, 2019

Sept. 11, 2019

2 1

NEP 32: Remove the financial functions from NumPy
by Warren Weckesser Sept. 11, 2019

Sept. 11, 2019

Github issue 2880 ("Get financial functions out of main namespace", https://github.com/numpy/numpy/issues/2880) has been open since 2013. In a recent community meeting, it was suggested that we create a NEP to propose the removal of the financial functions from NumPy. I have submitted "NEP 32: Remove the financial functions from NumPy" in a pull request at https://github.com/numpy/numpy/pull/14399. A copy of the latest version of the NEP is below. According to the NEP process document, "Once the PR is in place, the NEP should be announced on the mailing list for discussion (comments on the PR itself should be restricted to minor editorial and technical fixes)." This email is the announcement for NEP 32. The NEP includes a brief summary of the history of the financial functions, and has links to several relevant mailing list threads, dating back to when the functions were added to NumPy in 2008. I recommend reviewing those threads before commenting here. Warren ----- ================================================== NEP 32 — Remove the financial functions from NumPy ================================================== :Author: Warren Weckesser <warren.weckesser(a)gmail.com> :Status: Draft :Type: Standards Track :Created: 2019-08-30 Abstract -------- We propose deprecating and ultimately removing the financial functions [1]_ from NumPy. The functions will be moved to an independent repository, and provided to the community as a separate package with the name ``numpy_financial``. Motivation and scope -------------------- The NumPy financial functions [1]_ are the 10 functions ``fv``, ``ipmt``, ``irr``, ``mirr``, ``nper``, ``npv``, ``pmt``, ``ppmt``, ``pv`` and ``rate``. The functions provide elementary financial calculations such as future value, net present value, etc. These functions were added to NumPy in 2008 [2]_. In May, 2009, a request by Joe Harrington to add a function called ``xirr`` to the financial functions triggered a long thread about these functions [3]_. One important point that came up in that thread is that a "real" financial library must be able to handle real dates. The NumPy financial functions do not work with actual dates or calendars. The preference for a more capable library independent of NumPy was expressed several times in that thread. In June, 2009, D. L. Goldsmith expressed concerns about the correctness of the implementations of some of the financial functions [4]_. It was suggested then to move the financial functions out of NumPy to an independent package. In a GitHub issue in 2013 [5]_, Nathaniel Smith suggested moving the financial functions from the top-level namespace to ``numpy.financial``. He also suggested giving the functions better names. Responses at that time included the suggestion to deprecate them and move them from NumPy to a separate package. This issue is still open. Later in 2013 [6]_, it was suggested on the mailing list that these functions be removed from NumPy. The arguments for the removal of these functions from NumPy: * They are too specialized for NumPy. * They are not actually useful for "real world" financial calculations, because they do not handle real dates and calendars. * The definition of "correctness" for some of these functions seems to be a matter of convention, and the current NumPy developers do not have the background to judge their correctness. * There has been little interest among past and present NumPy developers in maintaining these functions. The main arguments for keeping the functions in NumPy are: * Removing these functions will be disruptive for some users. Current users will have to add the new ``numpy_financial`` package to their dependencies, and then modify their code to use the new package. * The functions provided, while not "industrial strength", are apparently similar to functions provided by spreadsheets and some calculators. Having them available in NumPy makes it easier for some developers to migrate their software to Python and NumPy. It is clear from comments in the mailing list discussions and in the GitHub issues that many current NumPy developers believe the benefits of removing the functions outweigh the costs. For example, from [5]_:: The financial functions should probably be part of a separate package -- Charles Harris If there's a better package we can point people to we could just deprecate them and then remove them entirely... I'd be fine with that too... -- Nathaniel Smith +1 to deprecate them. If no other package exists, it can be created if someone feels the need for that. -- Ralf Gommers I feel pretty strongly that we should deprecate these. If nobody on numpy’s core team is interested in maintaining them, then it is purely a drag on development for NumPy. -- Stephan Hoyer And from the 2013 mailing list discussion, about removing the functions from NumPy:: I am +1 as well, I don't think they should have been included in the first place. -- David Cournapeau But not everyone was in favor of removal:: The fin routines are tiny and don't require much maintenance once written. If we made an effort (putting up pages with examples of common financial calculations and collecting those under a topical web page, then linking to that page from various places and talking it up), I would think they could attract users looking for a free way to play with financial scenarios. [...] So, I would say we keep them. If ours are not the best, we should bring them up to snuff. -- Joe Harrington For an idea of the maintenance burden of the financial functions, one can look for all the GitHub issues [7]_ and pull requests [8]_ that have the tag ``component: numpy.lib.financial``. One method for measuring the effect of removing these functions is to find all the packages on GitHub that use them. Such a search can be performed with the ``python-api-inspect`` service [9]_. A search for all uses of the NumPy financial functions finds just eight repositories. (See the comments in [5]_ for the actual SQL query.) Implementation -------------- * Create a new Python package, ``numpy_financial``, to be maintained in the top-level NumPy github organization. This repository will contain the definitions and unit tests for the financial functions. The package will be added to PyPI so it can be installed with ``pip``. * Deprecate the financial functions in the ``numpy`` namespace, beginning in NumPy version 1.18. Remove the financial functions from NumPy version 1.20. Backward compatibility ---------------------- The removal of these functions breaks backward compatibility, as explained earlier. The effects are mitigated by providing the ``numpy_financial`` library. Alternatives ------------ The following alternatives were mentioned in [5]_: * *Maintain the functions as they are (i.e. do nothing).* A review of the history makes clear that this is not the preference of many NumPy developers. A recurring comment is that the functions simply do not belong in NumPy. When that sentiment is combined with the history of bug reports and the ongoing questions about the correctness of the functions, the conclusion is that the cleanest solution is deprecation and removal. * *Move the functions from the ``numpy`` namespace to ``numpy.financial``.* This was the initial suggestion in [5]_. Such a change does not address the maintenance issues, and doesn't change the misfit that many developers see between these functions and NumPy. It causes disruption for the current users of these functions without addressing what many developers see as the fundamental problem. Discussion ---------- Links to past mailing list discussions, and to relevant GitHub issues and pull requests, have already been given. References and footnotes ------------------------ .. [1] Financial functions, https://numpy.org/doc/1.17/reference/routines.financial.html .. [2] Numpy-discussion mailing list, "Simple financial functions for NumPy", https://mail.python.org/pipermail/numpy-discussion/2008-April/032353.html .. [3] Numpy-discussion mailing list, "add xirr to numpy financial functions?", https://mail.python.org/pipermail/numpy-discussion/2009-May/042645.html .. [4] Numpy-discussion mailing list, "Definitions of pv, fv, nper, pmt, and rate", https://mail.python.org/pipermail/numpy-discussion/2009-June/043188.html .. [5] Get financial functions out of main namespace, https://github.com/numpy/numpy/issues/2880 .. [6] Numpy-discussion mailing list, "Deprecation of financial routines", https://mail.python.org/pipermail/numpy-discussion/2013-August/067409.html .. [7] ``component: numpy.lib.financial`` issues, https://github.com/numpy/numpy/issues?utf8=%E2%9C%93&q=is%3Aissue+label%3A%… .. [8] ``component: numpy.lib.financial`` pull request, https://github.com/numpy/numpy/pulls?utf8=%E2%9C%93&q=is%3Apr+label%3A%22co… .. [9] Quansight-Labs/python-api-inspect, https://github.com/Quansight-Labs/python-api-inspect/ Copyright --------- This document has been placed in the public domain.

5 7