Mailman 3 July 2018 - NumPy-Discussion

mixed mode arithmetic
by Neal Becker July 11, 2023

July 11, 2023

I've been browsing the numpy source. I'm wondering about mixed-mode arithmetic on arrays. I believe the way numpy handles this is that it never does mixed arithmetic, but instead converts arrays to a common type. Arguably, that might be efficient for a mix of say, double and float. Maybe not. But for a mix of complex and a scalar type (say, CDouble * Double), it's clearly suboptimal in efficiency. So, do I understand this correctly? If so, is that something we should improve?

4 6

Invalid value encoutered : how to prevent numpy.where to do this?
by Eric Emsellem Feb. 18, 2023

Feb. 18, 2023

Dear all, I have a code using lots of "numpy.where" to make some constrained calculations as in: data = arange(10) result = np.where(data == 0, 0., 1./data) # or data1 = arange(10) data2 = arange(10)+1.0 result = np.where(data1 > data2, np.sqrt(data1-data2), np.sqrt(data2-data2)) which then produces warnings like: /usr/bin/ipython:1: RuntimeWarning: invalid value encountered in sqrt or for the first example: /usr/bin/ipython:1: RuntimeWarning: divide by zero encountered in divide How … [View More]

5 4

Re: [Numpy-discussion] example reading binary Fortran file
by Neil Martinsen-Burrell July 22, 2021

July 22, 2021

David Froger <david.froger.info <at> gmail.com> writes: > Hy,My question is about reading Fortran binary file (oh no this question > again...) I've posted this before, but I finally got it cleaned up for the Cookbook. For this purpose I use a subclass of file that has methods for reading unformatted Fortran data. See http://www.scipy.org/Cookbook/FortranIO/FortranFile. I'd gladly see this in numpy or scipy somewhere, but I'm not sure where it belongs. > program … [View More]makeArray > implicit none > integer,parameter:: nx=10,ny=20 > real(4),dimension(nx,ny):: ux,uy,p > integer :: i,j > open(11,file='uxuyp.bin',form='unformatted') > do i = 1,nx > do j = 1,ny > ux(i,j) = real(i*j) > uy(i,j) = real(i)/real(j) > p (i,j) = real(i) + real(j) > enddo > enddo > write(11) ux,uy > write(11) p > close(11) > end program makeArray When I run the above program compiled with gfortran on my Intel Mac, I can read it back with:: >>> import numpy as np >>> from fortranfile import FortranFile >>> f=FortranFile('uxuyp.bin', endian='<') >>> uxuy = f.readReals(prec='f') # 'f' for default reals >>> len(uxuy) 400 >>> ux = np.array(uxuy[:200]).reshape((20,10)).T >>> uy = np.array(uxuy[200:]).reshape((20,10)).T >>> p = f.readReals('f').reshape((20,10)).T >>> ux array([[ 1., 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20.], [ 2., 4., 6., 8., 10., 12., 14., 16., 18., 20., 22., 24., 26., 28., 30., 32., 34., 36., 38., 40.], [ 3., 6., 9., 12., 15., 18., 21., 24., 27., 30., 33., 36., 39., 42., 45., 48., 51., 54., 57., 60.], [ 4., 8., 12., 16., 20., 24., 28., 32., 36., 40., 44., 48., 52., 56., 60., 64., 68., 72., 76., 80.], [ 5., 10., 15., 20., 25., 30., 35., 40., 45., 50., 55., 60., 65., 70., 75., 80., 85., 90., 95., 100.], [ 6., 12., 18., 24., 30., 36., 42., 48., 54., 60., 66., 72., 78., 84., 90., 96., 102., 108., 114., 120.], [ 7.,Proxy-Connection: keep-alive Cache-Control: max-age=0 14., 21., 28., 35., 42., 49., 56., 63., 70., 77., 84., 91., 98., 105., 112., 119., 126., 133., 140.], [ 8., 16., 24., 32., 40., 48., 56., 64., 72., 80., 88., 96., 104., 112., 120., 128., 136., 144., 152., 160.], [ 9., 18., 27., 36., 45., 54., 63., 72., 81., 90., 99., 108., 117., 126., 135., 144., 153., 162., 171., 180.], [ 10., 20., 30., 40., 50., 60., 70., 80., 90., 100., 110., 120., 130., 140., 150., 160., 170., 180., 190., 200.]]) >>> uy array([[ 1. , 0.5 , 0.33333334, 0.25 , 0.2 , 0.16666667, 0.14285715, 0.125 , 0.11111111, 0.1 , 0.09090909, 0.08333334, 0.07692308, 0.07142857, 0.06666667, 0.0625 , 0.05882353, 0.05555556, 0.05263158, 0.05 ], [ 2. , 1. , 0.66666669, 0.5 , 0.40000001, 0.33333334, 0.2857143 , 0.25 , 0.22222222, 0.2 , 0.18181819, 0.16666667, 0.15384616, 0.14285715, 0.13333334, 0.125 , 0.11764706, 0.11111111, 0.10526316, 0.1 ], [ 3. , 1.5 , 1. , 0.75 , 0.60000002, 0.5 , 0.42857143, 0.375 , 0.33333334, 0.30000001, 0.27272728, 0.25 , 0.23076923, 0.21428572, 0.2 , 0.1875 , 0.17647059, 0.16666667, 0.15789473, 0.15000001], [ 4. , 2. , 1.33333337, 1. , 0.80000001, 0.66666669, 0.5714286 , 0.5 , 0.44444445, 0.40000001, 0.36363637, 0.33333334, 0.30769232, 0.2857143 , 0.26666668, 0.25 , 0.23529412, 0.22222222, 0.21052632, 0.2 ], [ 5. , 2.5 , 1.66666663, 1.25 , 1. , 0.83333331, 0.71428573, 0.625 , 0.55555558, 0.5 , 0.45454547, 0.41666666, 0.38461539, 0.35714287, 0.33333334, 0.3125 , 0.29411766, 0.27777779, 0.2631579 , 0.25 ], [ 6. , 3. , 2. , 1.5 , 1.20000005, 1. , 0.85714287, 0.75 , 0.66666669, 0.60000002, 0.54545456, 0.5 , 0.46153846, 0.42857143, 0.40000001, 0.375 , 0.35294119, 0.33333334, 0.31578946, 0.30000001], [ 7. , 3.5 , 2.33333325, 1.75 , 1.39999998, 1.16666663, 1. , 0.875 , 0.77777779, 0.69999999, 0.63636363, 0.58333331, 0.53846157, 0.5 , 0.46666667, 0.4375 , 0.41176471, 0.3888889 , 0.36842105, 0.34999999], [ 8. , 4. , 2.66666675, 2. , 1.60000002, 1.33333337, 1.14285719, 1. , 0.8888889 , 0.80000001, 0.72727275, 0.66666669, 0.61538464, 0.5714286 , 0.53333336, 0.5 , 0.47058824, 0.44444445, 0.42105263, 0.40000001], [ 9. , 4.5 , 3. , 2.25 , 1.79999995, 1.5 , 1.28571427, 1.125 , 1. , 0.89999998, 0.81818181, 0.75 , 0.69230771, 0.64285713, 0.60000002, 0.5625 , 0.52941179, 0.5 , 0.47368422, 0.44999999], [ 10. , 5. , 3.33333325, 2.5 , 2. , 1.66666663, 1.42857146, 1.25 , 1.11111116, 1. , 0.90909094, 0.83333331, 0.76923078, 0.71428573, 0.66666669, 0.625 , 0.58823532, 0.55555558, 0.52631581, 0.5 ]]) >>> p array([[ 2., 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21.], [ 3., 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22.], [ 4., 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23.], [ 5., 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24.], [ 6., 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25.], [ 7., 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26.], [ 8., 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27.], [ 9., 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28.], [ 10., 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29.], [ 11., 12., 13., 14., 15., 16., 17., 18., 19., 20., 21., 22., 23., 24., 25., 26., 27., 28., 29., 30.]]) Note that you have to provide the shape information for ux and uy because fortran writes them together as a stream of 400 numbers. -Neil [View Less]

10 10

Adoption of a Code of Conduct
by Stefan van der Walt Aug. 3, 2018

Aug. 3, 2018

Hi everyone, A while ago, SciPy (the library) adopted its Code of Conduct: https://docs.scipy.org/doc/scipy/reference/dev/conduct/code_of_conduct.html We worked hard to make that document friendly, while at the same time stating clearly the kinds of behavior that would and would not be tolerated. I propose that we adopt the SciPy code of conduct for NumPy as well. It is a good way to signal to newcomers that this is a community that cares about how people are treated. And I think we should … [View More]

20 51

Roadmap proposal, v3
by Stefan van der Walt Aug. 2, 2018

Aug. 2, 2018

Hi everyone, Please take a look at the latest roadmap proposal: https://github.com/numpy/numpy/pull/11611 This is a living document, so can easily be modified in the future, but we'd like to get in place a document that corresponds fairly closely with current community priorities. Best regards, Stéfan

7 11

backwards compatibility and deprecation policy NEP
by Ralf Gommers July 30, 2018

July 30, 2018

Hi all, Here is a first draft of a NEP on backwards compatibility and deprecation policy. This I think mostly formalized what we've done for the last couple of years, however I'm sure opinions and wish lists will differ here. Pull request: https://github.com/numpy/numpy/pull/11596 Rendered version: https://github.com/rgommers/numpy/blob/nep-backcompat/doc/neps/nep-0023-bac… Full text below (ducks). Cheers, Ralf ======================================================= NEP 23 - Backwards … [View More]compatibility and deprecation policy ======================================================= :Author: Ralf Gommers <ralf.gommers(a)gmail.com> :Status: Draft :Type: Process :Created: 2018-07-14 :Resolution: <url> (required for Accepted | Rejected | Withdrawn) Abstract -------- In this NEP we describe NumPy's approach to backwards compatibility, its deprecation and removal policy, and the trade-offs and decision processes for individual cases where breaking backwards compatibility is considered. Detailed description -------------------- NumPy has a very large user base. Those users rely on NumPy being stable and the code they write that uses NumPy functionality to keep working. NumPy is also actively maintained and improved -- and sometimes improvements require, or are made much easier, by breaking backwards compatibility. Finally, there are trade-offs in stability for existing users vs. avoiding errors or having a better user experience for new users. These competing needs often give rise to heated debates and delays in accepting or rejecting contributions. This NEP tries to address that by providing a policy as well as examples and rationales for when it is or isn't a good idea to break backwards compatibility. General principles: - Aim not to break users' code unnecessarily. - Aim never to change code in ways that can result in users silently getting incorrect results from their previously working code. - Backwards incompatible changes can be made, provided the benefits outweigh the costs. - When assessing the costs, keep in mind that most users do not read the mailing list, do not look at deprecation warnings, and sometimes wait more than one or two years before upgrading from their old version. And that NumPy has many hundreds of thousands or even a couple of million users, so "no one will do or use this" is very likely incorrect. - Benefits include improved functionality, usability and performance (in order of importance), as well as lower maintenance cost and improved future extensibility. - Bug fixes are exempt from the backwards compatibility policy. However in case of serious impact on users (e.g. a downstream library doesn't build anymore), even bug fixes may have to be delayed for one or more releases. - The Python API and the C API will be treated in the same way. Examples ^^^^^^^^ We now discuss a number of concrete examples to illustrate typical issues and trade-offs. **Changing the behavior of a function** ``np.histogram`` is probably the most infamous example. First, a new keyword ``new=False`` was introduced, this was then switched over to None one release later, and finally it was removed again. Also, it has a ``normed`` keyword that had behavior that could be considered either suboptimal or broken (depending on ones opinion on the statistics). A new keyword ``density`` was introduced to replace it; ``normed`` started giving ``DeprecationWarning`` only in v.1.15.0. Evolution of ``histogram``:: def histogram(a, bins=10, range=None, normed=False): # v1.0.0 def histogram(a, bins=10, range=None, normed=False, weights=None, new=False): #v1.1.0 def histogram(a, bins=10, range=None, normed=False, weights=None, new=None): #v1.2.0 def histogram(a, bins=10, range=None, normed=False, weights=None): #v1.5.0 def histogram(a, bins=10, range=None, normed=False, weights=None, density=None): #v1.6.0 def histogram(a, bins=10, range=None, normed=None, weights=None, density=None): #v1.15.0 # v1.15.0 was the first release where `normed` started emitting # DeprecationWarnings The ``new`` keyword was planned from the start to be temporary; such a plan forces users to change their code more than once. Such keywords (there have been other instances proposed, e.g. ``legacy_index`` in `NEP 21 <http://www.numpy.org/neps/nep-0021-advanced-indexing.html>`_) are not desired. The right thing to have done here would probably have been to deprecate ``histogram`` and introduce a new function ``hist`` in its place. **Returning a view rather than a copy** The ``ndarray.diag`` method used to return a copy. A view would be better for both performance and design consistency. This change was warned about (``FutureWarning``) in v.8.0, and in v1.9.0 ``diag`` was changed to return a *read-only* view. The planned change to a writeable view in v1.10.0 was postponed due to backwards compatibility concerns, and is still an open issue (gh-7661). What should have happened instead: nothing. This change resulted in a lot of discussions and wasted effort, did not achieve its final goal, and was not that important in the first place. Finishing the change to a *writeable* view in the future is not desired, because it will result in users silently getting different results if they upgraded multiple versions or simply missed the warnings. **Disallowing indexing with floats** Indexing an array with floats is asking for something ambiguous, and can be a sign of a bug in user code. After some discussion, it was deemed a good idea to deprecate indexing with floats. This was first tried for the v1.8.0 release, however in pre-release testing it became clear that this would break many libraries that depend on NumPy. Therefore it was reverted before release, to give those libraries time to fix their code first. It was finally introduced for v1.11.0 and turned into a hard error for v1.12.0. This change was disruptive, however it did catch real bugs in e.g. SciPy and scikit-learn. Overall the change was worth the cost, and introducing it in master first to allow testing, then removing it again before a release, is a useful strategy. Similar recent deprecations also look like good examples of cleanups/improvements: - removing deprecated boolean indexing (gh-8312) - deprecating truth testing on empty arrays (gh-9718) - deprecating ``np.sum(generator)`` (gh-10670, one issue with this one is that its warning message is wrong - this should error in the future). **Removing the financial functions** The financial functions (e.g. ``np.pmt``) are badly named, are present in the main NumPy namespace, and don't really fit well with NumPy's scope. They were added in 2008 after `a discussion < https://mail.python.org/pipermail/numpy-discussion/2008-April/032353.html>`_ on the mailing list where opinion was divided (but a majority in favor). At the moment these functions don't cause a lot of overhead, however there are multiple issues and PRs a year for them which cost maintainer time to deal with. And they clutter up the ``numpy`` namespace. Discussion in 2013 happened on removing them again (gh-2880). This case is borderline, but given that they're clearly out of scope, deprecation and removal out of at least the main ``numpy`` namespace can be proposed. Alternatively, document clearly that new features for financial functions are unwanted, to keep the maintenance costs to a minimum. **Examples of features not added because of backwards compatibility** TODO: do we have good examples here? Possibly subclassing related? Removing complete submodules ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ This year there have been suggestions to consider removing some or all of ``numpy.distutils``, ``numpy.f2py``, ``numpy.linalg``, and ``numpy.random``. The motivation was that all these cost maintenance effort, and that they slow down work on the core of Numpy (ndarrays, dtypes and ufuncs). The import on downstream libraries and users would be very large, and maintenance of these modules would still have to happen. Therefore this is simply not a good idea; removing these submodules should not happen even for a new major version of NumPy. Subclassing of ndarray ^^^^^^^^^^^^^^^^^^^^^^ Subclassing of ``ndarray`` is a pain point. ``ndarray`` was not (or at least not well) designed to be subclassed. Despite that, a lot of subclasses have been created even within the NumPy code base itself, and some of those (e.g. ``MaskedArray``, ``astropy.units.Quantity``) are quite popular. The main problems with subclasses are: - They make it hard to change ``ndarray`` in ways that would otherwise be backwards compatible. - Some of them change the behavior of ndarray methods, making it difficult to write code that accepts array duck-types. Subclassing ``ndarray`` has been officially discouraged for a long time. Of the most important subclasses, ``np.matrix`` will be deprecated (see gh-10142) and ``MaskedArray`` will be kept in NumPy (`NEP 17 <http://www.numpy.org/neps/nep-0017-split-out-maskedarray.html>`_). ``MaskedArray`` will ideally be rewritten in a way such that it uses only public NumPy APIs. For subclasses outside of NumPy, more work is needed to provide alternatives (e.g. mixins, see gh-9016 and gh-10446) or better support for custom dtypes (see gh-2899). Until that is done, subclasses need to be taken into account when making change to the NumPy code base. A future change in NumPy to not support subclassing will certainly need a major version increase. Policy ------ 1. Code changes that have the potential to silently change the results of a users' code must never be made (except in the case of clear bugs). 2. Code changes that break users' code (i.e. the user will see a clear exception) can be made, *provided the benefit is worth the cost* and suitable deprecation warnings have been raised first. 3. Deprecation warnings are in all cases warnings that functionality will be removed. If there is no intent to remove functionlity, then deprecation in documentation only or other types of warnings shall be used. 4. Deprecations for stylistic reasons (e.g. consistency between functions) are strongly discouraged. Deprecations: - shall include the version numbers of both when the functionality was deprecated and when it will be removed (either two releases after the warning is introduced, or in the next major version). - shall include information on alternatives to the deprecated functionality, or a reason for the deprecation if no clear alternative is available. - shall use ``VisibleDeprecationWarning`` rather than ``DeprecationWarning`` for cases of relevance to end users (as opposed to cases only relevant to libraries building on top of NumPy). - shall be listed in the release notes of the release where the deprecation happened. Removal of deprecated functionality: - shall be done after 2 releases (assuming a 6-monthly release cycle; if that changes, there shall be at least 1 year between deprecation and removal), unless the impact of the removal is such that a major version number increase is warranted. - shall be listed in the release notes of the release where the removal happened. Versioning: - removal of deprecated code can be done in any minor (but not bugfix) release. - for heavily used functionality (e.g. removal of ``np.matrix``, of a whole submodule, or significant changes to behavior for subclasses) the major version number shall be increased. In concrete cases where this policy needs to be applied, decisions are made according to the `NumPy governance model <https://docs.scipy.org/doc/numpy/dev/governance/index.html>`_. Functionality with more strict policies: - ``numpy.random`` has its own backwards compatibility policy, see `NEP 19 <http://www.numpy.org/neps/nep-0019-rng-policy.html>`_. - The file format for ``.npy`` and ``.npz`` files must not be changed in a backwards incompatible way. Alternatives ------------ **Being more agressive with deprecations.** The goal of being more agressive is to allow NumPy to move forward faster. This would avoid others inventing their own solutions (often in multiple places), as well as be a benefit to users without a legacy code base. We reject this alternative because of the place NumPy has in the scientific Python ecosystem - being fairly conservative is required in order to not increase the extra maintenance for downstream libraries and end users to an unacceptable level. **Semantic versioning.** This would change the versioning scheme for code removals; those could then only be done when the major version number is increased. Rationale for rejection: semantic versioning is relatively common in software engineering, however it is not at all common in the Python world. Also, it would mean that NumPy's version number simply starts to increase faster, which would be more confusing than helpful. gh-10156 contains more discussion on this alternative. Discussion ---------- TODO This section may just be a bullet list including links to any discussions regarding the NEP: - This includes links to mailing list threads or relevant GitHub issues. References and Footnotes ------------------------ .. [1] TODO Copyright --------- This document has been placed in the public domain. [1]_ [View Less]

6 25

NEP 15, 20 implementations waiting for review
by Matti Picus July 27, 2018

July 27, 2018

Two largish pull requests that implement approved NEPS are waiting for review: https://github.com/numpy/numpy/pull/11175 for expanded gufunc signatures (NEP 20) https://github.com/numpy/numpy/pull/10915 for merging multiarray and umath c-extension modules (NEP 15) I realize reviewer time is precious, and appreciate that these PRs are perhaps more complex to review than others, but it would be nice to keep the good momentum we have in the NEP process moving forward. Thanks, Matti

1 0

update to numpy-1.5.0 gives new warnings from scipy
by Neal Becker July 25, 2018

July 25, 2018

After update to numpy-1.5.0, I'm getting warnings from scipy. These probably come from my code using convolve. Does scipy need updating? /home/nbecker/.local/lib/python3.6/site-packages/scipy/fftpack/basic.py:160: FutureWarning: Using a non-tuple sequence for multidimensional indexing is deprecated; use `arr[tuple(seq)]` instead of `arr[seq]`. In the future this will be interpreted as an array index, `arr[np.array(seq)]`, which will result either in an error or a different result. z[index] =… [View More]

3 2

Commit privileges
by Charles R Harris July 23, 2018

July 23, 2018

Hi All, The NumPy Steering Council has been looking at commit rights for the NumPy developers hired at BIDS. We would like them to be able to label PRs, close issues, and merge simple fixes; doing that requires commit privileges. OTOH, it is also the case that people paid to work on NumPy don't automatically receive commit privileges. So it is a bit a quandary and we don't seem to have an official document codifying the giving of commit privileges, and the Github privileges are rather coarse … [View More]

3 2

mplcairo 0.1 release
by Antony Lee July 23, 2018

July 23, 2018

Dear all, I am pleased to announce the release of mplcairo 0.1 # Description mplcairo is a Matplotlib backend based on the well-known cairo library, supporting output to both raster (including interactively) and vector formats. In other words, it provides the functionality of Matplotlib's {,qt5,gtk3,wx,tk,macos}{agg,cairo}, pdf, ps, and svg backends. Per Matplotlib's standard API, the backend can be selected by calling matplotlib.use("module://mplcairo.qt") or setting your MPLBACKEND … [View More]

2 1