<html><body>
Hello,<div><br></div><div>Very well written article! It takes a lot of important things into account. I think a number of things should be mentioned, if only in the alternatives:</div><div><ul style="font-style:normal;font-variant-caps:normal;font-weight:normal;font-stretch:normal;font-size:15px;line-height:17px;font-family:"Helvetica Neue";color:rgb(0,0,0);background-color:rgba(0,0,0,0)"><li style="text-decoration:none">One major version number change, with lots of “major version change” deprecations grouped into it, along with an LTS release.</li><li style="text-decoration:none">The possibility of another major version change (possibly the same one) where we re-write all portions that were agreed upon (via NEPs) to be re-written, with a longer LTS release (3 years? 5?).</li><ul style="text-decoration:none"><li>I’m thinking this one could be similar to the Python 2 -> Python 3 transition. Note that this is different from having constant breakages, this will be a mostly one-time effort and one-time breakage.</li><li>We break the ABI, but not most of the C API.</li><li>We port at least bug fixes and possibly oft-requested functionality to the old version for a long time.</li><li>But we fix all of the little things that are agreed upon by the community to be “missing” or “wrong” in the current release. It may be a while before this is adopted but it’ll be really beneficial in the long run.</li><li>We ping the dev-discussions of most major downstream users (SciPy, all the scikits, Matplotlib, etc.) for their “pain points” and also if they think this is a good idea. This way, the amount of users included aren’t just those on the NumPy mailing list.</li><li>We enforce good practices in our code. For example, we will explicitly disallow subclassing from ndarray, we get rid of scalars, we fix the type system.</li></ul></ul><div>This may sound radical (I myself think so), but consider that if we get rid of a large amount of technical debt on the onset, have a reputation for a clean code-base (rather than one that’s decades old), then we could onboard a lot more active developers and existing developers can also get a lot more work done. I may be getting ahead of myself on this, but feel free to leave your thoughts and opinions.</div></div><div>
<br>
<div class=""><div>Best regards,</div><div>Hameer Abbasi</div>
Sent from <a href="https://www.helloastro.com">Astro</a> for Mac
</div><br><blockquote class="hm_quoted_text" style="padding-left:8px;margin:0;border-left:1px solid rgb(185,185,185);color:rgb(100,100,100)">
<div>On 22. Jul 2018 at 01:48, Ralf Gommers <<a href="mailto:ralf.gommers@gmail.com">ralf.gommers@gmail.com</a>> wrote:</div><p>
<br></p><div dir="ltr"><div>Hi all,</div><div><br></div><div>Here is a first draft of a NEP on backwards compatibility and deprecation policy. This I think mostly formalized what we've done for the last couple of years, however I'm sure opinions and wish lists will differ here.<br></div><div><br></div><div>Pull request: <a href="https://github.com/numpy/numpy/pull/11596">https://github.com/numpy/numpy/pull/11596</a></div><div><br></div><div>Rendered version: <a href="https://github.com/rgommers/numpy/blob/nep-backcompat/doc/neps/nep-0023-backwards-compatibility.rst">https://github.com/rgommers/numpy/blob/nep-backcompat/doc/neps/nep-0023-backwards-compatibility.rst</a></div><div><br></div><div>Full text below (ducks).<br></div><div><br></div><div>Cheers,<br></div><div>Ralf</div><div><br></div><div><br></div><div>=======================================================<br>NEP 23 - Backwards compatibility and deprecation policy<br>=======================================================<br><br>:Author: Ralf Gommers <<a href="mailto:ralf.gommers@gmail.com">ralf.gommers@gmail.com</a>><br>:Status: Draft<br>:Type: Process<br>:Created: 2018-07-14<br>:Resolution: <url> (required for Accepted | Rejected | Withdrawn)<br><br>Abstract<br>--------<br><br>In this NEP we describe NumPy's approach to backwards compatibility,<br>its deprecation and removal policy, and the trade-offs and decision<br>processes for individual cases where breaking backwards compatibility<br>is considered.<br><br><br>Detailed description<br>--------------------<br><br>NumPy has a very large user base. Those users rely on NumPy being stable<br>and the code they write that uses NumPy functionality to keep working.<br>NumPy is also actively maintained and improved -- and sometimes improvements<br>require, or are made much easier, by breaking backwards compatibility.<br>Finally, there are trade-offs in stability for existing users vs. avoiding<br>errors or having a better user experience for new users. These competing<br>needs often give rise to heated debates and delays in accepting or rejecting<br>contributions. This NEP tries to address that by providing a policy as well<br>as examples and rationales for when it is or isn't a good idea to break<br>backwards compatibility.<br><br>General principles:<br><br>- Aim not to break users' code unnecessarily.<br>- Aim never to change code in ways that can result in users silently getting<br> incorrect results from their previously working code.<br>- Backwards incompatible changes can be made, provided the benefits outweigh<br> the costs.<br>- When assessing the costs, keep in mind that most users do not read the mailing<br> list, do not look at deprecation warnings, and sometimes wait more than one or<br> two years before upgrading from their old version. And that NumPy has<br> many hundreds of thousands or even a couple of million users, so "no one will<br> do or use this" is very likely incorrect.<br>- Benefits include improved functionality, usability and performance (in order<br> of importance), as well as lower maintenance cost and improved future<br> extensibility.<br>- Bug fixes are exempt from the backwards compatibility policy. However in case<br> of serious impact on users (e.g. a downstream library doesn't build anymore),<br> even bug fixes may have to be delayed for one or more releases.<br>- The Python API and the C API will be treated in the same way.<br><br><br>Examples<br>^^^^^^^^<br><br>We now discuss a number of concrete examples to illustrate typical issues<br>and trade-offs.<br><br>**Changing the behavior of a function**<br><br>``np.histogram`` is probably the most infamous example.<br>First, a new keyword ``new=False`` was introduced, this was then switched<br>over to None one release later, and finally it was removed again.<br>Also, it has a ``normed`` keyword that had behavior that could be considered<br>either suboptimal or broken (depending on ones opinion on the statistics).<br>A new keyword ``density`` was introduced to replace it; ``normed`` started giving<br>``DeprecationWarning`` only in v.1.15.0. Evolution of ``histogram``::<br><br> def histogram(a, bins=10, range=None, normed=False): # v1.0.0<br><br> def histogram(a, bins=10, range=None, normed=False, weights=None, new=False): #v1.1.0<br><br> def histogram(a, bins=10, range=None, normed=False, weights=None, new=None): #v1.2.0<br><br> def histogram(a, bins=10, range=None, normed=False, weights=None): #v1.5.0<br><br> def histogram(a, bins=10, range=None, normed=False, weights=None, density=None): #v1.6.0<br><br> def histogram(a, bins=10, range=None, normed=None, weights=None, density=None): #v1.15.0<br> # v1.15.0 was the first release where `normed` started emitting<br> # DeprecationWarnings<br><br>The ``new`` keyword was planned from the start to be temporary; such a plan<br>forces users to change their code more than once. Such keywords (there have<br>been other instances proposed, e.g. ``legacy_index`` in<br>`NEP 21 <<a href="http://www.numpy.org/neps/nep-0021-advanced-indexing.html">http://www.numpy.org/neps/nep-0021-advanced-indexing.html</a>>`_) are not<br>desired. The right thing to have done here would probably have been to<br>deprecate ``histogram`` and introduce a new function ``hist`` in its place.<br><br>**Returning a view rather than a copy**<br><br>The ``ndarray.diag`` method used to return a copy. A view would be better for<br>both performance and design consistency. This change was warned about<br>(``FutureWarning``) in v.8.0, and in v1.9.0 ``diag`` was changed to return<br>a *read-only* view. The planned change to a writeable view in v1.10.0 was<br>postponed due to backwards compatibility concerns, and is still an open issue<br>(gh-7661).<br><br>What should have happened instead: nothing. This change resulted in a lot of<br>discussions and wasted effort, did not achieve its final goal, and was not that<br>important in the first place. Finishing the change to a *writeable* view in<br>the future is not desired, because it will result in users silently getting<br>different results if they upgraded multiple versions or simply missed the<br>warnings.<br><br>**Disallowing indexing with floats**<br><br>Indexing an array with floats is asking for something ambiguous, and can be a<br>sign of a bug in user code. After some discussion, it was deemed a good idea<br>to deprecate indexing with floats. This was first tried for the v1.8.0<br>release, however in pre-release testing it became clear that this would break<br>many libraries that depend on NumPy. Therefore it was reverted before release,<br>to give those libraries time to fix their code first. It was finally<br>introduced for v1.11.0 and turned into a hard error for v1.12.0.<br><br>This change was disruptive, however it did catch real bugs in e.g. SciPy and<br>scikit-learn. Overall the change was worth the cost, and introducing it in<br>master first to allow testing, then removing it again before a release, is a<br>useful strategy.<br><br>Similar recent deprecations also look like good examples of<br>cleanups/improvements:<br><br>- removing deprecated boolean indexing (gh-8312)<br>- deprecating truth testing on empty arrays (gh-9718)<br>- deprecating ``np.sum(generator)`` (gh-10670, one issue with this one is that<br> its warning message is wrong - this should error in the future).<br><br>**Removing the financial functions**<br><br>The financial functions (e.g. ``np.pmt``) are badly named, are present in the<br>main NumPy namespace, and don't really fit well with NumPy's scope.<br>They were added in 2008 after<br>`a discussion <<a href="https://mail.python.org/pipermail/numpy-discussion/2008-April/032353.html">https://mail.python.org/pipermail/numpy-discussion/2008-April/032353.html</a>>`_<br>on the mailing list where opinion was divided (but a majority in favor).<br>At the moment these functions don't cause a lot of overhead, however there are<br>multiple issues and PRs a year for them which cost maintainer time to deal<br>with. And they clutter up the ``numpy`` namespace. Discussion in 2013 happened<br>on removing them again (gh-2880).<br><br>This case is borderline, but given that they're clearly out of scope,<br>deprecation and removal out of at least the main ``numpy`` namespace can be<br>proposed. Alternatively, document clearly that new features for financial<br>functions are unwanted, to keep the maintenance costs to a minimum.<br><br>**Examples of features not added because of backwards compatibility**<br><br>TODO: do we have good examples here? Possibly subclassing related?<br><br><br>Removing complete submodules<br>^^^^^^^^^^^^^^^^^^^^^^^^^^^^<br><br>This year there have been suggestions to consider removing some or all of<br>``numpy.distutils``, ``numpy.f2py``, ``numpy.linalg``, and ``numpy.random``.<br>The motivation was that all these cost maintenance effort, and that they slow<br>down work on the core of Numpy (ndarrays, dtypes and ufuncs).<br><br>The import on downstream libraries and users would be very large, and<br>maintenance of these modules would still have to happen. Therefore this is<br>simply not a good idea; removing these submodules should not happen even for<br>a new major version of NumPy.<br><br><br>Subclassing of ndarray<br>^^^^^^^^^^^^^^^^^^^^^^<br><br>Subclassing of ``ndarray`` is a pain point. ``ndarray`` was not (or at least<br>not well) designed to be subclassed. Despite that, a lot of subclasses have<br>been created even within the NumPy code base itself, and some of those (e.g.<br>``MaskedArray``, ``astropy.units.Quantity``) are quite popular. The main<br>problems with subclasses are:<br><br>- They make it hard to change ``ndarray`` in ways that would otherwise be<br> backwards compatible.<br>- Some of them change the behavior of ndarray methods, making it difficult to<br> write code that accepts array duck-types.<br><br>Subclassing ``ndarray`` has been officially discouraged for a long time. Of<br>the most important subclasses, ``np.matrix`` will be deprecated (see gh-10142)<br>and ``MaskedArray`` will be kept in NumPy (`NEP 17<br><<a href="http://www.numpy.org/neps/nep-0017-split-out-maskedarray.html">http://www.numpy.org/neps/nep-0017-split-out-maskedarray.html</a>>`_).<br>``MaskedArray`` will ideally be rewritten in a way such that it uses only<br>public NumPy APIs. For subclasses outside of NumPy, more work is needed to<br>provide alternatives (e.g. mixins, see gh-9016 and gh-10446) or better support<br>for custom dtypes (see gh-2899). Until that is done, subclasses need to be<br>taken into account when making change to the NumPy code base. A future change<br>in NumPy to not support subclassing will certainly need a major version<br>increase.<br><br><br>Policy<br>------<br><br>1. Code changes that have the potential to silently change the results of a users'<br> code must never be made (except in the case of clear bugs).<br>2. Code changes that break users' code (i.e. the user will see a clear exception)<br> can be made, *provided the benefit is worth the cost* and suitable deprecation<br> warnings have been raised first.<br>3. Deprecation warnings are in all cases warnings that functionality will be removed.<br> If there is no intent to remove functionlity, then deprecation in documentation<br> only or other types of warnings shall be used.<br>4. Deprecations for stylistic reasons (e.g. consistency between functions) are<br> strongly discouraged.<br><br>Deprecations:<br><br>- shall include the version numbers of both when the functionality was deprecated<br> and when it will be removed (either two releases after the warning is<br> introduced, or in the next major version).<br>- shall include information on alternatives to the deprecated functionality, or a<br> reason for the deprecation if no clear alternative is available.<br>- shall use ``VisibleDeprecationWarning`` rather than ``DeprecationWarning``<br> for cases of relevance to end users (as opposed to cases only relevant to<br> libraries building on top of NumPy).<br>- shall be listed in the release notes of the release where the deprecation happened.<br><br>Removal of deprecated functionality:<br><br>- shall be done after 2 releases (assuming a 6-monthly release cycle; if that changes,<br> there shall be at least 1 year between deprecation and removal), unless the<br> impact of the removal is such that a major version number increase is<br> warranted.<br>- shall be listed in the release notes of the release where the removal happened.<br><br>Versioning:<br><br>- removal of deprecated code can be done in any minor (but not bugfix) release.<br>- for heavily used functionality (e.g. removal of ``np.matrix``, of a whole submodule,<br> or significant changes to behavior for subclasses) the major version number shall<br> be increased.<br><br>In concrete cases where this policy needs to be applied, decisions are made according<br>to the `NumPy governance model<br><<a href="https://docs.scipy.org/doc/numpy/dev/governance/index.html">https://docs.scipy.org/doc/numpy/dev/governance/index.html</a>>`_.<br><br>Functionality with more strict policies:<br><br>- ``numpy.random`` has its own backwards compatibility policy,<br> see `NEP 19 <<a href="http://www.numpy.org/neps/nep-0019-rng-policy.html">http://www.numpy.org/neps/nep-0019-rng-policy.html</a>>`_.<br>- The file format for ``.npy`` and ``.npz`` files must not be changed in a backwards<br> incompatible way.<br><br><br>Alternatives<br>------------<br><br>**Being more agressive with deprecations.**<br><br>The goal of being more agressive is to allow NumPy to move forward faster.<br>This would avoid others inventing their own solutions (often in multiple<br>places), as well as be a benefit to users without a legacy code base. We<br>reject this alternative because of the place NumPy has in the scientific Python<br>ecosystem - being fairly conservative is required in order to not increase the<br>extra maintenance for downstream libraries and end users to an unacceptable<br>level.<br><br>**Semantic versioning.**<br><br>This would change the versioning scheme for code removals; those could then<br>only be done when the major version number is increased. Rationale for<br>rejection: semantic versioning is relatively common in software engineering,<br>however it is not at all common in the Python world. Also, it would mean that<br>NumPy's version number simply starts to increase faster, which would be more<br>confusing than helpful. gh-10156 contains more discussion on this alternative.<br><br><br>Discussion<br>----------<br><br>TODO<br><br>This section may just be a bullet list including links to any discussions<br>regarding the NEP:<br><br>- This includes links to mailing list threads or relevant GitHub issues.<br><br><br>References and Footnotes<br>------------------------<br><br>.. [1] TODO<br><br><br>Copyright<br>---------<br><br>This document has been placed in the public domain. [1]_<br></div><div><br></div><div><br></div></div>
<br><span style="white-space:pre-wrap" class="hm_plaintext"><div>_______________________________________________<br>NumPy-Discussion mailing list<br><a href="mailto:NumPy-Discussion@python.org">NumPy-Discussion@python.org</a><br><a href="https://mail.python.org/mailman/listinfo/numpy-discussion">https://mail.python.org/mailman/listinfo/numpy-discussion</a><br></div></span>
</blockquote>
</div></body></html>