On Wed, Dec 30, 2020 at 4:05 PM Ilhan Polat wrote:

Hi Ralf,

This reads really nice. Thanks to everyone who contributed.

Before nitpicking here and there, and sticking my head for others, is this is a finished discussion and only stylistic feedback is expected?

It's not. I removed everything that was controversial last time around and only added things that are basically the way we already do things, so I don't really expect major issues. But it is a big rewrite, so some discussion is certainly expected. Also is it preferred here or in the PR? GitHub is really not designed for

extended discussions and here it if there are two subjects are discussed simultaneously it just becomes difficult to follow (maybe it's a bias due to my dislike of mailing lists).

Agreed. The idea is we post NEPs to the mailing list, and major issues get discussed here. If there's smaller comments like stylistic things or small errors, commenting on the PR for those is much easier.

One of the less mentioned point is about what the tipping point is for the benefits outweighing the compatibility breakage sin and how to get a feeling for it. Because for a typical user, every break is just a break. Nobody will squint their eyes to see the reasoning behind it downstream. Thus this is more of a declaration of "yes as maintainers we are ready for facing the consequences but it had to be done because such and such".

That's very hard to describe, since it relies so much on previous experience and qualitative judgements. That's the main reason why I had more examples before, but they just led to more discussion about those examples - so that didn't quite have the intended effect. I am not asking to initiate a power discussion ala "who has the mod hammer"

but rather what constitutes as a valid business case for a breakage proposal. A few generic lines about that would go a long way. Because we are in the same situation with scipy.linalg in which, what to do is crystal clear but how to do it without breaking anything is herding the cats hence I am genuinely curious how to go about this.

If anyone has a good proposal, that'd be great. But I find it hard to come up with those few lines right now. Cheers, Ralf

Best, ilhan

On Wed, Dec 30, 2020 at 3:07 PM Ralf Gommers wrote:

...

Hi all,

Here is a long overdue update of the draft NEP about backwards compatibility and deprecation policy: https://github.com/numpy/numpy/pull/18097

- This is NEP 23: https://numpy.org/neps/nep-0023-backwards-compatibility.html - Link to the previous mailing list discussion: https://mail.python.org/pipermail/numpy-discussion/2018-July/078432.html

It would be nice to get this NEP to Accepted status. Main changes are:

- Removed all examples that people objected to - Removed all content regarding versioning - Restructured sections, and added "Strategies related to deprecations" (using suggestions by @njsmith and @shoyer). - Added concrete examples of deprecations, and a more thorough description of how to go about adding warnings incl. Sphinx directives, using `stacklevel`, etc.

As always, feedback here or on the PR is very welcome!

Cheers, Ralf

Abstract --------

In this NEP we describe NumPy's approach to backwards compatibility, its deprecation and removal policy, and the trade-offs and decision processes for individual cases where breaking backwards compatibility is considered.

Motivation and Scope --------------------

NumPy has a very large user base. Those users rely on NumPy being stable and the code they write that uses NumPy functionality to keep working. NumPy is also actively maintained and improved -- and sometimes improvements require, or are made much easier by, breaking backwards compatibility. Finally, there are trade-offs in stability for existing users vs. avoiding errors or having a better user experience for new users. These competing needs often give rise to long debates and to delays in accepting or rejecting contributions. This NEP tries to address that by providing a policy as well as examples and rationales for when it is or isn't a good idea to break backwards compatibility.

In scope for this NEP are:

- Principles of NumPy's approach to backwards compatibility. - How to deprecate functionality, and when to remove already deprecated functionality. - Decision making process for deprecations and removals.

Out of scope are:

- Making concrete decisions about deprecations of particular functionality. - NumPy's versioning scheme.

General principles ------------------

When considering proposed changes that are backwards incompatible, the main principles the NumPy developers use when making a decision are:

1. Changes need to benefit users more than they harm them. 2. NumPy is widely used so breaking changes should by default be assumed to be fairly harmful. 3. Decisions should be based on data and actual effects on users and downstream packages rather than, e.g., appealing to the docs or for stylistic reasons. 4. Silently getting a wrong answer is much worse than getting a loud error.

When assessing the costs of proposed changes, keep in mind that most users do not read the mailing list, do not look at deprecation warnings, and sometimes wait more than one or two years before upgrading from their old version. And that NumPy has millions of users, so "no one will do or use this" is very likely incorrect.

Benefits include improved functionality, usability and performance, as well as lower maintenance cost and improved future extensibility.

Fixes for clear bugs are exempt from this backwards compatibility policy. However in case of serious impact on users (e.g. a downstream library doesn't build anymore or would start giving incorrect results), even bug fixes may have to be delayed for one or more releases.

Strategies related to deprecations ----------------------------------

Getting hard data on the impact of a deprecation of often difficult. Strategies that can be used to assess such impact include:

- Use a code search engine ([1]_) or static ([2]_) or dynamic ([3]_) code analysis tools to determine where and how the functionality is used. - Testing prominent downstream libraries against a development build of NumPy containing the proposed change to get real-world data on its impact. - Making a change in master and reverting it, if needed, before a release. We do encourage other packages to test against NumPy's master branch, so this often turns up issues quickly.

If the impact is unclear or significant, it is often good to consider alternatives to deprecations. For example discouraging use in documentation only, or moving the documentation for the functionality to a less prominent place or even removing it completely. Commenting on open issues related to it that they are low-prio or labeling them as "wontfix" will also be a signal to users, and reduce the maintenance effort needing to be spent.

Implementing deprecations and removals --------------------------------------

Deprecation warnings are necessary in all cases where functionality will eventually be removed. If there is no intent to remove functionality, then it should not be deprecated either. A "please don't use this" in the documentation or other type of warning should be used instead.

Deprecations:

- shall include the version number of the release in which the functionality was deprecated. - shall include information on alternatives to the deprecated functionality, or a reason for the deprecation if no clear alternative is available. - shall use ``VisibleDeprecationWarning`` rather than ``DeprecationWarning`` for cases of relevance to end users. For cases only relevant to downstream libraries, a regular ``DeprecationWarning`` is fine. *Rationale: regular deprecation warnings are invisible by default; library authors should be aware how deprecations work and test for them, but we can't expect this from all users.* - shall be listed in the release notes of the release where the deprecation is first present. - shall set a ``stacklevel``, so the warning appears to come from the correct place. - shall be mentioned in the documentation for the functionality. A ``.. deprecated::`` directive can be used for this.

Examples of good deprecation warnings:

.. code-block:: python

warnings.warn('np.asscalar(a) is deprecated since NumPy 1.16.0, use ' 'a.item() instead', DeprecationWarning, stacklevel=3)

warnings.warn("Importing from numpy.testing.utils is deprecated " "since 1.15.0, import from numpy.testing instead.", DeprecationWarning, stacklevel=2)

# A change in NumPy 1.14.0 for Python 3 loadtxt/genfromtext, slightly # tweaked in this NEP (original didn't have version number). warnings.warn( "Reading unicode strings without specifying the encoding " "argument is deprecated since NumPy 1.14.0. Set the encoding, " "use None for the system default.", np.VisibleDeprecationWarning, stacklevel=2)

Removal of deprecated functionality:

- shall be done after at least 2 releases (assuming the current 6-monthly release cycle; if that changes, there shall be at least 1 year between deprecation and removal). - shall be listed in the release notes of the release where the removal happened. - can be done in any minor (but not bugfix) release.

For backwards incompatible changes that aren't "deprecate and remove" but for which code will start behaving differently, a ``FutureWarning`` should be used. Release notes, mentioning version number and using ``stacklevel`` should be done in the same way as for deprecation warnings. A ``.. versionchanged::`` directive can be used in the documentation to indicate when the behavior changed:

.. code-block:: python

def argsort(self, axis=np._NoValue, ...): """ Parameters ---------- axis : int, optional Axis along which to sort. If None, the default, the flattened array is used.

.. versionchanged:: 1.13.0 Previously, the default was documented to be -1, but that was in error. At some future date, the default will change to -1, as originally intended. Until then, the axis should be given explicitly when ``arr.ndim > 1``, to avoid a FutureWarning. """ ... warnings.warn( "In the future the default for argsort will be axis=-1, not the " "current None, to match its documentation and np.argsort. " "Explicitly pass -1 or None to silence this warning.", MaskedArrayFutureWarning, stacklevel=3)

Decision making ~~~~~~~~~~~~~~~

In concrete cases where this policy needs to be applied, decisions are made according to the `NumPy governance model https://docs.scipy.org/doc/numpy/dev/governance/index.html`_.

All deprecations must be proposed on the mailing list, in order to give everyone with an interest in NumPy development to be able to comment. Removal of deprecated functionality does not need discussion on the mailing list.

Functionality with more strict deprecation policies ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

- ``numpy.random`` has its own backwards compatibility policy, see `NEP 19 http://www.numpy.org/neps/nep-0019-rng-policy.html`_. - The file format for ``.npy`` and ``.npz`` files must not be changed in a backwards incompatible way.

Example cases -------------

We now discuss a few concrete examples from NumPy's history to illustrate typical issues and trade-offs.

**Changing the behavior of a function**

``np.histogram`` is probably the most infamous example. First, a new keyword ``new=False`` was introduced, this was then switched over to None one release later, and finally it was removed again. Also, it has a ``normed`` keyword that had behavior that could be considered either suboptimal or broken (depending on ones opinion on the statistics). A new keyword ``density`` was introduced to replace it; ``normed`` started giving ``DeprecationWarning`` only in v.1.15.0. Evolution of ``histogram``::

def histogram(a, bins=10, range=None, normed=False): # v1.0.0

def histogram(a, bins=10, range=None, normed=False, weights=None, new=False): #v1.1.0

def histogram(a, bins=10, range=None, normed=False, weights=None, new=None): #v1.2.0

def histogram(a, bins=10, range=None, normed=False, weights=None): #v1.5.0

def histogram(a, bins=10, range=None, normed=False, weights=None, density=None): #v1.6.0

def histogram(a, bins=10, range=None, normed=None, weights=None, density=None): #v1.15.0 # v1.15.0 was the first release where `normed` started emitting # DeprecationWarnings

The ``new`` keyword was planned from the start to be temporary. Such a plan forces users to change their code more than once, which is almost never the right thing to do. Instead, a better approach here would have been to deprecate ``histogram`` and introduce a new function ``hist`` in its place.

**Disallowing indexing with floats**

Indexing an array with floats is asking for something ambiguous, and can be a sign of a bug in user code. After some discussion, it was deemed a good idea to deprecate indexing with floats. This was first tried for the v1.8.0 release, however in pre-release testing it became clear that this would break many libraries that depend on NumPy. Therefore it was reverted before release, to give those libraries time to fix their code first. It was finally introduced for v1.11.0 and turned into a hard error for v1.12.0.

This change was disruptive, however it did catch real bugs in, e.g., SciPy and scikit-learn. Overall the change was worth the cost, and introducing it in master first to allow testing, then removing it again before a release, is a useful strategy.

Similar deprecations that also look like good examples of cleanups/improvements:

- removing deprecated boolean indexing (in 2016, see `gh-8312 < https://github.com/numpy/numpy/pull/8312>`__) - deprecating truth testing on empty arrays (in 2017, see `gh-9718 < https://github.com/numpy/numpy/pull/9718>`__)

**Removing the financial functions**

The financial functions (e.g. ``np.pmt``) had short non-descriptive names, were present in the main NumPy namespace, and didn't really fit well within NumPy's scope. They were added in 2008 after `a discussion < https://mail.python.org/pipermail/numpy-discussion/2008-April/032353.html

...

`_ on the mailing list where opinion was divided (but a majority in favor). The financial functions didn't cause a lot of overhead, however there were still multiple issues and PRs a year for them which cost maintainer time to deal with. And they cluttered up the ``numpy`` namespace. Discussion on removing them happened in 2013 (gh-2880, rejected) and then again in 2019 (:ref:`NEP32`, accepted without significant complaints).

Given that they were clearly outside of NumPy's scope, moving them to a separate ``numpy-financial`` package and removing them from NumPy after a deprecation period made sense.

Alternatives ------------

**Being more aggressive with deprecations.**

The goal of being more aggressive is to allow NumPy to move forward faster. This would avoid others inventing their own solutions (often in multiple places), as well as be a benefit to users without a legacy code base. We reject this alternative because of the place NumPy has in the scientific Python ecosystem - being fairly conservative is required in order to not increase the extra maintenance for downstream libraries and end users to an unacceptable level.

Discussion ----------

- `Mailing list discussion on the first version of this NEP in 2018 < https://mail.python.org/pipermail/numpy-discussion/2018-July/078432.html

...

`__

References and Footnotes ------------------------

- `Issue requesting semantic versioning < https://github.com/numpy/numpy/issues/10156>`__

.. [1] https://searchcode.com/

.. [2] https://github.com/Quansight-Labs/python-api-inspect

.. [3] https://github.com/data-apis/python-record-api

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

On Wed, 2020-12-30 at 11:43 -0600, Sebastian Berg wrote:

On Wed, 2020-12-30 at 16:27 +0100, Ralf Gommers wrote: <snip>

...

That's very hard to describe, since it relies so much on previous experience and qualitative judgements. That's the main reason why I had more examples before, but they just led to more discussion about those examples - so that didn't quite have the intended effect.

<snip> I only took a short course and used this very little. I am sure there are many here with industry experience where the use of Q&A is every day work.

One concept from there is to create a risk/danger and probability assessment, which can be ad-hoc for your product.  An example just to make something up:

I am not sure anyone finds this interesting or if fits to the NEP specifically [1], but I truly think it can be useful (although maybe it doesn't need to be formalized). So I fleshed it out: https://hackmd.io/WuS1rCzrTYOTgzUfRJUOnw (also pasted it below) My reasoning for suggesting it is that a process/formalism (no matter how ridiculous it may seem at first) for how to assess the impact of a backward compatible change can be helpful by: conceptualizing, clearly separating backward incompatible impact assessment from benefits assessment, making it easier to follow a decision/thought processes, and allowing some nuance [2]. I actually believe that it can help with difficult decisions, even if only applied occasionally, and that it is not a burden because it provides fairly steps. Will it be useful often? Maybe not. But every time there is a proposal and we pause and hesitate because it is unclear whether it is worth the backcompat impact, I think this can provide a way to discuss it and come to a decision as objectively as possible. (And no, I do not think that any of the categories or mitigation strategies are an exact science.) Cheers, Sebastian [1] This is additional to the proposed promises such as a two releases of deprecations and discussing most/all deprecations on the mailing list, which are unrelated. It is rather to provide a formalism where currently only the examples give points of reference. [2] There is a reason that also the Python version is short and intentionally fuzzy: https://www.python.org/dev/peps/pep-0387/ and https://discuss.python.org/t/pep-387-backwards-compatibilty-policy/4421  There are just few definite rules that can be formalized, so a framework for diligent assessment seems the best we can do (if we want to). Assessing impactHere “impact” means how unmodified code may be negatively affected by a change ignoring any deprecation period. To get an idea about how much impact a change has, try to list all potential impacts. This will often be just a single item (user of function x has to replace it with y), but it could be multiple different ones. After listing all potential impacts rank them on the following two scales (do not yet think about how to make the transition easier): 1. Severity (How bad is the impact for an affected user?)Minor: A performance regression or change in (undocumented) warning/error category will fall here. This type of change would normally not require a deprecation cycle or special consideration.Typical: Code must be updated to avoid an error, the update is simple to do in a way that works both on existing and future NumPy versions.Severe: Code will error or crash, and there is no simple work around or fix.Critical: Code returns incorrect results. A change requiring massive effort may fall here. A hard crash (e.g. segfault) in itself is typically not critical. 2. Likelihood (How many users does the change affect?)Rare: Change has very few impacted users (or even no known users after a code search). The normal assumption is that there is always someone affected, but a rarely used keyword argument of an already rarely used function will fall here.Limited: Change is in a rarely used function or function argument. Another possibility is that it affects only a small group of very advanced users.Common: Change affects a bigger audience or multiple large downstream libraries.Ubiquitous: Change affects a large fraction of NumPy users. The categories will not always be perfectly clear. That is OK. Rather than establishing precise guidelines, the purpose is a structured processes that can be reviewed. When the impact is exceptionally difficult to assess, it is often feasible to try a change on the development branch while signalling willigness to revert it. Downstream libraries test against it (and the release candidate) which gives a chance to correct an originally optimistic assessment. After assessing each impact, it will fall somewhere on the following table: Severity\LikelyhoodRareLimitedCommonUbiquitousMinorokokok?Typicalok?no?Severeno?noCriticalno?nonono Note that all changes should normally follow the two release deprecation warning policy (except “minor” ones). The “no” fields means a change is clearly unacceptable, although a NEP can always overrule it. This table only assesses the “impact”. It does not assess how the impact compares to the benefits of the proposed change. This must be favourable no matter how small the impact is. However, by assessing the impact, it will be easier to weigh it against the benefit. (Note that the table is not symmetric. An impact with “critical” severity is unlikely to be considered even when no known users are impacted.) Mitigation and arguing of benefitsAny change falling outside the “ok” fields requires careful consideration. When an impact is larger, you can try to mitigate it and “move” on the table. Some possible reasons for this are: * A avoidable warning for at least two releases (the policy for any change that modifies behaviour) reduces a change one category (usually from “typical” to “minor” severity). * The severity category may be reduced by creating an easy work around (i.e. to move it from “sever” to “typical”). * Sometimes a change may break working code, but also fix existing bugs, this can offset the severity. In extreme cases, this may warrant classifying a change as a bug-fix. * For particularly noisy changes (i.e. ubiquitous category) considering fixing downstream packages, delay the warning (or use a PendingDeprecationWarning). Simply prolonging the the deprecation period is also an option. This reduces how many users struggle with the change and smoothens the transition. * Exceptionally clear documentation and communication could be used to ensure that the impact is more acceptable. This may not be enough to move a “category” by itself, but also helps. After mitigation, the benefits can be assessed: * Any benefit of the change can be argued to “offset” the impact. If this is necessary, a broad community discussion on the mailing list is required. It should be clear that this does not actually “mitigate” the impact but rather argues that the benefit outweighs it. These are not a fixed set of rules, but provide a framework to assess and then try to mitigate the impact of a proposed change to an acceptable level. Arguing that a benefit can overcome multiple “impact” categories will require exceptionally large benefits, and most likely a NEP. For example a change with an initial impact classification of “severe” and “ubiquitous” is unlikely to even be considered unless the severity can be reduced. Many deprecations will fall somewhere below or equal to a “typical and limited” impact (i.e. removal of an uncommon function argument). They recieve a deprecation warning to make the impact acceptable with a brief discussiong that the change itself is worthwhile (i.e. the API is much cleaner afterwards). Any more disruptive change requires broad community discussion. This needs at least a discussion on the NumPy mailing list and it is likely that the person proposing it will be asked to write a NEP. Summary and reasoning for this processessThe aim of this process and table is to provide a loose formalism with the goal of: * Diligence: Following this process ensures detailed assessment of its impact without being distracted by the benefits. This is achieved by following well defined steps:Listing each potential impact (usually one).Assessing the severity.Assessing the likelihood.Discussing what steps are/can be taken to lower the impact ignoring any benefits.If the impact is not low at this point, this should prompt considering and listing of alternatives.Argue that the benefits outweigh the remaining impact. (This is a distinct step: the original impact assessment stands as it was.) * Transparency: Using this process for difficult decisions makes it easier for the reviewer and community to follow how a decision was made and criticize it. * Nuance: When the it is clear that an impact is larger than typical with will prompt more care and thought. In some cases it may also clarify that a change is lower impact than expected on first sight. * Experience: Using a similar formalism for many changes makes it easier to learn from past decisions by providing an approach to compare and conceptualize them. We aim to follow these steps in the future for difficult decisions. In general, any reviewer and community member may ask for this process to be followed for a proposed change, if the change is difficult, it will be worth the effort. If it is very low impact it will be quick to clarify why. NOTE: At this time the process is new and is expected to require clarification. ExamplesIt should be stressed again, that the categories will rarely be clear and intentially are categorized with some uncertainty below. Even unclear categories can help in forming a more clear idea of a change. HistogramThe “histogram” example doesn’t really add much with respect to this process. But noting the duplicate effort/impact would move probably move it into a more severe category than most deprecations. That makes it a more difficult decision and indicates that careful thought should be spend on alternatives. Integer indexing requirement * Severity: Typical–Severe (although fairly easy, users often had to do many changes) * Likelihood: Ubiquitous How ubiquitous it really was became probably only clear after the (rc?) release. The change would now probably go through a NEP as it initially falls into the lower right part of the table. To get into the “acceptable” part of the table we note that: 1. Real bugs were caught in the processes (argued to reduce severity) 2. The deprecation was delayed and longer than normally (argued to mitigate the number of affected users by giving much more time) Even with these considerations, it still has a larger impact and clearly requires careful thought and community discussion about the benefits. Removing financial functions * Severity: Severe (on the high end) * Likelihood: Limited (maybe common) While not used by a large user base (limited), the removal is disurptive (severe). The change ultimately required a NEP, since it is not easy to weigh the maintainence advantage of removing the functions against the impact to their users. The NEP included the reduction of the severity by providing a work- around: A pip installable package as a drop-in replacement (reducing the severity). For heavy users of these functions this will still be more severe than most deprecations, but it lowered the impact assessment enough to consider the benefit of removal to outweigh the impact.

On Sat, 2021-01-02 at 18:06 +0100, Ralf Gommers wrote:

On Sat, Jan 2, 2021 at 3:55 AM Sebastian Berg < sebastian@sipsolutions.net> wrote:

...

On Wed, 2020-12-30 at 11:43 -0600, Sebastian Berg wrote:

On Wed, 2020-12-30 at 16:27 +0100, Ralf Gommers wrote:

<snip>

That's very hard to describe, since it relies so much on previous experience and qualitative judgements. That's the main reason why I had more examples before, but they just led to more discussion about those examples - so that didn't quite have the intended effect.

<snip>

I only took a short course and used this very little. I am sure there are many here with industry experience where the use of Q&A is every day work.

Thanks for thinking about this Sebastian.

I used to use such a risk management approach fairly regularly, and it can be useful. In general it's something you do for a larger design change or new product, rather than for an individual change. It helps get an overview of the main risks, and prompts thinking about risks you may have missed.

Yeah, I guess its for new products mostly, to compile many risks and make it easier to compare them. And yes, we do not have "many" risks, unless you would compile this for the complete changelog between one or more versions.

...

One concept from there is to create a risk/danger and probability assessment, which can be ad-hoc for your product.  An example just to make something up:

I am not sure anyone finds this interesting or if fits to the NEP specifically [1], but I truly think it can be useful (although maybe it doesn't need to be formalized). So I fleshed it out: https://hackmd.io/WuS1rCzrTYOTgzUfRJUOnw (also pasted it below)

I'd be happy to try it. It does feel a bit too much to put all that content into the NEP though. Maybe we can just add a more brief "assess severity and likelihood and severity of your proposed change, and include that assessment when proposing a deprecation. See <here> for more details". And then we can link to a wiki page or separate doc page, that we can then easily update without it being a NEP revision.

Yes, it adds a lot of content and I don't want to force it on anyone or into the NEP, in that sense it is more brainstorming than a very concrete proposal. And I am also fine with just dropping it, whatever others think is useful. I fleshed it out a bit, because I actually think it ends up representing fairly well how I currently try to approach this and I think it may be useful when a proposal gets stuck because it is unclear whether it is worth the pain/risk. Cheers, Sebastian

Cheers, Ralf

...

My reasoning for suggesting it is that a process/formalism (no matter how ridiculous it may seem at first) for how to assess the impact of a backward compatible change can be helpful by: conceptualizing, clearly separating backward incompatible impact assessment from benefits assessment, making it easier to follow a decision/thought processes, and allowing some nuance [2].

I actually believe that it can help with difficult decisions, even if only applied occasionally, and that it is not a burden because it provides fairly steps. Will it be useful often? Maybe not. But every time there is a proposal and we pause and hesitate because it is unclear whether it is worth the backcompat impact, I think this can provide a way to discuss it and come to a decision as objectively as possible. (And no, I do not think that any of the categories or mitigation strategies are an exact science.)

Cheers,

Sebastian

[1] This is additional to the proposed promises such as a two releases of deprecations and discussing most/all deprecations on the mailing list, which are unrelated. It is rather to provide a formalism where currently only the examples give points of reference. [2] There is a reason that also the Python version is short and intentionally fuzzy: https://www.python.org/dev/peps/pep-0387/ and

https://discuss.python.org/t/pep-387-backwards-compatibilty-policy/4421  There are just few definite rules that can be formalized, so a framework for diligent assessment seems the best we can do (if we want to).

Assessing impact Here “impact” means how unmodified code may be negatively affected by a change ignoring any deprecation period.

To get an idea about how much impact a change has, try to list all potential impacts. This will often be just a single item (user of function  x has to replace it with y), but it could be multiple different ones. *After* listing all potential impacts rank them on the following two scales (do not yet think about how to make the transition easier):

   1. *Severity* (How bad is the impact for an affected user?)    - Minor: A performance regression or change in (undocumented)       warning/error category will fall here. This type of change would normally       not require a deprecation cycle or special consideration.       - Typical: Code must be updated to avoid an error, the update is       simple to do in a way that works both on existing and future NumPy versions.       - Severe: Code will error or crash, and there is no simple work       around or fix.       - Critical: Code returns incorrect results. A change requiring       massive effort may fall here. A hard crash (e.g. segfault) in itself is       typically *not* critical.    2. *Likelihood* (How many users does the change affect?)    - Rare: Change has very few impacted users (or even no known users       after a code search). The normal assumption is that there is always someone       affected, but a rarely used keyword argument of an already rarely used       function will fall here.       - Limited: Change is in a rarely used function or function       argument. Another possibility is that it affects only a small group of very       advanced users.       - Common: Change affects a bigger audience or multiple large       downstream libraries.       - Ubiquitous: Change affects a large fraction of NumPy users.

The categories will not always be perfectly clear. That is OK. Rather than establishing precise guidelines, the purpose is a structured *processes* that can be reviewed. When the impact is exceptionally difficult to assess, it is often feasible to try a change on the development branch while signalling willigness to revert it. Downstream libraries test against it (and the release candidate) which gives a chance to correct an originally optimistic assessment.

After assessing each impact, it will fall somewhere on the following table: Severity\LikelyhoodRareLimitedCommonUbiquitous *Minor* ok ok ok? *Typical* ok? no? *Severe* no? no *Critical* no? no no no Note that all changes should normally follow the two release deprecation warning policy (except “minor” ones). The “no” fields means a change is clearly unacceptable, although a NEP can always overrule it. This table only assesses the “impact”. It does not assess how the impact compares to the benefits of the proposed change. This must be favourable no matter how small the impact is. However, by assessing the impact, it will be easier to weigh it against the benefit. (Note that the table is not symmetric. An impact with “critical” severity is unlikely to be considered even when no known users are impacted.)

...

Mitigation and arguing of benefits Any change falling outside the “ok” fields requires careful consideration. When an impact is larger, you can try to mitigate it and “move” on

< https://hackmd.io/WuS1rCzrTYOTgzUfRJUOnw#Mitigation-and-arguing-of-benefits the table. Some possible reasons for this are:

   - A avoidable warning for at least two releases (the policy for any    change that modifies behaviour) reduces a change one category (usually from    “typical” to “minor” severity).    - The severity category may be reduced by creating an easy work around    (i.e. to move it from “sever” to “typical”).    - Sometimes a change may break working code, but also fix *existing* bugs,    this can offset the severity. In extreme cases, this may warrant    classifying a change as a bug-fix.    - For particularly noisy changes (i.e. ubiquitous category)    considering fixing downstream packages, delay the warning (or use a    PendingDeprecationWarning). Simply prolonging the the deprecation    period is also an option. This reduces how many users struggle with the    change and smoothens the transition.    - Exceptionally clear documentation and communication could be used to    ensure that the impact is more acceptable. This may not be enough to move a    “category” by itself, but also helps.

After mitigation, the benefits can be assessed:

   - Any benefit of the change can be argued to “offset” the impact. If    this is necessary, a broad community discussion on the mailing list is    required. It should be clear that this does not actually “mitigate” the    impact but rather argues that the benefit outweighs it.

These are not a fixed set of rules, but provide a framework to assess and then try to mitigate the impact of a proposed change to an acceptable level. Arguing that a benefit can overcome multiple “impact” categories will require exceptionally large benefits, and most likely a NEP. For example a change with an initial impact classification of “severe” and “ubiquitous” is unlikely to even be considered unless the severity can be reduced. Many deprecations will fall somewhere below or equal to a “typical and limited” impact (i.e. removal of an uncommon function argument). They recieve a deprecation warning to make the impact acceptable with a brief discussiong that the change itself is worthwhile (i.e. the API is much cleaner afterwards). Any more disruptive change requires broad community discussion. This needs at least a discussion on the NumPy mailing list and it is likely that the person proposing it will be asked to write a NEP.

< https://hackmd.io/WuS1rCzrTYOTgzUfRJUOnw#Summary-and-reasoning-for-this-proc...

...

Summary and reasoning for this processess The aim of this process and table is to provide a loose formalism with the goal of:

   - *Diligence:* Following this process ensures detailed assessment of    its impact without being distracted by the benefits. This is achieved by    following well defined steps:       1. Listing each potential impact (usually one).       2. Assessing the severity.       3. Assessing the likelihood.       4. Discussing what steps are/can be taken to lower the impact *ignoring       any benefits*.       5. If the impact is not low at this point, this should prompt       considering and listing of alternatives.       6. Argue that the benefits outweigh the remaining impact. (This is       a distinct step: the original impact assessment stands as it was.)    - *Transparency:* Using this process for difficult decisions makes it    easier for the reviewer and community to follow how a decision was made and    criticize it.    - *Nuance:* When the it is clear that an impact is larger than typical    with will prompt more care and thought. In some cases it may also clarify    that a change is lower impact than expected on first sight.    - *Experience:* Using a similar formalism for many changes makes it    easier to learn from past decisions by providing an approach to compare and    conceptualize them.

We aim to follow these steps in the future for difficult decisions. In general, any reviewer and community member may ask for this process to be followed for a proposed change, if the change is difficult, it will be worth the effort. If it is very low impact it will be quick to clarify why. NOTE: At this time the process is new and is expected to require clarification. https://hackmd.io/WuS1rCzrTYOTgzUfRJUOnw#ExamplesExamples It should be stressed again, that the categories will rarely be clear and intentially are categorized with some uncertainty below. Even unclear categories can help in forming a more clear idea of a change. https://hackmd.io/WuS1rCzrTYOTgzUfRJUOnw#HistogramHistogram The “histogram” example doesn’t really add much with respect to this process. But noting the duplicate effort/impact would move probably move it into a more severe category than most deprecations. That makes it a more difficult decision and indicates that careful thought should be spend on alternatives. < https://hackmd.io/WuS1rCzrTYOTgzUfRJUOnw#Integer-indexing-requirement

...

Integer indexing requirement

   - Severity: Typical–Severe (although fairly easy, users often had to    do many changes)    - Likelihood: Ubiquitous

How ubiquitous it really was became probably only clear after the (rc?) release. The change would now probably go through a NEP as it initially falls into the lower right part of the table. To get into the “acceptable” part of the table we note that:

   1. Real bugs were caught in the processes (argued to reduce severity)    2. The deprecation was delayed and longer than normally (argued to    mitigate the number of affected users by giving much more time)

Even with these considerations, it still has a larger impact and clearly requires careful thought and community discussion about the benefits. < https://hackmd.io/WuS1rCzrTYOTgzUfRJUOnw#Removing-financial-functions

...

Removing financial functions

   - Severity: Severe (on the high end)    - Likelihood: Limited (maybe common)

While not used by a large user base (limited), the removal is disurptive (severe). The change ultimately required a NEP, since it is not easy to weigh the maintainence advantage of removing the functions against the impact to their users. The NEP included the reduction of the severity by providing a work- around: A pip installable package as a drop-in replacement (reducing the severity). For heavy users of these functions this will still be more severe than most deprecations, but it lowered the impact assessment enough to consider the benefit of removal to outweigh the impact. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

On Tue, Jan 26, 2021, at 00:25, Ralf Gommers wrote:

The update PR was merged after a lot more review on GitHub. I propose we change the status of this NEP to Accepted. We'll merge a PR to do so unless there are objections within the next five days.

Thanks for the heads-up, Ralf. I am happy to have the NEP accepted. Stéfan