Policy for minimum numpy version
Hi All, In gh-6058 <https://github.com/scipy/scipy/pull/gh-6058>, I’ve run into a test failure because a numpy function was not available in version 1.7 (the function being np.fft.rfftfreq). This is easy enough to work around, but I wonder when we should decide to move up the minimum required numpy version. Debian stable currently offers numpy 1.8.2. Ubuntu 12.04 LTS (which is EOL next April) is stuck on 1.6.1, so they’re already left behind. Ubuntu 14.04 LTS has 1.8.2. Has anyone ran into compatibility issues with 1.7 before? In _lib._numpy_compat, there is another 1.7 workaround in place for np.testing.assert_warns. -Eric Q.
Most recently I've wished for np.full and np.full_like, though I admit that they're very easy to work around. More difficult to replace is np.partition and np.argpartition, which are extremely useful in the few cases that they're needed but also non-trivial to implement efficiently. Typically the work-arounds look like this (not my code): https://github.com/lemaoliu/Agtarbidir/blob/8762efcb29b37724c2af13f9393d1eaa... On Wed, Jul 27, 2016 at 12:52 PM, Eric Quintero <ericq@caltech.edu> wrote:
Hi All,
In gh-6058 <https://github.com/scipy/scipy/pull/gh-6058>, I’ve run into a test failure because a numpy function was not available in version 1.7 (the function being np.fft.rfftfreq). This is easy enough to work around, but I wonder when we should decide to move up the minimum required numpy version.
Debian stable currently offers numpy 1.8.2. Ubuntu 12.04 LTS (which is EOL next April) is stuck on 1.6.1, so they’re already left behind. Ubuntu 14.04 LTS has 1.8.2.
Has anyone ran into compatibility issues with 1.7 before? In _lib._numpy_compat, there is another 1.7 workaround in place for np.testing.assert_warns.
-Eric Q.
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org https://mail.scipy.org/mailman/listinfo/scipy-dev
Here is an example of `np.percentile` workarounds for `scipy.stats.iqr`: https://github.com/scipy/scipy/pull/5808 -Joe On Wed, Jul 27, 2016 at 2:18 PM, CJ Carey <perimosocordiae@gmail.com> wrote:
Most recently I've wished for np.full and np.full_like, though I admit that they're very easy to work around.
More difficult to replace is np.partition and np.argpartition, which are extremely useful in the few cases that they're needed but also non-trivial to implement efficiently. Typically the work-arounds look like this (not my code): https://github.com/lemaoliu/Agtarbidir/blob/8762efcb29b37724c2af13f9393d1eaa...
On Wed, Jul 27, 2016 at 12:52 PM, Eric Quintero <ericq@caltech.edu> wrote:
Hi All,
In gh-6058, I’ve run into a test failure because a numpy function was not available in version 1.7 (the function being np.fft.rfftfreq). This is easy enough to work around, but I wonder when we should decide to move up the minimum required numpy version.
Debian stable currently offers numpy 1.8.2. Ubuntu 12.04 LTS (which is EOL next April) is stuck on 1.6.1, so they’re already left behind. Ubuntu 14.04 LTS has 1.8.2.
Has anyone ran into compatibility issues with 1.7 before? In _lib._numpy_compat, there is another 1.7 workaround in place for np.testing.assert_warns.
-Eric Q.
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org https://mail.scipy.org/mailman/listinfo/scipy-dev
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org https://mail.scipy.org/mailman/listinfo/scipy-dev
Note also that numpy now has wheels available for all the common platforms, so the cost of installing a new numpy is much, much lower than it used to be. (And installing a new numpy is easier than installing a new scipy anyway :-).) On Jul 27, 2016 10:52 AM, "Eric Quintero" <ericq@caltech.edu> wrote:
Hi All,
In gh-6058 <https://github.com/scipy/scipy/pull/gh-6058>, I’ve run into a test failure because a numpy function was not available in version 1.7 (the function being np.fft.rfftfreq). This is easy enough to work around, but I wonder when we should decide to move up the minimum required numpy version.
Debian stable currently offers numpy 1.8.2. Ubuntu 12.04 LTS (which is EOL next April) is stuck on 1.6.1, so they’re already left behind. Ubuntu 14.04 LTS has 1.8.2.
Has anyone ran into compatibility issues with 1.7 before? In _lib._numpy_compat, there is another 1.7 workaround in place for np.testing.assert_warns.
-Eric Q.
_______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org https://mail.scipy.org/mailman/listinfo/scipy-dev
On Wed, Jul 27, 2016 at 7:52 PM, Eric Quintero <ericq@caltech.edu> wrote:
Hi All,
In gh-6058 <https://github.com/scipy/scipy/pull/gh-6058>, I’ve run into a test failure because a numpy function was not available in version 1.7 (the function being np.fft.rfftfreq). This is easy enough to work around, but I wonder when we should decide to move up the minimum required numpy version.
There's no conclusion here yet, but the other responses to this email were examples of pain points with 1.8 and that the cost of upgrading numpy has gone down. In general I think we bump the minimum version if the costs are starting to outweigh the benefits. That's probably the case for 1.7 now. Looking back, we typically have supported 4 numpy versions with a scipy release. Right now we're at 5 (1.7 - 1.11), and numpy 1.12 will likely be out before the next scipy release. So my conclusion: +1 for bumping the minimum numpy version to 1.8.2. Ralf
Debian stable currently offers numpy 1.8.2. Ubuntu 12.04 LTS (which is EOL next April) is stuck on 1.6.1, so they’re already left behind. Ubuntu 14.04 LTS has 1.8.2.
Has anyone ran into compatibility issues with 1.7 before? In _lib._numpy_compat, there is another 1.7 workaround in place for np.testing.assert_warns.
On 3 August 2016 at 09:48, Ralf Gommers <ralf.gommers@gmail.com> wrote:
There's no conclusion here yet, but the other responses to this email were examples of pain points with 1.8 and that the cost of upgrading numpy has gone down.
In general I think we bump the minimum version if the costs are starting to outweigh the benefits. That's probably the case for 1.7 now. Looking back, we typically have supported 4 numpy versions with a scipy release. Right now we're at 5 (1.7 - 1.11), and numpy 1.12 will likely be out before the next scipy release.
What is the use case that requires an 2 years and 9 months old numpy and the latest, bleeding edge, scipy? The deprecation cycle is of two versions, so I think it should be enough to support the last two numpy versions at the time of the previous scipy release. So, scipy 0.19 would support from numpy 1.10, that was released in October last year, and people would have had a year to adapt. Note also that pip's default behaviour is to upgrade every dependency. Side question: do we check the other way around? Will a new numpy work with an old scipy? Say, numpy 1.11 with scipy 0.13 (as old as 1.8)? /David.
On Wed, Aug 3, 2016 at 10:32 AM, Daπid <davidmenhur@gmail.com> wrote:
On 3 August 2016 at 09:48, Ralf Gommers <ralf.gommers@gmail.com> wrote:
There's no conclusion here yet, but the other responses to this email
were
examples of pain points with 1.8 and that the cost of upgrading numpy has gone down.
In general I think we bump the minimum version if the costs are starting to outweigh the benefits. That's probably the case for 1.7 now. Looking back, we typically have supported 4 numpy versions with a scipy release. Right now we're at 5 (1.7 - 1.11), and numpy 1.12 will likely be out before the next scipy release.
What is the use case that requires an 2 years and 9 months old numpy and the latest, bleeding edge, scipy?
See the responses to the last time you asked this, those are still valid: https://mail.scipy.org/pipermail/scipy-dev/2014-December/020266.html :)
The deprecation cycle is of two versions, so I think it should be enough to support the last two numpy versions at the time of the previous scipy release. So, scipy 0.19 would support from numpy 1.10, that was released in October last year, and people would have had a year to adapt.
Note also that pip's default behaviour is to upgrade every dependency.
That pip behavior is completely braindead, and there is agreement by the pip/packaging crowd that it needs to be changed to a more minimalist upgrade strategy. The debate is just about how to introduce the change.
Side question: do we check the other way around? Will a new numpy work with an old scipy? Say, numpy 1.11 with scipy 0.13 (as old as 1.8)?
Yes it should, except for the few things that needed to be adapted in newer scipy versions for removals of deprecated features and unintentional backwards compat breaks. Older scipy binaries will also work with a newer numpy. Ralf
On 3 August 2016 at 12:48, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Wed, Aug 3, 2016 at 10:32 AM, Daπid <davidmenhur@gmail.com> wrote:
On 3 August 2016 at 09:48, Ralf Gommers <ralf.gommers@gmail.com> wrote:
There's no conclusion here yet, but the other responses to this email were examples of pain points with 1.8 and that the cost of upgrading numpy has gone down.
In general I think we bump the minimum version if the costs are starting to outweigh the benefits. That's probably the case for 1.7 now. Looking back, we typically have supported 4 numpy versions with a scipy release. Right now we're at 5 (1.7 - 1.11), and numpy 1.12 will likely be out before the next scipy release.
What is the use case that requires an 2 years and 9 months old numpy and the latest, bleeding edge, scipy?
See the responses to the last time you asked this, those are still valid: https://mail.scipy.org/pipermail/scipy-dev/2014-December/020266.html :)
I totally forgot I ever said that, sorry. Thank you for bearing with my fish memory.
A lot more packages depend on numpy than on scipy, so upgrading numpy can have way more impact.
I see. I guess this is mostly relevant as a part of a large package repository, where there is a higher chance of finding outdated packages. On the other hand, numpy is pretty backwards compatible, except for long deprecation cycles, so there is usually plenty of time to upgrade (one year of deprecation, plus one year of scipy catching up in my scheme). All the backwards incompatible changes I can think of so far required fairly minor adjustments (for example, random_integers -> randint), unless I have forgotten something (fish memory). Or, of course, the codebase is huge, but I have no experience there.
And there may be institutes/companies that ship a fixed version of numpy that needs to be supported for a long time.
My question in this case remains, if they ship an ancient numpy, why would they need the latest scipy? If the component is mission critical and upgrading is a risk, I'd expect the same would be true for scipy. Most other cases would be covered by using virtualenvs or anaconda. And I would add one reason to encourage people to keep up to date: deprecation cycles work under the assumption that developers try every version, and use the warnings to adapt their code. If someone were to upgrade directly from numpy 1.8 to 1.12, the safety net is circumvented, and they may find unexpected changes in behaviour. /David.
So far, it seems no-one has argued against bumping the minimum version to 1.8. If this is the necessary level of consensus, what steps need to happen to make this transition? -Eric Q.
On Aug 5, 2016, at 7:06 AM, Daπid <davidmenhur@gmail.com> wrote:
On 3 August 2016 at 12:48, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Wed, Aug 3, 2016 at 10:32 AM, Daπid <davidmenhur@gmail.com> wrote:
On 3 August 2016 at 09:48, Ralf Gommers <ralf.gommers@gmail.com> wrote:
There's no conclusion here yet, but the other responses to this email were examples of pain points with 1.8 and that the cost of upgrading numpy has gone down.
In general I think we bump the minimum version if the costs are starting to outweigh the benefits. That's probably the case for 1.7 now. Looking back, we typically have supported 4 numpy versions with a scipy release. Right now we're at 5 (1.7 - 1.11), and numpy 1.12 will likely be out before the next scipy release.
What is the use case that requires an 2 years and 9 months old numpy and the latest, bleeding edge, scipy?
See the responses to the last time you asked this, those are still valid: https://mail.scipy.org/pipermail/scipy-dev/2014-December/020266.html :)
I totally forgot I ever said that, sorry. Thank you for bearing with my fish memory.
A lot more packages depend on numpy than on scipy, so upgrading numpy can have way more impact.
I see. I guess this is mostly relevant as a part of a large package repository, where there is a higher chance of finding outdated packages. On the other hand, numpy is pretty backwards compatible, except for long deprecation cycles, so there is usually plenty of time to upgrade (one year of deprecation, plus one year of scipy catching up in my scheme).
All the backwards incompatible changes I can think of so far required fairly minor adjustments (for example, random_integers -> randint), unless I have forgotten something (fish memory). Or, of course, the codebase is huge, but I have no experience there.
And there may be institutes/companies that ship a fixed version of numpy that needs to be supported for a long time.
My question in this case remains, if they ship an ancient numpy, why would they need the latest scipy? If the component is mission critical and upgrading is a risk, I'd expect the same would be true for scipy. Most other cases would be covered by using virtualenvs or anaconda.
And I would add one reason to encourage people to keep up to date: deprecation cycles work under the assumption that developers try every version, and use the warnings to adapt their code. If someone were to upgrade directly from numpy 1.8 to 1.12, the safety net is circumvented, and they may find unexpected changes in behaviour.
/David. _______________________________________________ SciPy-Dev mailing list SciPy-Dev@scipy.org https://mail.scipy.org/mailman/listinfo/scipy-dev
On Thu, Aug 11, 2016 at 5:05 AM, Eric Quintero <ericq@caltech.edu> wrote:
So far, it seems no-one has argued against bumping the minimum version to 1.8. If this is the necessary level of consensus,
I think it is.
what steps need to happen to make this transition?
A PR with changes to setup.py, scipy/__init__.py, pavement.py and doc/release/0.19.0-notes.rst for changing 1.7.1/1.7.2 to 1.8.2 at least. And then in the same or another PR all the special-casing and tests skipped for 1.7.x can be removed.
-Eric Q.
On Aug 5, 2016, at 7:06 AM, Daπid <davidmenhur@gmail.com> wrote:
On 3 August 2016 at 12:48, Ralf Gommers <ralf.gommers@gmail.com> wrote:
On Wed, Aug 3, 2016 at 10:32 AM, Daπid <davidmenhur@gmail.com> wrote:
On 3 August 2016 at 09:48, Ralf Gommers <ralf.gommers@gmail.com>
wrote:
There's no conclusion here yet, but the other responses to this email were examples of pain points with 1.8 and that the cost of upgrading numpy has gone down.
In general I think we bump the minimum version if the costs are
starting
to outweigh the benefits. That's probably the case for 1.7 now. Looking back, we typically have supported 4 numpy versions with a scipy release. Right now we're at 5 (1.7 - 1.11), and numpy 1.12 will likely be out before the next scipy release.
What is the use case that requires an 2 years and 9 months old numpy and the latest, bleeding edge, scipy?
See the responses to the last time you asked this, those are still valid: https://mail.scipy.org/pipermail/scipy-dev/2014-December/020266.html :)
I totally forgot I ever said that, sorry. Thank you for bearing with my fish memory.
No worries.
A lot more packages depend on numpy than on scipy, so upgrading numpy
can have way more impact.
I see. I guess this is mostly relevant as a part of a large package repository, where there is a higher chance of finding outdated packages. On the other hand, numpy is pretty backwards compatible, except for long deprecation cycles, so there is usually plenty of time to upgrade (one year of deprecation, plus one year of scipy catching up in my scheme).
One year is not a lot at all for deployed server installs, companies shipping qualified sets of packages, etc. And even occasional users won't be updating actively. For people reading this mailing list a year is a lot, but those are not average users.
All the backwards incompatible changes I can think of so far required fairly minor adjustments (for example, random_integers -> randint), unless I have forgotten something (fish memory). Or, of course, the codebase is huge, but I have no experience there.
And there may be institutes/companies that ship a fixed version of
numpy that needs to be supported for a long time.
My question in this case remains, if they ship an ancient numpy, why would they need the latest scipy?
They may need a specific feature or bugfix.
If the component is mission critical
and upgrading is a risk, I'd expect the same would be true for scipy.
Sure, but any extra package to upgrade is extra work. And work is time/money. This really comes down to putting the pain somewhere, either with some devs that run into an issue with older numpy when making a PR, or with a very hard to estimate (but certainly much larger) group of users.
Most other cases would be covered by using virtualenvs or anaconda.
And I would add one reason to encourage people to keep up to date: deprecation cycles work under the assumption that developers try every version, and use the warnings to adapt their code. If someone were to upgrade directly from numpy 1.8 to 1.12, the safety net is circumvented, and they may find unexpected changes in behaviour.
Most (ideally all) of those should be exceptions or things like shape changes, not silent changes in numerical values. Anyway, again this a cost/benefit trade-off. Ralf
participants (6)
-
CJ Carey -
Daπid -
Eric Quintero -
Joseph Fox-Rabinovitz -
Nathaniel Smith -
Ralf Gommers