Re: [Numpy-discussion] Proposal of timeline for dropping Python 2.7 support
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
See Thomas's reply quoted below (it was rejected by the mailing list since he's not subscribed): On Nov 9, 2017 01:24, "Thomas Kluyver" <thomas@kluyver.me.uk> wrote: On Thu, Nov 9, 2017, at 08:52 AM, Nathaniel Smith wrote: On Nov 8, 2017 23:59, "Ralf Gommers" <ralf.gommers@gmail.com> wrote: Regarding http://www.python3statement.org/: I'd say that as long as there are people who want to spend their energy on the LTS release (contributors *and* enough maintainer power to review/merge/release), we should not actively prevent them from doing that. Yeah, agreed. I don't feel like this is incompatible with the spirit of python3statement.org, though looking at the text I can see how it's not clear. My guess is they'd be happy to adjust the text, especially if it lets them add numpy :-). CC'ing Thomas and Matthias. Thanks Nathaniel. We have (IMO) left a degree of deliberate ambiguity around precisely what 'drop support' means, because it's not going to be the same for all projects. The nature of open source also means that there can be ambiguity over what 'support' entails and who is considered part of the project. I would say that the idea of the statement is compatible with an LTS release series receiving critical bugfixes beyond 2020, while the main energy of the project is focused on Py3-only feature releases. [If numpy-discussion doesn't allow non-member posts, feel free to pass this on or quote it in on-list messages] Thomas
![](https://secure.gravatar.com/avatar/4d4ea6148fef59dff9fa0fc8c309496a.jpg?s=120&d=mm&r=g)
Hi all, Apologies if this mail appear out of thread I just subscribed to respond.
Happy to see NumPy at least having this conversation ! I agree with Thomas, we're pretty loose on what dropping support means; one of the main reason for the Python-3-Statement is communication to users and other project; and covey that there is a strong intent that you have until 2020 to get ready (if not before). The voice of NumPy have a huge weight in the balance. I quickly went through the thread and have a few responses:
And that is why you should decide of doing it at some point, and telling it to the world and the sooner you decide and advertise it (regardless of effective "deadline" the better. The Scientific Python is in a catch 22 position; Most of the ecosystem will not drop 2.7 because "numpy is still compatible python 2.7", and numpy does not drop it because "many packages rely on numpy support for 2.7".
We'd have to make sure we could persuade pypi to give the older version for Windows, by default - I don't know if that is possible.
I don't think there is, though if you tag a release with `requires_python>3.3`, then pip 9+ users on python 2.7 will not even realise there are new release compatible only with 3.3+. Technically you can make numpy a meta-package that requires numpy-27 on windows only... but it has its own drawbacks.
These two are _in practice_ against each other; if you do major cleaning then most backports will have a hard time being auto applied (just a warning). If you have a team that want to do a LTS I would suggest "cleaning" only when you are actually touching some code and the python-2 support code is in the way. not cleaning "for the sake of cleaning" at least until the 2 code base are far enough. We have a bot on Jupyter/Matplotlib that help to backport PRs to older branches. I'm happy to open it to the numpy org if it helps.
The hardest part is communication. And not just "We're dropping in 2020" but also "We still care about you, 2.7 users", ans especially tell 2.7 users and old pip users how to correctly pip their dependency on numpy (to still get LTS if LTS there is) One more thing, there is a lot of of discussion about a "Volonteer LTS", you may also want to consider a partnership with a company for an officially recommended commercial offer. Thanks, -- Matthias On Thu, Nov 9, 2017 at 10:21 AM, Nathaniel Smith <njs@pobox.com> wrote:
![](https://secure.gravatar.com/avatar/96dd777e397ab128fedab46af97a3a4a.jpg?s=120&d=mm&r=g)
On Thu, Nov 9, 2017 at 11:24 AM, Matthias Bussonnier < bussonniermatthias@gmail.com> wrote:
One thing worth considering might be making the release that drops 2.7 NumPy 2.0 just so that there is a clear break point. Then if someone wants to continue the 1.x line of releases supporting 2.7 they can do so. ISTR that git now has some features that might aid in that. If the `requires` bit works well with pip then we can also share the same pip page (maybe). <snip> Chuck
![](https://secure.gravatar.com/avatar/4aa06fcfdb62e99be5fce44abd631406.jpg?s=120&d=mm&r=g)
On Nov 9, 2017, at 13:35, Charles R Harris <charlesr.harris@gmail.com> wrote:
One thing worth considering might be making the release that drops 2.7 NumPy 2.0 just so that there is a clear break point. Then if someone wants to continue the 1.x line of releases supporting 2.7 they can do so. ISTR that git now has some features that might aid in that. If the `requires` bit works well with pip then we can also share the same pip page (maybe).
I personally think this is definitely advisable. FWIW Bokeh will definitely be bumping major numbers when dropping Python 2 support, or classic notebook support, etc. Bryan
![](https://secure.gravatar.com/avatar/851ff10fbb1363b7d6111ac60194cc1c.jpg?s=120&d=mm&r=g)
In astropy we had a similar discussion about version numbers, and decided to make 2.0 the LTS that still supports python 2.7 and 3.0 the first that does not. If we're discussing jumping a major number, we could do the same for numpy. (Admittedly, it made a bit more sense with the numbering scheme astropy had adopted anyway.) -- Marten
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
Fortunately we can wait until we're a bit closer before we have to make any final decision on the version numbering :-) Right now though it would be good to start communicating to users/downstreams about whatever our plans our though, so they can make plans. Here's a first attempt at some text we can put in the documentation and point people to -- any thoughts, on either the plan or the wording? ---- DRAFT TEXT - NOT FINAL - DO NOT POST THIS TO HACKERNEWS OK? OK ---- The Python core team plans to stop supporting Python 2 in 2020. The NumPy project has supported both Python 2 and Python 3 in parallel since 2010, and has found that supporting Python 2 is an increasing burden on our limited resources; thus, we plan to eventually drop Python 2 support as well. Now that we're entering the final years of community-supported Python 2, the NumPy project wants to clarify our plans, with the goal of to helping our downstream ecosystem make plans and accomplish the transition with as little disruption as possible. Our current plan is as follows: Until **December 31, 2018**, all NumPy releases will fully support both Python 2 and Python 3. Starting on **January 1, 2019**, any new feature releases will support only Python 3. The last Python-2-supporting release will be designated as a long-term support (LTS) release, meaning that we will continue to merge bug-fixes and make bug-fix releases for a longer period than usual. Specifically, it will be supported by the community until **December 31, 2019**. On **January 1, 2020** we will raise a toast to Python 2, and community support for the last Python-2-supporting release will come to an end. However, it will continue to be available on PyPI indefinitely, and if any commercial vendors wish to extend the LTS support past this point then we are open to letting them use the LTS branch in the official NumPy repository to coordinate that. If you are a NumPy user who requires ongoing Python 2 support in 2020 or later, then please contact your vendor. If you are a vendor who wishes to continue to support NumPy on Python 2 in 2020+, please get in touch; ideally we'd like you to get involved in maintaining the LTS before it actually hits end-of-life, so we can make a clean handoff. To minimize disruption, running 'pip install numpy' on Python 2 will continue to give the last working release in perpetuity; but after January 1, 2019 it may not contain the latest features, and after January 1, 2020 it may not contain the latest bug fixes. For more information on the scientific Python ecosystem's transition to Python-3-only, see: http://www.python3statement.org/ For more information on porting your code to run on Python 3, see: https://docs.python.org/3/howto/pyporting.html ---- Thoughts? -n On Thu, Nov 9, 2017 at 12:53 PM, Marten van Kerkwijk <m.h.vankerkwijk@gmail.com> wrote:
-- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/4d4ea6148fef59dff9fa0fc8c309496a.jpg?s=120&d=mm&r=g)
'pip install ... will' to 'pip install ... should' especially for 2.7 users it's rarer to have an up to date enough pip (9+) to respect the requires_python metadata. A mention to the py3statement would be appreciated :-) especially if you decide to sign it. You might want to also be a bit more positive on the python 2 burden (that's the stick) and add a phrase about "allowing you to implement features for python 3 users that are incompatible with having compatibility with python 2". The "supported by the community" should (IMHO) be made slightly clearer, as well as what bug fix you expect core dev to _do_ vs _accept_, but that can be a separate document somewhere else to refer to. Thanks ! -- M On Nov 9, 2017 17:52, "Nathaniel Smith" <njs@pobox.com> wrote: Fortunately we can wait until we're a bit closer before we have to make any final decision on the version numbering :-) Right now though it would be good to start communicating to users/downstreams about whatever our plans our though, so they can make plans. Here's a first attempt at some text we can put in the documentation and point people to -- any thoughts, on either the plan or the wording? ---- DRAFT TEXT - NOT FINAL - DO NOT POST THIS TO HACKERNEWS OK? OK ---- The Python core team plans to stop supporting Python 2 in 2020. The NumPy project has supported both Python 2 and Python 3 in parallel since 2010, and has found that supporting Python 2 is an increasing burden on our limited resources; thus, we plan to eventually drop Python 2 support as well. Now that we're entering the final years of community-supported Python 2, the NumPy project wants to clarify our plans, with the goal of to helping our downstream ecosystem make plans and accomplish the transition with as little disruption as possible. Our current plan is as follows: Until **December 31, 2018**, all NumPy releases will fully support both Python 2 and Python 3. Starting on **January 1, 2019**, any new feature releases will support only Python 3. The last Python-2-supporting release will be designated as a long-term support (LTS) release, meaning that we will continue to merge bug-fixes and make bug-fix releases for a longer period than usual. Specifically, it will be supported by the community until **December 31, 2019**. On **January 1, 2020** we will raise a toast to Python 2, and community support for the last Python-2-supporting release will come to an end. However, it will continue to be available on PyPI indefinitely, and if any commercial vendors wish to extend the LTS support past this point then we are open to letting them use the LTS branch in the official NumPy repository to coordinate that. If you are a NumPy user who requires ongoing Python 2 support in 2020 or later, then please contact your vendor. If you are a vendor who wishes to continue to support NumPy on Python 2 in 2020+, please get in touch; ideally we'd like you to get involved in maintaining the LTS before it actually hits end-of-life, so we can make a clean handoff. To minimize disruption, running 'pip install numpy' on Python 2 will continue to give the last working release in perpetuity; but after January 1, 2019 it may not contain the latest features, and after January 1, 2020 it may not contain the latest bug fixes. For more information on the scientific Python ecosystem's transition to Python-3-only, see: http://www.python3statement.org/ For more information on porting your code to run on Python 3, see: https://docs.python.org/3/howto/pyporting.html ---- Thoughts? -n On Thu, Nov 9, 2017 at 12:53 PM, Marten van Kerkwijk <m.h.vankerkwijk@gmail.com> wrote:
-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
![](https://secure.gravatar.com/avatar/96dd777e397ab128fedab46af97a3a4a.jpg?s=120&d=mm&r=g)
On Thu, Nov 9, 2017 at 6:52 PM, Nathaniel Smith <njs@pobox.com> wrote:
I've put up an NEP <https://github.com/numpy/numpy/pull/10006> for the proposal. <snip> Chuck
![](https://secure.gravatar.com/avatar/998f5c5403f3657437a3afbf6a16e24b.jpg?s=120&d=mm&r=g)
On Nov 9, 2017 20:52, "Nathaniel Smith" <njs@pobox.com> wrote: Fortunately we can wait until we're a bit closer before we have to make any final decision on the version numbering :-) Right now though it would be good to start communicating to users/downstreams about whatever our plans our though, so they can make plans. Here's a first attempt at some text we can put in the documentation and point people to -- any thoughts, on either the plan or the wording? ---- DRAFT TEXT - NOT FINAL - DO NOT POST THIS TO HACKERNEWS OK? OK ---- The Python core team plans to stop supporting Python 2 in 2020. The NumPy project has supported both Python 2 and Python 3 in parallel since 2010, and has found that supporting Python 2 is an increasing burden on our limited resources; thus, we plan to eventually drop Python 2 support as well. Now that we're entering the final years of community-supported Python 2, the NumPy project wants to clarify our plans, with the goal of to helping our downstream ecosystem make plans and accomplish the transition with as little disruption as possible. Our current plan is as follows: Until **December 31, 2018**, all NumPy releases will fully support both Python 2 and Python 3. Starting on **January 1, 2019**, any new feature releases will support only Python 3. The last Python-2-supporting release will be designated as a long-term support (LTS) release, meaning that we will continue to merge bug-fixes and make bug-fix releases for a longer period than usual. Specifically, it will be supported by the community until **December 31, 2019**. On **January 1, 2020** we will raise a toast to Python 2, and community support for the last Python-2-supporting release will come to an end. However, it will continue to be available on PyPI indefinitely, and if any commercial vendors wish to extend the LTS support past this point then we are open to letting them use the LTS branch in the official NumPy repository to coordinate that. If you are a NumPy user who requires ongoing Python 2 support in 2020 or later, then please contact your vendor. If you are a vendor who wishes to continue to support NumPy on Python 2 in 2020+, please get in touch; ideally we'd like you to get involved in maintaining the LTS before it actually hits end-of-life, so we can make a clean handoff. To minimize disruption, running 'pip install numpy' on Python 2 will continue to give the last working release in perpetuity; but after January 1, 2019 it may not contain the latest features, and after January 1, 2020 it may not contain the latest bug fixes. For more information on the scientific Python ecosystem's transition to Python-3-only, see: http://www.python3statement.org/ For more information on porting your code to run on Python 3, see: https://docs.python.org/3/howto/pyporting.html ---- Thoughts? -n On Thu, Nov 9, 2017 at 12:53 PM, Marten van Kerkwijk <m.h.vankerkwijk@gmail.com> wrote:
Might it make sense to do this in a synchronized manner with scipy? So both numpy and scipy drop support for python 2 on the first release after December 31 2018, and numpy's first python3-only release comes before (or simultaneously with) scipy's. Then scipy can set is minimum supported numpy version to be the first python3-only version. That allows scipy to have a clean, obvious point where scipy supports only the latest numpy. This will diverge later, but it seems to be a relatively safe place to bring them back into sync.
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Nov 12, 2017 1:12 PM, "Todd" <toddrjen@gmail.com> wrote: Might it make sense to do this in a synchronized manner with scipy? So both numpy and scipy drop support for python 2 on the first release after December 31 2018, and numpy's first python3-only release comes before (or simultaneously with) scipy's. Then scipy can set is minimum supported numpy version to be the first python3-only version. That allows scipy to have a clean, obvious point where scipy supports only the latest numpy. This will diverge later, but it seems to be a relatively safe place to bring them back into sync. That's really a question for the scipy devs on the scipy mailing list. There's substantial overlap between the numpy and scipy communities, but not everyone is on both lists and they're distinct projects that sometimes have unique issues to worry about. I'd like to see numpy's downstream projects become more aggressive about dropping support for old numpy versions in general, but there's no technical reason that scipy's first 3-only release couldn't continue to support one or more numpy 2+3 releases. -n
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
Apparently this is actually uncontroversial, the discussion's died down (see also the comments on Chuck's PR [1]), and anyone who wanted to object has had more than a week to do so, so... I guess we can say this is what's happening and start publicizing it to our users! A direct link to the rendered NEP in the repo is: https://github.com/numpy/numpy/blob/master/doc/neps/dropping-python2.7-propo... (I guess that at some point it will also show up on docs.scipy.org.) -n [1] https://github.com/numpy/numpy/pull/10006 On Thu, Nov 9, 2017 at 5:52 PM, Nathaniel Smith <njs@pobox.com> wrote:
-- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/871426dddc1a9f702316c1ca03a33d9b.jpg?s=120&d=mm&r=g)
Since Konrad Hinsen no longer follows the NumPy discussion list for lack of time, he has not posted here - but he has commented about this on Twitter and written up a good blog post: http://blog.khinsen.net/posts/2017/11/16/a-plea-for-stability-in-the-scipy-e... In a field where scientific code is expected to last and be developed on a timescale of decades, the change of pace with Python 2 and 3 is harder to handle. Regards, Peter On Wed, Nov 15, 2017 at 2:19 AM, Nathaniel Smith <njs@pobox.com> wrote:
![](https://secure.gravatar.com/avatar/81e62cb212edf2a8402c842b120d9f31.jpg?s=120&d=mm&r=g)
I've actually engaged with him on Twitter too but just to repeat one part here : scarce academic resources to maintain code is not an argument. Out of all places, it is academia that should have come up with or should have contributed greatly to open-source instead of paper-writing frenzy among each other. As many people already have written many blog posts/tweets etc. academia does not value software as scientific products but demand software continuously. As an ex-academician I can safely ignore that argument.Scientific code is expected to be maintained properly. I understand the sentiment but blocking progress because of legacy code is a burden on the posterity and a luxury for the past. On Fri, Nov 17, 2017 at 1:35 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
![](https://secure.gravatar.com/avatar/96dd777e397ab128fedab46af97a3a4a.jpg?s=120&d=mm&r=g)
On Fri, Nov 17, 2017 at 5:35 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
Konrad has been making that argument for a long time, and I don't know what the long term solution is. However, the use of Fortran as a benchmark of stability is a bit misleading. Fortran was undergoing rapid development in the years before Fortran 77, with Fortran II, DEC Fortran, and Rational Fortran being somewhat incompatible variations on the theme, and it was only with the standardization of the language that stability could be assumed. And if you wrote in a language that didn't survive the winnowing, Algol-68 for instance, you were in trouble. But even apart from from the languages, the hardware was changing, with different floating point formats on different hardware, so that prior to IEEE-754 the results of computations carried out on one machine were not always the same as results on another. Such differences still persist, with dependencies on math library and compiler versions, choice of rounding, and hardware, although with much reduced in effect. The C language is another example, the lack of a C99 standard, maintained compiler on Windows for Python 2.7 being one of the considerations in dropping support for Python 2. And let us not overlook C++, which, IMHO, has only reached its Fortran 77 equivalent with C++11. I think the take away here is that we are still in the very early days of scientific computing with Python, it has really only been coming on strong for maybe five years. Those of us, including Konrad, who were early adopters scouting the terrain, were bound to end up with a few arrows in our back. Early adoption is always a gamble, with a tradeoff between the risk of choosing a language that mutates or dies, versus the payoff of using a language that blossoms and makes life easier. In my mind, Python 3.5 is the rough equivalent of Fortran 77, or maybe Fortran 95, and I don't know when the Python scientific stack will truly settle, but I expect it will be sometime in the next 5-10 years. At that point, we may want to look at having a "reference" version of NumPy, but I think it is still too early to make such a guarantee, although we do try to avoid being too disruptive while still making progress. These considerations are probably cold comfort to folks like Konrad who have extensive code bases, some probably dating back to Numeric, that they need to maintain, but I do think things will get better. <snip> Chuck
![](https://secure.gravatar.com/avatar/5dde29b54a3f1b76b2541d0a4a9b232c.jpg?s=120&d=mm&r=g)
On Fri, Nov 17, 2017 at 4:35 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
sure -- but I do not get what the problem is here! from his post: """ The disappearance of Python 2 will leave much scientific software orphaned, and many published results irreproducible. """ This is an issue we should all be concerned about, and, in fact, the scipy community has been particularly active in the reproducibility realm. BUT: that statement makes NO SENSE. dropping Python2 support in numpy (or any other package) means that newer versions of numpy will not run on py2 -- but if you want to reproduce results, you need to run the code WITH THE VERSION THAT WAS USED IN THE FIRST PLACE. So if someone publishes something based on code written in python2.7 and numpy 1.13, then it is not helpful for reproducibility at all for numpy 1.18 (or 2.*, or whatever we call it) to run on python2. So there is no issue here. Potential issues will arise post 2020, when maybe python2.7 (and numpy 1.13) will no longer run on an up to date OS. But the OS vendors do a pretty good job of backward compatibility -- so we've got quite a few years to go on that. And it will also be important that older versions of packages are available -- but as long as we don't delete the archives, that should be the case for a good long while. So not sure what the problem is here. note relevant for reproducibility,but I have always been puzzled that folks often desperately want to run the very latest numpy on an old Python (2.6, 1.5, ....) if you can update your numy, update your darn Python too! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
![](https://secure.gravatar.com/avatar/d9ac9213ada4a807322f99081296784b.jpg?s=120&d=mm&r=g)
On Fri, Nov 17, 2017, at 13:12, Chris Barker wrote: that are already invested in NumPy or SciPy's APIs, those that will rely on it in the future, and those that are still undecided about whether to use these tools. For those heavily invested such as Konrad, API changes and a language upgrade may seem like a particularly bad situation. Heck, none of us enjoyed having to port all of our code to Python 3, but in reality the changes required were much fewer than commonly imagined and are documented. But in the same way you cause some pain by changing APIs, *not* changing APIs carries a penalty too, more for the other groups I mentioned. The ability to change APIs, albeit slowly, allows cleaner and more intuitive future code, fewer surprises, and makes the environment much more enjoyable to use. We can do a better job of advertising NumPy's deprecation policy. A quick Google search for "x deprecation policy" didn't manage to find it, but did pick up: - http://scikit-learn.org/stable/developers/contributing.html#deprecation - http://scikit-image.org/docs/dev/contribute.html#deprecation-cycle - https://docs.scipy.org/doc/scipy-1.0.0/reference/dev/deprecations.html All the above packages, as well as NumPy, include a section on API changes in their release notes. We may benefit from standardizing deprecation conventions across the community, so that there is a very clear expectation on how often to run your code to be able to see all relevant warnings and fix them. Best regards Stéfan
![](https://secure.gravatar.com/avatar/4d4ea6148fef59dff9fa0fc8c309496a.jpg?s=120&d=mm&r=g)
Hi all, Apologies if this mail appear out of thread I just subscribed to respond.
Happy to see NumPy at least having this conversation ! I agree with Thomas, we're pretty loose on what dropping support means; one of the main reason for the Python-3-Statement is communication to users and other project; and covey that there is a strong intent that you have until 2020 to get ready (if not before). The voice of NumPy have a huge weight in the balance. I quickly went through the thread and have a few responses:
And that is why you should decide of doing it at some point, and telling it to the world and the sooner you decide and advertise it (regardless of effective "deadline" the better. The Scientific Python is in a catch 22 position; Most of the ecosystem will not drop 2.7 because "numpy is still compatible python 2.7", and numpy does not drop it because "many packages rely on numpy support for 2.7".
We'd have to make sure we could persuade pypi to give the older version for Windows, by default - I don't know if that is possible.
I don't think there is, though if you tag a release with `requires_python>3.3`, then pip 9+ users on python 2.7 will not even realise there are new release compatible only with 3.3+. Technically you can make numpy a meta-package that requires numpy-27 on windows only... but it has its own drawbacks.
These two are _in practice_ against each other; if you do major cleaning then most backports will have a hard time being auto applied (just a warning). If you have a team that want to do a LTS I would suggest "cleaning" only when you are actually touching some code and the python-2 support code is in the way. not cleaning "for the sake of cleaning" at least until the 2 code base are far enough. We have a bot on Jupyter/Matplotlib that help to backport PRs to older branches. I'm happy to open it to the numpy org if it helps.
The hardest part is communication. And not just "We're dropping in 2020" but also "We still care about you, 2.7 users", ans especially tell 2.7 users and old pip users how to correctly pip their dependency on numpy (to still get LTS if LTS there is) One more thing, there is a lot of of discussion about a "Volonteer LTS", you may also want to consider a partnership with a company for an officially recommended commercial offer. Thanks, -- Matthias On Thu, Nov 9, 2017 at 10:21 AM, Nathaniel Smith <njs@pobox.com> wrote:
![](https://secure.gravatar.com/avatar/96dd777e397ab128fedab46af97a3a4a.jpg?s=120&d=mm&r=g)
On Thu, Nov 9, 2017 at 11:24 AM, Matthias Bussonnier < bussonniermatthias@gmail.com> wrote:
One thing worth considering might be making the release that drops 2.7 NumPy 2.0 just so that there is a clear break point. Then if someone wants to continue the 1.x line of releases supporting 2.7 they can do so. ISTR that git now has some features that might aid in that. If the `requires` bit works well with pip then we can also share the same pip page (maybe). <snip> Chuck
![](https://secure.gravatar.com/avatar/4aa06fcfdb62e99be5fce44abd631406.jpg?s=120&d=mm&r=g)
On Nov 9, 2017, at 13:35, Charles R Harris <charlesr.harris@gmail.com> wrote:
One thing worth considering might be making the release that drops 2.7 NumPy 2.0 just so that there is a clear break point. Then if someone wants to continue the 1.x line of releases supporting 2.7 they can do so. ISTR that git now has some features that might aid in that. If the `requires` bit works well with pip then we can also share the same pip page (maybe).
I personally think this is definitely advisable. FWIW Bokeh will definitely be bumping major numbers when dropping Python 2 support, or classic notebook support, etc. Bryan
![](https://secure.gravatar.com/avatar/851ff10fbb1363b7d6111ac60194cc1c.jpg?s=120&d=mm&r=g)
In astropy we had a similar discussion about version numbers, and decided to make 2.0 the LTS that still supports python 2.7 and 3.0 the first that does not. If we're discussing jumping a major number, we could do the same for numpy. (Admittedly, it made a bit more sense with the numbering scheme astropy had adopted anyway.) -- Marten
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
Fortunately we can wait until we're a bit closer before we have to make any final decision on the version numbering :-) Right now though it would be good to start communicating to users/downstreams about whatever our plans our though, so they can make plans. Here's a first attempt at some text we can put in the documentation and point people to -- any thoughts, on either the plan or the wording? ---- DRAFT TEXT - NOT FINAL - DO NOT POST THIS TO HACKERNEWS OK? OK ---- The Python core team plans to stop supporting Python 2 in 2020. The NumPy project has supported both Python 2 and Python 3 in parallel since 2010, and has found that supporting Python 2 is an increasing burden on our limited resources; thus, we plan to eventually drop Python 2 support as well. Now that we're entering the final years of community-supported Python 2, the NumPy project wants to clarify our plans, with the goal of to helping our downstream ecosystem make plans and accomplish the transition with as little disruption as possible. Our current plan is as follows: Until **December 31, 2018**, all NumPy releases will fully support both Python 2 and Python 3. Starting on **January 1, 2019**, any new feature releases will support only Python 3. The last Python-2-supporting release will be designated as a long-term support (LTS) release, meaning that we will continue to merge bug-fixes and make bug-fix releases for a longer period than usual. Specifically, it will be supported by the community until **December 31, 2019**. On **January 1, 2020** we will raise a toast to Python 2, and community support for the last Python-2-supporting release will come to an end. However, it will continue to be available on PyPI indefinitely, and if any commercial vendors wish to extend the LTS support past this point then we are open to letting them use the LTS branch in the official NumPy repository to coordinate that. If you are a NumPy user who requires ongoing Python 2 support in 2020 or later, then please contact your vendor. If you are a vendor who wishes to continue to support NumPy on Python 2 in 2020+, please get in touch; ideally we'd like you to get involved in maintaining the LTS before it actually hits end-of-life, so we can make a clean handoff. To minimize disruption, running 'pip install numpy' on Python 2 will continue to give the last working release in perpetuity; but after January 1, 2019 it may not contain the latest features, and after January 1, 2020 it may not contain the latest bug fixes. For more information on the scientific Python ecosystem's transition to Python-3-only, see: http://www.python3statement.org/ For more information on porting your code to run on Python 3, see: https://docs.python.org/3/howto/pyporting.html ---- Thoughts? -n On Thu, Nov 9, 2017 at 12:53 PM, Marten van Kerkwijk <m.h.vankerkwijk@gmail.com> wrote:
-- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/4d4ea6148fef59dff9fa0fc8c309496a.jpg?s=120&d=mm&r=g)
'pip install ... will' to 'pip install ... should' especially for 2.7 users it's rarer to have an up to date enough pip (9+) to respect the requires_python metadata. A mention to the py3statement would be appreciated :-) especially if you decide to sign it. You might want to also be a bit more positive on the python 2 burden (that's the stick) and add a phrase about "allowing you to implement features for python 3 users that are incompatible with having compatibility with python 2". The "supported by the community" should (IMHO) be made slightly clearer, as well as what bug fix you expect core dev to _do_ vs _accept_, but that can be a separate document somewhere else to refer to. Thanks ! -- M On Nov 9, 2017 17:52, "Nathaniel Smith" <njs@pobox.com> wrote: Fortunately we can wait until we're a bit closer before we have to make any final decision on the version numbering :-) Right now though it would be good to start communicating to users/downstreams about whatever our plans our though, so they can make plans. Here's a first attempt at some text we can put in the documentation and point people to -- any thoughts, on either the plan or the wording? ---- DRAFT TEXT - NOT FINAL - DO NOT POST THIS TO HACKERNEWS OK? OK ---- The Python core team plans to stop supporting Python 2 in 2020. The NumPy project has supported both Python 2 and Python 3 in parallel since 2010, and has found that supporting Python 2 is an increasing burden on our limited resources; thus, we plan to eventually drop Python 2 support as well. Now that we're entering the final years of community-supported Python 2, the NumPy project wants to clarify our plans, with the goal of to helping our downstream ecosystem make plans and accomplish the transition with as little disruption as possible. Our current plan is as follows: Until **December 31, 2018**, all NumPy releases will fully support both Python 2 and Python 3. Starting on **January 1, 2019**, any new feature releases will support only Python 3. The last Python-2-supporting release will be designated as a long-term support (LTS) release, meaning that we will continue to merge bug-fixes and make bug-fix releases for a longer period than usual. Specifically, it will be supported by the community until **December 31, 2019**. On **January 1, 2020** we will raise a toast to Python 2, and community support for the last Python-2-supporting release will come to an end. However, it will continue to be available on PyPI indefinitely, and if any commercial vendors wish to extend the LTS support past this point then we are open to letting them use the LTS branch in the official NumPy repository to coordinate that. If you are a NumPy user who requires ongoing Python 2 support in 2020 or later, then please contact your vendor. If you are a vendor who wishes to continue to support NumPy on Python 2 in 2020+, please get in touch; ideally we'd like you to get involved in maintaining the LTS before it actually hits end-of-life, so we can make a clean handoff. To minimize disruption, running 'pip install numpy' on Python 2 will continue to give the last working release in perpetuity; but after January 1, 2019 it may not contain the latest features, and after January 1, 2020 it may not contain the latest bug fixes. For more information on the scientific Python ecosystem's transition to Python-3-only, see: http://www.python3statement.org/ For more information on porting your code to run on Python 3, see: https://docs.python.org/3/howto/pyporting.html ---- Thoughts? -n On Thu, Nov 9, 2017 at 12:53 PM, Marten van Kerkwijk <m.h.vankerkwijk@gmail.com> wrote:
-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion
![](https://secure.gravatar.com/avatar/96dd777e397ab128fedab46af97a3a4a.jpg?s=120&d=mm&r=g)
On Thu, Nov 9, 2017 at 6:52 PM, Nathaniel Smith <njs@pobox.com> wrote:
I've put up an NEP <https://github.com/numpy/numpy/pull/10006> for the proposal. <snip> Chuck
![](https://secure.gravatar.com/avatar/998f5c5403f3657437a3afbf6a16e24b.jpg?s=120&d=mm&r=g)
On Nov 9, 2017 20:52, "Nathaniel Smith" <njs@pobox.com> wrote: Fortunately we can wait until we're a bit closer before we have to make any final decision on the version numbering :-) Right now though it would be good to start communicating to users/downstreams about whatever our plans our though, so they can make plans. Here's a first attempt at some text we can put in the documentation and point people to -- any thoughts, on either the plan or the wording? ---- DRAFT TEXT - NOT FINAL - DO NOT POST THIS TO HACKERNEWS OK? OK ---- The Python core team plans to stop supporting Python 2 in 2020. The NumPy project has supported both Python 2 and Python 3 in parallel since 2010, and has found that supporting Python 2 is an increasing burden on our limited resources; thus, we plan to eventually drop Python 2 support as well. Now that we're entering the final years of community-supported Python 2, the NumPy project wants to clarify our plans, with the goal of to helping our downstream ecosystem make plans and accomplish the transition with as little disruption as possible. Our current plan is as follows: Until **December 31, 2018**, all NumPy releases will fully support both Python 2 and Python 3. Starting on **January 1, 2019**, any new feature releases will support only Python 3. The last Python-2-supporting release will be designated as a long-term support (LTS) release, meaning that we will continue to merge bug-fixes and make bug-fix releases for a longer period than usual. Specifically, it will be supported by the community until **December 31, 2019**. On **January 1, 2020** we will raise a toast to Python 2, and community support for the last Python-2-supporting release will come to an end. However, it will continue to be available on PyPI indefinitely, and if any commercial vendors wish to extend the LTS support past this point then we are open to letting them use the LTS branch in the official NumPy repository to coordinate that. If you are a NumPy user who requires ongoing Python 2 support in 2020 or later, then please contact your vendor. If you are a vendor who wishes to continue to support NumPy on Python 2 in 2020+, please get in touch; ideally we'd like you to get involved in maintaining the LTS before it actually hits end-of-life, so we can make a clean handoff. To minimize disruption, running 'pip install numpy' on Python 2 will continue to give the last working release in perpetuity; but after January 1, 2019 it may not contain the latest features, and after January 1, 2020 it may not contain the latest bug fixes. For more information on the scientific Python ecosystem's transition to Python-3-only, see: http://www.python3statement.org/ For more information on porting your code to run on Python 3, see: https://docs.python.org/3/howto/pyporting.html ---- Thoughts? -n On Thu, Nov 9, 2017 at 12:53 PM, Marten van Kerkwijk <m.h.vankerkwijk@gmail.com> wrote:
Might it make sense to do this in a synchronized manner with scipy? So both numpy and scipy drop support for python 2 on the first release after December 31 2018, and numpy's first python3-only release comes before (or simultaneously with) scipy's. Then scipy can set is minimum supported numpy version to be the first python3-only version. That allows scipy to have a clean, obvious point where scipy supports only the latest numpy. This will diverge later, but it seems to be a relatively safe place to bring them back into sync.
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Nov 12, 2017 1:12 PM, "Todd" <toddrjen@gmail.com> wrote: Might it make sense to do this in a synchronized manner with scipy? So both numpy and scipy drop support for python 2 on the first release after December 31 2018, and numpy's first python3-only release comes before (or simultaneously with) scipy's. Then scipy can set is minimum supported numpy version to be the first python3-only version. That allows scipy to have a clean, obvious point where scipy supports only the latest numpy. This will diverge later, but it seems to be a relatively safe place to bring them back into sync. That's really a question for the scipy devs on the scipy mailing list. There's substantial overlap between the numpy and scipy communities, but not everyone is on both lists and they're distinct projects that sometimes have unique issues to worry about. I'd like to see numpy's downstream projects become more aggressive about dropping support for old numpy versions in general, but there's no technical reason that scipy's first 3-only release couldn't continue to support one or more numpy 2+3 releases. -n
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
Apparently this is actually uncontroversial, the discussion's died down (see also the comments on Chuck's PR [1]), and anyone who wanted to object has had more than a week to do so, so... I guess we can say this is what's happening and start publicizing it to our users! A direct link to the rendered NEP in the repo is: https://github.com/numpy/numpy/blob/master/doc/neps/dropping-python2.7-propo... (I guess that at some point it will also show up on docs.scipy.org.) -n [1] https://github.com/numpy/numpy/pull/10006 On Thu, Nov 9, 2017 at 5:52 PM, Nathaniel Smith <njs@pobox.com> wrote:
-- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/871426dddc1a9f702316c1ca03a33d9b.jpg?s=120&d=mm&r=g)
Since Konrad Hinsen no longer follows the NumPy discussion list for lack of time, he has not posted here - but he has commented about this on Twitter and written up a good blog post: http://blog.khinsen.net/posts/2017/11/16/a-plea-for-stability-in-the-scipy-e... In a field where scientific code is expected to last and be developed on a timescale of decades, the change of pace with Python 2 and 3 is harder to handle. Regards, Peter On Wed, Nov 15, 2017 at 2:19 AM, Nathaniel Smith <njs@pobox.com> wrote:
![](https://secure.gravatar.com/avatar/81e62cb212edf2a8402c842b120d9f31.jpg?s=120&d=mm&r=g)
I've actually engaged with him on Twitter too but just to repeat one part here : scarce academic resources to maintain code is not an argument. Out of all places, it is academia that should have come up with or should have contributed greatly to open-source instead of paper-writing frenzy among each other. As many people already have written many blog posts/tweets etc. academia does not value software as scientific products but demand software continuously. As an ex-academician I can safely ignore that argument.Scientific code is expected to be maintained properly. I understand the sentiment but blocking progress because of legacy code is a burden on the posterity and a luxury for the past. On Fri, Nov 17, 2017 at 1:35 PM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
![](https://secure.gravatar.com/avatar/96dd777e397ab128fedab46af97a3a4a.jpg?s=120&d=mm&r=g)
On Fri, Nov 17, 2017 at 5:35 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
Konrad has been making that argument for a long time, and I don't know what the long term solution is. However, the use of Fortran as a benchmark of stability is a bit misleading. Fortran was undergoing rapid development in the years before Fortran 77, with Fortran II, DEC Fortran, and Rational Fortran being somewhat incompatible variations on the theme, and it was only with the standardization of the language that stability could be assumed. And if you wrote in a language that didn't survive the winnowing, Algol-68 for instance, you were in trouble. But even apart from from the languages, the hardware was changing, with different floating point formats on different hardware, so that prior to IEEE-754 the results of computations carried out on one machine were not always the same as results on another. Such differences still persist, with dependencies on math library and compiler versions, choice of rounding, and hardware, although with much reduced in effect. The C language is another example, the lack of a C99 standard, maintained compiler on Windows for Python 2.7 being one of the considerations in dropping support for Python 2. And let us not overlook C++, which, IMHO, has only reached its Fortran 77 equivalent with C++11. I think the take away here is that we are still in the very early days of scientific computing with Python, it has really only been coming on strong for maybe five years. Those of us, including Konrad, who were early adopters scouting the terrain, were bound to end up with a few arrows in our back. Early adoption is always a gamble, with a tradeoff between the risk of choosing a language that mutates or dies, versus the payoff of using a language that blossoms and makes life easier. In my mind, Python 3.5 is the rough equivalent of Fortran 77, or maybe Fortran 95, and I don't know when the Python scientific stack will truly settle, but I expect it will be sometime in the next 5-10 years. At that point, we may want to look at having a "reference" version of NumPy, but I think it is still too early to make such a guarantee, although we do try to avoid being too disruptive while still making progress. These considerations are probably cold comfort to folks like Konrad who have extensive code bases, some probably dating back to Numeric, that they need to maintain, but I do think things will get better. <snip> Chuck
![](https://secure.gravatar.com/avatar/5dde29b54a3f1b76b2541d0a4a9b232c.jpg?s=120&d=mm&r=g)
On Fri, Nov 17, 2017 at 4:35 AM, Peter Cock <p.j.a.cock@googlemail.com> wrote:
sure -- but I do not get what the problem is here! from his post: """ The disappearance of Python 2 will leave much scientific software orphaned, and many published results irreproducible. """ This is an issue we should all be concerned about, and, in fact, the scipy community has been particularly active in the reproducibility realm. BUT: that statement makes NO SENSE. dropping Python2 support in numpy (or any other package) means that newer versions of numpy will not run on py2 -- but if you want to reproduce results, you need to run the code WITH THE VERSION THAT WAS USED IN THE FIRST PLACE. So if someone publishes something based on code written in python2.7 and numpy 1.13, then it is not helpful for reproducibility at all for numpy 1.18 (or 2.*, or whatever we call it) to run on python2. So there is no issue here. Potential issues will arise post 2020, when maybe python2.7 (and numpy 1.13) will no longer run on an up to date OS. But the OS vendors do a pretty good job of backward compatibility -- so we've got quite a few years to go on that. And it will also be important that older versions of packages are available -- but as long as we don't delete the archives, that should be the case for a good long while. So not sure what the problem is here. note relevant for reproducibility,but I have always been puzzled that folks often desperately want to run the very latest numpy on an old Python (2.6, 1.5, ....) if you can update your numy, update your darn Python too! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
![](https://secure.gravatar.com/avatar/d9ac9213ada4a807322f99081296784b.jpg?s=120&d=mm&r=g)
On Fri, Nov 17, 2017, at 13:12, Chris Barker wrote: that are already invested in NumPy or SciPy's APIs, those that will rely on it in the future, and those that are still undecided about whether to use these tools. For those heavily invested such as Konrad, API changes and a language upgrade may seem like a particularly bad situation. Heck, none of us enjoyed having to port all of our code to Python 3, but in reality the changes required were much fewer than commonly imagined and are documented. But in the same way you cause some pain by changing APIs, *not* changing APIs carries a penalty too, more for the other groups I mentioned. The ability to change APIs, albeit slowly, allows cleaner and more intuitive future code, fewer surprises, and makes the environment much more enjoyable to use. We can do a better job of advertising NumPy's deprecation policy. A quick Google search for "x deprecation policy" didn't manage to find it, but did pick up: - http://scikit-learn.org/stable/developers/contributing.html#deprecation - http://scikit-image.org/docs/dev/contribute.html#deprecation-cycle - https://docs.scipy.org/doc/scipy-1.0.0/reference/dev/deprecations.html All the above packages, as well as NumPy, include a section on API changes in their release notes. We may benefit from standardizing deprecation conventions across the community, so that there is a very clear expectation on how often to run your code to be able to see all relevant warnings and fix them. Best regards Stéfan
participants (11)
-
Bryan Van de ven
-
Charles R Harris
-
Chris Barker
-
Ilhan Polat
-
Marten van Kerkwijk
-
Matthias Bussonnier
-
Nathaniel Smith
-
Peter Cock
-
Ralf Gommers
-
Stefan van der Walt
-
Todd