On Sat, Apr 15, 2017 at 5:19 PM, Nathaniel Smith wrote:

...

Hi All,

It may be early to discuss dropping support for Python 2.7, but there is a disturbance in the force that suggests that it might be worth looking forward to the year 2020 when Python itself will drop support for 2.7. There is also a website, http://www.python3statement.org, where several

On Fri, Apr 14, 2017 at 5:19 PM, Charles R Harris wrote: projects

...

in the scientific python stack have pledged to be Python 2.7 free by that date. Given that, a preliminary discussion of the subject might be interesting, if only to gather information of where the community currently stands.

One reasonable position would that numpy releases that happen while 2.7 is supported upstream will also support 2.7, and releases after that won't.

From numpy's perspective, I feel like the most important reason to continue supporting 2.7 is our ability to convince people to keep upgrading. (Not the only reason, but the most important.) What I mean is: if we dropped 2.7 support tomorrow then it wouldn't actually make numpy unavailable on python 2.7; it would just mean that lots of users stayed at 1.12 indefinitely. Which is awkward, but it wouldn't be the end of the world – numpy is mature software and 1.12 works pretty well. The big problem IMO would be if this then meant that lots of downstream projects felt that they had to continue supporting 1.12 going forward, which makes it very difficult for us to effectively ship new features or even bug fixes – I mean, we can ship them, but no-one will use them. And if a downstream project finds a bug in numpy and can't upgrade numpy, then the tendency is to work around it instead of reporting it upstream. I think this is the main thing we want to avoid.

+1

This kind of means that we're at the mercy of downstream projects, though – if scipy/pandas/etc. decide they want to support 2.7 until 2022, it might be in our best interest to do the same. But there's a collective action problem here: we want to keep supporting 2.7 so long as they do, but at the same time they may feel they need to keep supporting 2.7 as long as we do. And all of us would prefer to drop 2.7 support sooner rather than later, but we might all get stuck

because we're waiting for someone else to move first.

I don't quite agree about being stuck. These kind of upgrades should and usually do go top of stack to bottom. Something like Jupyter which is mostly an end user tool goes first (they announced 2020 quite a while ago), domain specific packages go at a similar time, then scipy & co, and only after that numpy. Cython will be even later I'm sure - it still supports Python 2.6.

So my suggestion would be that numpy make some official announcement that our plan is to drop support for python 2 immediately after cpython upstream does.

Not quite sure CPython schedule is relevant - important bug fixes haven't been making it into 2.7 for a very long time now, so the only change is the rare security patch.

If worst comes to worst we can always decide to extend it at the time... but if we make the announcement now, then it's less likely that we'll need to :-).

I'd be in favor of putting out a schedule in coordination with scipy/pandas/etc, but it probably should look more like - 2020: what's on http://www.python3statement.org/ now - 2021: scipy / pandas / scikit-learn / etc. - 2022: numpy Ralf

Another interesting project to look at here is django, since they occupy a similar place in the ecosystem (e.g. last I checked numpy and django are the two most-imported python packages on github): https://www.djangoproject.com/weblog/2015/jun/25/roadmap/ Their approach isn't directly applicable, because unlike us they have a strict time-based release schedule, defined support period for each release, and a distinction between regular and long-term support releases, where regular releases act sort of like pre-releases-on-steroids for the next LTS release. But basically what they settled on is philosophically similar to what I'm suggesting: they don't want an LTS to be supporting 2.7 beyond when cpython is supporting it. Then on top of that they don't want to support 2.7 in the regular releases leading up to that LTS either, so the net effect is that their last release with 2.7 support came out last week, and it will be supported until 2020 :-). And another useful precedent I think is that they announced this two years ago, back in 2015; if we make an announcement now, we'll be be giving a similar amount of warning.

-n

-- Nathaniel J. Smith -- https://vorpus.org _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

On Sat, Apr 15, 2017 at 7:02 PM, Nathaniel Smith wrote:

On Fri, Apr 14, 2017 at 10:47 PM, Ralf Gommers wrote:

...

On Sat, Apr 15, 2017 at 5:19 PM, Nathaniel Smith wrote:

...

...

From numpy's perspective, I feel like the most important reason to continue supporting 2.7 is our ability to convince people to keep upgrading. (Not the only reason, but the most important.) What I mean is: if we dropped 2.7 support tomorrow then it wouldn't actually make numpy unavailable on python 2.7; it would just mean that lots of users stayed at 1.12 indefinitely. Which is awkward, but it wouldn't be the end of the world – numpy is mature software and 1.12 works pretty well. The big problem IMO would be if this then meant that lots of downstream projects felt that they had to continue supporting 1.12 going forward, which makes it very difficult for us to effectively ship new features or even bug fixes – I mean, we can ship them, but no-one will use them. And if a downstream project finds a bug in numpy and can't upgrade numpy, then the tendency is to work around it instead of reporting it upstream. I think this is the main thing we want to avoid.

+1

...

This kind of means that we're at the mercy of downstream projects, though – if scipy/pandas/etc. decide they want to support 2.7 until 2022, it might be in our best interest to do the same. But there's a collective action problem here: we want to keep supporting 2.7 so long as they do, but at the same time they may feel they need to keep supporting 2.7 as long as we do. And all of us would prefer to drop 2.7 support sooner rather than later, but we might all get stuck

because we're waiting for someone else to move first.

I don't quite agree about being stuck. These kind of upgrades should and usually do go top of stack to bottom. Something like Jupyter which is mostly an end user tool goes first (they announced 2020 quite a while ago), domain specific packages go at a similar time, then scipy & co, and only after

[...] that

...

numpy. Cython will be even later I'm sure - it still supports Python 2.6.

To make sure we're on the same page about what "2020" means here: the latest release of IPython is 5.0, which came out in July last year. This is the last release that supports py2; they dropped support for py2 in master months ago, and 6.0 (whose schedule has been slipping, but I think should be out Any Time Now?) won't support py2. Their plan is to keep backporting bug fixes to 5.x until the end of 2017; after that the core team won't support py2 at all. And they've also announced that if volunteers want to step up to maintain 5.x after that, then they're willing to keep accepting pull requests until July 2019.

Refs: https://blog.jupyter.org/2016/07/08/ipython-5-0-released/ https://github.com/jupyter/roadmap/blob/master/accepted/migr ation-to-python-3-only.md

I suspect that in practice that "end of 2017" date will the end-of-support date for most intents and purposes. And for numpy with its vaguely defined support periods, I think it makes most sense to talk in terms of release dates;

agreed, release dates makes sense. we don't want to be doing some kind of LTS scheme.

so if we want to compare apples-to-apples, my suggestion is that numpy drops py2 support in 2020 and in that sense ipython dropped py2 support in july last year.

...

...

So my suggestion would be that numpy make some official announcement that our plan is to drop support for python 2 immediately after cpython upstream does.

Not quite sure CPython schedule is relevant - important bug fixes haven't been making it into 2.7 for a very long time now, so the only change is the rare security patch.

Huh? 2.7 gets tons of changes: https://github.com/python/cpyt hon/commits/2.7

You're right. My experience is ending up on bugs.python.org when debugging and the answer to "can this be backported to 2.7" usually being no - but it looks like my experience is skewed by distutils, which is not exactly well maintained.

Officially CPython has 2 modes for releases: "regular support" and "security fixes only". 2.7 is special – it get regular support, and then on top of that it also has a special exception to allow certain kinds of major changes, like the ssl module backports.

If you know of important bug fixes that they're missing then I think they'd like to know :-).

Anyway, the reason the CPython schedule is relevant is that once they drop support, it *will* stop getting security patches, so it will become increasingly impossible to use safely.

For web stuff yes, but not all that relevant for scientific work.

...

...

If worst comes to worst we can always decide to extend it at the time... but if we make the announcement now, then it's less likely that we'll need to :-).

I'd be in favor of putting out a schedule in coordination with scipy/pandas/etc, but it probably should look more like - 2020: what's on http://www.python3statement.org/ now - 2021: scipy / pandas / scikit-learn / etc.

Um... pandas is already on python3statement.org right now :-)

...

- 2022: numpy

Honestly I don't see why we should plan to support python 2 a day longer than our major downstream dependencies. That was the point of my first paragraph: for us the main benefit to supporting 2 is to avoid forcing our downstream dependencies to pin an old numpy. What's that extra year get us if they've already moved on?

The other odd thing about this schedule is that you're suggesting that the organizing principle should be that the stack switches from top-of-stack to bottom... but then you left out the bottom of the stack! :-)

I don't think of Python as part of the stack, because it's not upgradeable for most users (except for with conda). It's more like having a base platform (OS + compilers + Python version) on which you install a scientific stack which has numpy as its lowest level component. Ralf

On Fri, 14 Apr 2017 22:19:42 -0700 Nathaniel Smith wrote:

From numpy's perspective, I feel like the most important reason to continue supporting 2.7 is our ability to convince people to keep upgrading. (Not the only reason, but the most important.) What I mean is: if we dropped 2.7 support tomorrow then it wouldn't actually make numpy unavailable on python 2.7; it would just mean that lots of users stayed at 1.12 indefinitely. Which is awkward, but it wouldn't be the end of the world – numpy is mature software and 1.12 works pretty well. The big problem IMO would be if this then meant that lots of downstream projects felt that they had to continue supporting 1.12 going forward, which makes it very difficult for us to effectively ship new features or even bug fixes – I mean, we can ship them, but no-one will use them.

Everyone using Python 3, which is a large and growing number of people, will be able to use the new features. I think the model you've outlined above -- a kind of "LTS" Numpy version that supports 2.7 (with some amount of maintenance going on, at least to fix important bugs), and later feature releases being 3.x-only, is the right way forward. It will lighten maintenance of later versions, allow the Numpy codebase to use modern Python idioms and stdlib features, and will leave 2.x maintenance to people who really care about it. You may already have heard of it, but Django 1.11, which was just released, is the last feature release to support Python 2. Further feature releases of Django will only support Python 3. https://docs.djangoproject.com/en/1.11/releases/1.11/ Regards Antoine.

What do we think about the trade-offs of having a shared 2.7/3.x codebase going forward? As Python3 adds more nontrivial features, keeping compatibility with 2.7 becomes more burdensome. Will there be a separate py2-numpy branch/repo at some point before ending support? On Apr 15, 2017 4:48 AM, "Julian Taylor" wrote:

On 15.04.2017 02:19, Charles R Harris wrote:

...

Hi All,

It may be early to discuss dropping support for Python 2.7, but there is a disturbance in the force that suggests that it might be worth looking forward to the year 2020 when Python itself will drop support for 2.7. There is also a website, http://www.python3statement.org http://www.python3statement.org/, where several projects in the scientific python stack have pledged to be Python 2.7 free by that date. Given that, a preliminary discussion of the subject might be interesting, if only to gather information of where the community currently stands.

Chuck

I am very against planning to drop it. Numpy is the lowest part of the scipy stack so it is not our decision to do so and we don't gain that much by doing so. Lets discuss this in 3 years or when the distributions kick out python2.7 (which won't happen before ~2022). There is no point doing so now. Also PyPy does not plan on dropping 2.7 by that time.

Also before we even consider this we need to fix our python3 support. This means getting the IO functions (https://github.com/numpy/numpy/pull/4208) in order and adding a string type that people are less reluctant to use than the 4 byte unicode we currently offer. _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

On 15.04.2017 16:17, Marten van Kerkwijk wrote:

Hi All,

I think Nathaniel had a good summary. My own 2¢ are mostly about the burden of supporting python2. I have only recently attempted to make changes in the C codebase of numpy and one of the reasons I found this more than a little daunting is the complex web of include files. In this respect, the python3/2 split is certainly not the biggest hindrance, but it was also not particularly helpful for understanding to have "translations" of python2 macros to python3 equivalents in npy_3kcompat.h: for newcomers, it would seem helpful if they could read the Python3 C-API and be able to understand what is going on.

Of course, the above also proves Julian's point: for strings in particular, numpy still has a bit to go to be fully python3-ized.

Finally, as for pypy: they just made a huge effort to become compatible with python3; is their plan really to stick with python2 much beyond 2020?

http://doc.pypy.org/en/latest/faq.html#how-long-will-pypy-support-python2 According to that Python2 support will be available as long as PyPy itself exists.