On 06.11.2017 11:10, Ralf Gommers wrote:

On Mon, Nov 6, 2017 at 7:25 AM, Charles R Harris mailto:charlesr.harris@gmail.com> wrote:

Hi All,

Thought I'd toss this out there. I'm tending towards better sooner than later in dropping Python 2.7 support as we are starting to run up against places where we would like to use Python 3 features. That is particularly true on Windows where the 2.7 compiler is really old and lacks C99 compatibility.

This is probably the most pressing reason to drop 2.7 support. We seem to be expending a lot of effort lately on this stuff. I was previously advocating being more conservative than the timeline you now propose, but this is the pain point that I think gets me over the line.

Would dropping python2 support for windows earlier than the other platforms a reasonable approach? I am not a big fan of to dropping python2 support before 2020, but I have no issue with dropping python2 support on windows earlier as it is our largest pain point.

Hi, On Wed, Nov 8, 2017 at 7:08 PM, Julian Taylor wrote:

On 06.11.2017 11:10, Ralf Gommers wrote:

...

On Mon, Nov 6, 2017 at 7:25 AM, Charles R Harris mailto:charlesr.harris@gmail.com> wrote:

Hi All,

Thought I'd toss this out there. I'm tending towards better sooner than later in dropping Python 2.7 support as we are starting to run up against places where we would like to use Python 3 features. That is particularly true on Windows where the 2.7 compiler is really old and lacks C99 compatibility.

This is probably the most pressing reason to drop 2.7 support. We seem to be expending a lot of effort lately on this stuff. I was previously advocating being more conservative than the timeline you now propose, but this is the pain point that I think gets me over the line.

Would dropping python2 support for windows earlier than the other platforms a reasonable approach? I am not a big fan of to dropping python2 support before 2020, but I have no issue with dropping python2 support on windows earlier as it is our largest pain point.

I wonder about this too. I can imagine there are a reasonable number of people using older Linux distributions on which they cannot upgrade to a recent Python 3, but is that likely to be true for Windows? We'd have to make sure we could persuade pypi to give the older version for Windows, by default - I don't know if that is possible. Cheers, Matthew

On Thu, Nov 9, 2017 at 12:15 PM, Nathaniel Smith wrote:

On Nov 8, 2017 16:51, "Matthew Brett" wrote:

Hi,

On Wed, Nov 8, 2017 at 7:08 PM, Julian Taylor wrote:

...

On 06.11.2017 11:10, Ralf Gommers wrote:

...

On Mon, Nov 6, 2017 at 7:25 AM, Charles R Harris mailto:charlesr.harris@gmail.com> wrote:

Hi All,

Thought I'd toss this out there. I'm tending towards better sooner than later in dropping Python 2.7 support as we are starting to run up against places where we would like to use Python 3 features. That is particularly true on Windows where the 2.7 compiler is really old and lacks C99 compatibility.

This is probably the most pressing reason to drop 2.7 support. We seem to be expending a lot of effort lately on this stuff. I was previously advocating being more conservative than the timeline you now propose, but this is the pain point that I think gets me over the line.

Would dropping python2 support for windows earlier than the other platforms a reasonable approach? I am not a big fan of to dropping python2 support before 2020, but I have no issue with dropping python2 support on windows earlier as it is our largest pain point.

I wonder about this too. I can imagine there are a reasonable number of people using older Linux distributions on which they cannot upgrade to a recent Python 3,

My impression is that this is increasingly rare, actually. I believe RHEL is still shipping 2.6 by default, which we've already dropped support for, and if you want RH python then they provide supported 2.7 and 3.latest through exactly the same channels. Ubuntu 14.04 is end-of-life in April 2019, so pretty irrelevant if we're talking about 2019 for dropping support, and 16.04 ships with 3.5. Plus with docker, conda, PPAs, etc., getting a recent python is easier than its ever been.

...

but

is that likely to be true for Windows?

We'd have to make sure we could persuade pypi to give the older version for Windows, by default - I don't know if that is possible.

Currently it's not – if pip doesn't see a Windows wheel, it'll try downloading and building an sdist. There's a mechanism for sdists to declare what version of python they support but (thanks to the jupyter folks for implementing this), but that's all. The effect is that if we release a version that drops support for py2 entirely, then 'pip install' on py2 will continue to work and give the last supported version, but if we release a version that drops py2 on Windows but keeps it on other platforms then 'pip install' on py2 on Windows will just stop working entirely.

This is possible to fix – it's just software – but I'm not volunteering...

Given the release cycle of pip (slow) and the bandwidth required to implement this, I think that this is likely a showstopper for Windows-only-3.x-only. Another consideration is that choices made by numpy tend to propagate to the rest of the ecosystem, and support for Python versions that's OS-independent is nicer than Windows special-casing. And yet another is that when we do finally drop 2.7, I think we'd want to get the full benefits of doing so. That's new 3.x features (@ in particular), cleaning up lots of support code, etc. For those reasons I think we should balance the pain and benefits of 2.7 support and just pick a date to drop it completely, not just on Windows. Regarding http://www.python3statement.org/: I'd say that as long as there are people who want to spend their energy on the LTS release (contributors *and* enough maintainer power to review/merge/release), we should not actively prevent them from doing that. Ralf

Hello, I have a question and hope that you can help me. The doc for vstack mentions that "this function continues to be supported for backward compatibility, but you should prefer np.concatenate or np.stack." Using vstack was convenient because "the arrays must have the same shape along all but the first axis." So it was possible to stack an array (3,) and (2, 3) to a (3, 3) array without using e.g. atleast_2d on the (3,) array. Is there a possibility to mimic that behavior with np.concatenate or np.stack? Joe

On Thu, Nov 9, 2017 at 1:30 AM, Joe wrote:

Hello,

I have a question and hope that you can help me.

The doc for vstack mentions that "this function continues to be supported

for backward compatibility, but you should prefer np.concatenate or np.stack."

Using vstack was convenient because "the arrays must have the same shape

along all but the first axis."

So it was possible to stack an array (3,) and (2, 3) to a (3, 3) array

without using e.g. atleast_2d on the (3,) array.

Is there a possibility to mimic that behavior with np.concatenate or

np.stack? Quite frankly, I ignore the documentation as I think it's recommendation is wrong in these cases. Vive la vstack! -- Robert Kern

On Wed, Nov 8, 2017 at 2:50 PM, Matthew Brett wrote:

Hi,

On Wed, Nov 8, 2017 at 7:08 PM, Julian Taylor wrote:

...

On 06.11.2017 11:10, Ralf Gommers wrote:

...

On Mon, Nov 6, 2017 at 7:25 AM, Charles R Harris mailto:charlesr.harris@gmail.com> wrote:

Hi All,

Thought I'd toss this out there. I'm tending towards better sooner than later in dropping Python 2.7 support as we are starting to run up against places where we would like to use Python 3 features. That is particularly true on Windows where the 2.7 compiler is really old and lacks C99 compatibility.

This is probably the most pressing reason to drop 2.7 support. We seem to be expending a lot of effort lately on this stuff. I was previously advocating being more conservative than the timeline you now propose, but this is the pain point that I think gets me over the line.

Would dropping python2 support for windows earlier than the other platforms a reasonable approach? I am not a big fan of to dropping python2 support before 2020, but I have no issue with dropping python2 support on windows earlier as it is our largest pain point.

I wonder about this too. I can imagine there are a reasonable number of people using older Linux distributions on which they cannot upgrade to a recent Python 3, but is that likely to be true for Windows?

We'd have to make sure we could persuade pypi to give the older version for Windows, by default - I don't know if that is possible.

Pip repo names and actual module names don't have to be the same. One potential work-around would be to make a 'numpylts' repo on PyPi which is the 1.17 version with support for Python 2.7 and bug-fix releases as required. This will still cause regressions but it's a matter of modifying `requirements.txt` in downstream Python 2.7 packages and not much else. E.g. in `requirements.txt`: numpy; python_version>"3.0" numpylts; python_version<"3.0" In both cases you still call `import numpy` in the code. Robert -- Robert McLeod, Ph.D. robbmcleod@gmail.com robbmcleod@protonmail.com www.entropyreduction.al

For this to be efficient, it should be done soon enough to allow downstream projects to adapt their requirements.txt. Release managers: how much more effort would it be to upload current numpy to both numpy and numpylts?

I'm not quite sure I see the point. you would ask downstream to change `numpy` to `numpylts` instead of `numpy` to `numpy<2` ? Also I think then you have the risk of having for example pandas saying `numpy<2` and scipy saying `numpylts` and now the pasckages are incompatible ? -- M On Mon, Nov 13, 2017 at 12:11 AM, Daπid wrote:

On 10 November 2017 at 23:03, Robert McLeod wrote:

...

E.g. in `requirements.txt`:

numpy; python_version>"3.0" numpylts; python_version<"3.0"

In both cases you still call `import numpy` in the code.

For this to be efficient, it should be done soon enough to allow downstream projects to adapt their requirements.txt.

Release managers: how much more effort would it be to upload current numpy to both numpy and numpylts?

/David.

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

The trouble is PyPi doesn't allow multiple branches. So if you upload NumPy 2.0 wheels, then you cannot turn around and upload 1.18.X bug-fix patches. At least, this is my understanding of PyPi.

That's perfectly feasible. We've been maintaining a 6.x (Python 3 only) and 5.x (Python2+3) of IPython for about a year now. -- M On Mon, Nov 13, 2017 at 10:08 AM, Robert McLeod wrote:

On Mon, Nov 13, 2017 at 7:47 AM, Matthias Bussonnier wrote:

...

...

For this to be efficient, it should be done soon enough to allow downstream projects to adapt their requirements.txt. Release managers: how much more effort would it be to upload current numpy to both numpy and numpylts?

I'm not quite sure I see the point. you would ask downstream to change `numpy` to `numpylts` instead of `numpy` to `numpy<2` ?

Also I think then you have the risk of having for example pandas saying `numpy<2` and scipy saying `numpylts` and now the pasckages are incompatible ?

The trouble is PyPi doesn't allow multiple branches. So if you upload NumPy 2.0 wheels, then you cannot turn around and upload 1.18.X bug-fix patches. At least, this is my understanding of PyPi.

-- Robert McLeod, Ph.D. robbmcleod@gmail.com robbmcleod@protonmail.com www.entropyreduction.al

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

If a wheel is not available for the client platform, pip will try to install the latest version of the source distribution (.tar.gz or .zip) which I think is the cause of the problem here. -- Olivier ​

On Mon, Nov 13, 2017, at 10:26, Matthias Bussonnier wrote:

Unless the sdist is tagged with require_python and users have recent-enough pip.

Is this documented anywhere? I couldn't find it via Google, and suspect it may be widely useful in the next few months. Stéfan

I am in very supportive of this plan. For Matplotlib the intention is to do a mpl2.2LTS early 2018 and a mpl3.0 (no major API breaks other than dropping py2 support) summer 2018 with the same meaning of LTS. I also had thought about bumping the minimum numpy version of Matplotlib to the first py3 only version when it is out. There is no technical reason, but it seems nicely symmetric. In general we all need to get better about dropping support for old versions of dependencies (I am throwing stones from inside my glass house). The prolonged support of py2 has warped our idea of how long old versions of things need to be supported and it imposes real costs up and down the stack. Tom On Mon, Nov 13, 2017 at 1:26 PM Matthias Bussonnier < bussonniermatthias@gmail.com> wrote:

...

If a wheel is not available for the client platform, pip will try to install the latest version of the source distribution (.tar.gz or .zip) which I think is the cause of the problem here.

Unless the sdist is tagged with require_python and users have recent-enough pip. Which is what was referred to earlier as "Automagically". This behavior is "new" (Nov/Dec 2016). The upstream patches were written (in part) by the IPython/Jupyter team, for this exact purpose, to not install an incompatible sdists. (Works great I can share download graphs for IPython[0])

It _does_ require to have a version of pip which is not decades old though, and may not work if you use a pypi proxy which is not pep 503 compliant (which happens, we got bug report, users then complained to IT who fixed it). -- M

On Mon, Nov 13, 2017 at 10:14 AM, Olivier Grisel wrote:

...

If a wheel is not available for the client platform, pip will try to install the latest version of the source distribution (.tar.gz or .zip) which I think is the cause of the problem here.

-- Olivier

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

On Mon, Nov 13, 2017 at 10:26:31AM -0800, Matthias Bussonnier wrote:

This behavior is "new" (Nov/Dec 2016). [snip] It _does_ require to have a version of pip which is not decades old

Just to check that I am not misunderstanding: the version of pip should not be more than a year old; "decades old" is just French hyperbola? Do I understand right? Gaël

Just to check that I am not misunderstanding: the version of pip should not be more than a year old; "decades old" is just French hyperbola? Do I understand right?

Yes, sorry if you can't hear my french accent in writing, I can hear yours :-) There is also a "softer" requirement on setuptools which needs to be recent enough to 1) understand requires_python on machine that will _create_ the sdist/wheel. or 2) accept requires_python as a kwarg (even if does nothing), for linux system that will install from sdist. But by end of 2018 that will be a 3 or 4 years old setuptools.

Right, the requirement is pip 9, which is currently one year old and will be >2 years old by the time this matters for numpy. It does turn out that there's a bimodal distribution in the wild, where people tend to either use an up to date pip, or else use some truly ancient pip that some Linux LTS distro shipped 5 years ago. Numpy isn't the only project that > will be forcing people to upgrade, though, so I think this will work itself out. Especially since in the broken case what happens is that users end up running our setup.py on an unsupported version of python, so we'll be able to detect that and print some loud and informative message.

Correct, we did that for IPython, got a large spike of sdist-download from Py2+old_pip when we released a Py3 only, the spike disappeared after a few days. We still had a handful of bug reports from people thinking the "You must upgrade pip" message was not relevant, and we realised people pinned ipython with IPython==5.0.0 instead of IPython<6. So the "Loud informative message" should also like tell user how to pin numpy if they can't upgrade pip. -- Matthias On Mon, Nov 13, 2017 at 1:33 PM, Nathaniel Smith wrote:

On Nov 13, 2017 12:03, "Gael Varoquaux" wrote:

On Mon, Nov 13, 2017 at 10:26:31AM -0800, Matthias Bussonnier wrote:

...

This behavior is "new" (Nov/Dec 2016). [snip] It _does_ require to have a version of pip which is not decades old

Just to check that I am not misunderstanding: the version of pip should not be more than a year old; "decades old" is just French hyperbola? Do I understand right?

Right, the requirement is pip 9, which is currently one year old and will be

...

2 years old by the time this matters for numpy.

It does turn out that there's a bimodal distribution in the wild, where people tend to either use an up to date pip, or else use some truly ancient pip that some Linux LTS distro shipped 5 years ago. Numpy isn't the only project that will be forcing people to upgrade, though, so I think this will work itself out. Especially since in the broken case what happens is that users end up running our setup.py on an unsupported version of python, so we'll be able to detect that and print some loud and informative message.

-n

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

On Tue, Nov 7, 2017 at 7:24 AM, Chris Barker wrote:

On Sun, Nov 5, 2017 at 10:25 AM, Charles R Harris < charlesr.harris@gmail.com> wrote:

...

the timeline I've been playing with is to keep Python 2.7 support through 2018, which given our current pace, would be for NumPy 1.15 and 1.16. After that 1.16 would become a long term support release with backports of critical bug fixes

+1

I think py2.7 is going to be around for a long time yet -- which means we really do want to keep the long term support -- which may be quite some time. But that's doesn't mean people insisting on no upgrading PYthon need to get the latest and greatest numpy.

Also -- if py2.7 continues to see the use I expect it will well past when pyton.org officially drops it, I wouldn't be surprised if a Python2.7 Windows build based on a newer compiler would come along -- perhaps by Anaconda or conda-forge, or ???

I suspect that this will indeed happen. I am aware of multiple companies following this path already (building python + numpy themselves with a newer MS compiler). David

On Mon, Nov 6, 2017 at 6:14 PM, Charles R Harris wrote:

Also -- if py2.7 continues to see the use I expect it will well past when

...

...

pyton.org officially drops it, I wouldn't be surprised if a Python2.7 Windows build based on a newer compiler would come along -- perhaps by Anaconda or conda-forge, or ???

I suspect that this will indeed happen. I am aware of multiple companies following this path already (building python + numpy themselves with a newer MS compiler).

I think Anaconda is talking about distributing a compiler, but what that will be on windows is anyone's guess. When we drop 2.7, there is a lot of compatibility crud that it would be nice to get rid of, and if we do that then NumPy will no longer compile against 2.7. I suspect some companies have just been putting off the task of upgrading to Python 3, which should be pretty straight forward these days apart from system code that needs to do a lot of work with bytes.

I agree, and if there is a compelling reason to upgrade, folks WILL do it. But I've been amazed over the years at folks' desire to stick with what they have! And I'm guilty too, anything new I start with py3, but older larger codebases are still py2, I just can't find the energy to spend a the week or so it would probably take to update everything... But in the original post, the Windows Compiler issue was mentioned, so there seems to be two reasons to drop py2: A) wanting to use py3 only features. B) wanting to use never C (C++?) compiler features. I suggest we be clear about which of these is driving the decisions, and explicit about the goals. That is, if (A) is critical, we don't even have to talk about (B) But we could choose to do (B) without doing (A) -- I suspect there will be a user base for that.... -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov

On Tue, Nov 7, 2017 at 11:40 PM, Nathaniel Smith wrote:

<snip>

Right now, the decision in front of us is what to tell people who ask about numpy's py2 support plans, so that they can make their own plans. Given what we know right now, I don't think we should promise to keep support past 2018. If we get there and the situation's changed, and there's both desire and means to extend support we can revisit that. But's better to under-promise and possibly over-deliver, instead of promising to support py2 until after it becomes a millstone around our necks and then realizing we haven't warned anyone and are stuck supporting it another year beyond that...

-n

NumPy (and to a lesser extent SciPy) is in a tough position being at the bottom of many scientific Python programming stacks. Whenever you drop Python 2 support is going to upset someone. It is too ambitious to pledge to drop support for Python 2.7 no later than 2020, coinciding with the Python development team’s timeline for dropping support for Python 2.7? If that looks doable, NumPy could sign up to http://www.python3statement.org/ Regards, Peter

On Wed, 2017-11-08 at 18:15 +0100, Ilhan Polat wrote:

I was about to send the same thing. I think this matter became a vim/emacs issue and Py2 supporters won't take any arguments anymore. But if Instagram can do it, it means that legacy code argument is a matter of will but not a technicality. https://thenewstack.io/instagr am-makes-smooth-move-python-3/

Also people are really going out of their ways such as Tauthon https: //github.com/naftaliharris/tauthon to stay with Python2. To be honest, I'm convinced that this is a sentimental debate after seeing this fork.

In my opinion it is fine for us to drop support for python 2 in master relatively soon (as proposed here). But I guess we will need to a "LTS" release which means some extra maintenance burden until 2020. I could hope those who really need it jumping in to carry some of that (and by 2020 my guess is if anyone still wants to support it longer, we won't stop you, but I doubt the current core devs, at least not me, would be very interested in it). So in my opinion, the current NumPy is excellent and very stable, anyone who needs fancy new stuff likely also wants other fancy new stuff so will soon have to use python 3 anyway.... Which means, if we think the extra burden of a "LTS" is lower then the current hassle, lets do it :). Also downstream seems only half a reason to me, since downstream normally supports much outdated versions anyway? - Sebastian

On Wed, Nov 8, 2017 at 5:50 PM, Peter Cock

...

wrote: On Tue, Nov 7, 2017 at 11:40 PM, Nathaniel Smith wrote:

...

<snip>

Right now, the decision in front of us is what to tell people who

ask about

...

numpy's py2 support plans, so that they can make their own plans. Given what we know right now, I don't think we should promise to keep support past 2018. If we get there and the situation's changed, and there's both desire and means to extend support we can revisit that. But's better to under-promise and possibly over-deliver, instead of promising to support py2 until after it becomes a millstone around our necks and then realizing we haven't warned anyone and are stuck supporting it another year beyond that...

-n

NumPy (and to a lesser extent SciPy) is in a tough position being at the bottom of many scientific Python programming stacks. Whenever you drop Python 2 support is going to upset someone.

It is too ambitious to pledge to drop support for Python 2.7 no later than 2020, coinciding with the Python development team’s timeline for dropping support for Python 2.7?

If that looks doable, NumPy could sign up to http://www.python3stat ement.org/

Regards,

Peter _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

_______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion

Anaconda's compilers are for Linux (gcc 7.2) and Mac (llvm/clang 4.0.1) right now. We would like to have clang target all platforms, but that's a lot of development effort. We are also exploring ways of keeping package ecosystems in line, so that building and managing a self-consistent set of python 2.7 packages with a new Visual Studio version or msys2 might be easier. No timeline to report, on that, though. Breaking with the python.org ABI is pretty painful. On Wed, Nov 8, 2017 at 4:17 PM, Bryan Van de ven wrote:

...

On Nov 8, 2017, at 10:50, Peter Cock wrote:

NumPy (and to a lesser extent SciPy) is in a tough position being at the bottom of many scientific Python programming stacks. Whenever you drop Python 2 support is going to upset someone.

Existing versions of NumPy will still exist and continue to work with Python 2.7. If users want to say with Python 2.7, that's fine, they will just have to rely on those older/LTS versions. I personally would be happy for projects at the bottom of stacks to take an activist stance and make decisions to actively encourage movement to Python 3.

...

It is too ambitious to pledge to drop support for Python 2.7 no later than 2020, coinciding with the Python development team’s timeline for dropping support for Python 2.7?

Developing NumPy is hard, as it is. Everything that can be done to simplify things for the current maintainers and help attract new contributors should be done. It is not reasonable to ask a few (largely volunteer) people to shoulder the burden and difficulties of supporting Python 2.7 for several additional *years* of their life.

I agree entirely with Nick Coghlan's comments from another discussion, and think they apply equally well in this instance:

""" While it's entirely admirable that many upstream developers are generous enough to help their end users work around this inertia, in the long run doing so is detrimental for everyone concerned, as long term sustaining engineering for old releases is genuinely demotivating for upstream developers (it's a good job, but a lousy way to spend your free time) and for end users, working around institutional inertia this way reduces the pressure to actually get the situation addressed properly. """

Thanks,

Bryan _______________________________________________ NumPy-Discussion mailing list NumPy-Discussion@python.org https://mail.python.org/mailman/listinfo/numpy-discussion