Revisit of PEP 394 -- The "python" Command on Unix-Like Systems
Hello everyone, tl;dr Let's make sure that PEP 394 says "/usr/bin/python should be Python 3 by early 2020" As Python 2's end of life is approaching, various Linux distributions are taking measures to tackle the issue, raising awareness, porting packages and tools to Python 3, having only Python 3 preinstalled on their base images, and some have even made the ultimate change: making /usr/bin/python point to the Python 3 interpreter. Some references [0][1][2][3][4]. Various upstream Python projects have also started down to the path to completely dropping Python 2 support in order to start taking full advantage of Python 3 only features: http://www.python3statement.org/ In Fedora, we'd like to change /usr/bin/python to point to Python 3 by 2020 (when PEP 404 kicks in). However, we'd rather not do it without upstream's support in PEP 394. Hence we would like to drive the change of PEP 394 to either say that /usr/bin/python should point to Python 3, or to say it is up to the distribution maintainers. (And we would prefer the first option, so that the situation may eventually become consistent.) The message we want to send is: "everything that can be ported to Python 3 should be ported, and everything that can run on Python 3 should run on it." However, having /usr/bin/python point to Python 2 is continuing the concrete mindset for Linux users that "the default" version of Python is still Python 2. Those mixed signals are especially problematic with maintainers that aren't familiar with the Python ecosystem (e.g. they only use a Python-based buildsystem). PEP 394 says that it "is anticipated that there will eventually come a time where the third party ecosystem surrounding Python 3 is sufficiently mature for this recommendation to be updated to suggest that the python symlink refer to python3 rather than python2," and that its recommendation will be periodically reviewed. We believe the ecosystem is now mature enough, and so we should set a specific target timeline for distributions to make that switch (aligning it with the end of community support for the Python 2.7 branch). [0] https://fedoraproject.org/wiki/FinalizingFedoraSwitchtoPython3 [1] https://lists.opensuse.org/opensuse-factory/2017-06/msg00740.html [2] https://wiki.ubuntu.com/Python/3 [3] https://wiki.openmandriva.org/en/3.02/Release_Notes#Python Regards, Charalampos Stratakis Software Engineer Python Maintenance Team, Red Hat
On 25 July 2017 at 22:02, Charalampos Stratakis
In Fedora, we'd like to change /usr/bin/python to point to Python 3 by 2020 (when PEP 404 kicks in). However, we'd rather not do it without upstream's support in PEP 394. Hence we would like to drive the change of PEP 394 to either say that /usr/bin/python should point to Python 3, or to say it is up to the distribution maintainers. (And we would prefer the first option, so that the situation may eventually become consistent.)
Some additional context to this idea for non-Fedora-using folks: while Python 2 and hence "/usr/bin/python" have been missing from our default installations since the release of Fedora 23 in 2015, it's still pretty easy to inadvertently get them back: - an unqualified "dnf install python-<whatever>" will usually do it - any package with "/usr/bin/python" in a shebang line implicitly gains a dependency on "python2" At the distribution level, our plan is to prevent the latter by way of Fedora's packaging policy and associated automated tests (i.e. all Python scripts in packages built for Fedora itself will be required to qualify the required Python version), while the former will be handled through RPM macros and virtual provides (essentially, there'll be a cutover release, probably Fedora 32 in early 2020, where "dnf install python-foo" starts to give you the Python 3 version of things instead of the Python 2 version). At the PEP 394 level, what I'd personally like to see us do is to split the "Recommendations" section into two separate sections: "Python User Recommendations" and "Platform Publisher Recommendations" "Python User Recommendations" would cover three main points: 1. Recommend the use of virtual environments and Python level packaging tools to decouple Python development activities from distro choices of default Python runtimes 2. Recommend writing hybrid Python 2/3 code when specifically targeting "/usr/bin/python" (including favouring either 'print(fmt.(*args)) and 'print(fmt % args)' or 'from __future__ import print_function' over multi-argument print statements) 3. Recommend targeting "/usr/bin/python2" instead of "/usr/bin/python" when publishing specifically Python 2 code "Platform Publisher Recommendations" would be updated to say: 1. "/usr/bin/python" should point to either Python 2.7 or Python 3.5+ (to ensure that bytes objects support mod-formatting) 2. If it's updated to point to Python 3.x, then system administrators switching it back to Python 2.7 *should* be a supported operation (i.e. it shouldn't break the OS itself if they do that) 3. When a Python 2.7 stack is provided, make sure to also provide "virtualenv" or some other mechanism for user level environment management 4. Regardless of whether "/usr/bin/python" points to Python 2 or Python 3, that default stack should provide the "six" and "future" cross-compatibility libraries to make life easier for folks writing unpackaged hybrid 2/3 code that can't readily indicate any third party library dependencies As far as the specifics of actually managing a cutover in "/usr/bin/python" itself go, we think that's mainly a question to be resolved between distro developers and their particular user community. For Fedora for example, we think that dropping "/usr/bin/python" from the repositories entirely would be too large of a compatibility break, whereas we're comfortable with the idea that by 2020, it will be reasonable to require that anything still running in "/usr/bin/python" be hybrid Python 2/3 code, such that finally changing the default won't cause too much immediate breakage (and for things that do still break, the recommended fix will be code modernisation to get into the hybrid subset prior to upgrading the OS). There's unfortunately no completely non-disruptive way to manage this shift (hence why it's taking so long), but we think this is a reasonable approach that allows each distro to devise a migration plan that makes sense for them and their userbase while still allow Python end users to write readable cross-distro code that doesn't particular care whether it's run under Python 2 or Python 3, and for open source Python project maintainers to provide developer guidelines that are entirely independent of particular distro's choices about default Python runtimes. Before taking this proposal to python-dev, I'd turn the general concept into an actual PR with specific proposed wording changes, but I figured it made sense for us to seek some initial feedback here before doing that. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Thanks for getting this discussion going again. The more I talk to people about the upcoming EOL for Python 2.7, the more I realize that while 2.5 years *seems* like a long way away, it’ll be here sooner that many are planning for. “Winter is coming” is a phrase that pops into my mind (although maybe it should be Summer, as I think Python 3 invokes a lot more optimism than the White Walkers. :) A couple of top-line points first: * As I mentioned to Nick off-list, I think we should amend PEP 394 rather than supersede it. It’s widely cited as *the* upstream policy on the matter, so I think we’ll get more bang for the buck, and less overall confusion, by updating it rather than replacing it. * My personal preferences have been evolving on this subject too, so I won’t claim to represent the Debian ecosystem here. That’s a discussion to be had over in that forum. For now, take all of the following as my personal opinion.
On Jul 25, 2017, at 08:02, Charalampos Stratakis
wrote: tl;dr Let's make sure that PEP 394 says "/usr/bin/python should be Python 3 by early 2020”
I think that’s just a bit too early. Consider the timeline. Guido has proclaimed[*] that Python 2.7 will EOL at Pycon 2020, which will likely be in the May-June timeframe. I think any changes we make should either be timed to coincide with that event, or happen in the latter part of 2020. It’s also very likely that Python 2.7 support won’t abruptly end on that date, but instead will continue with some level of security-only, source-only releases for a while after. For the moment though, let’s not quibble on the timing. [*] At the Language Summit 2017; sorry no reference handy atm.
Various upstream Python projects have also started down to the path to completely dropping Python 2 support in order to start taking full advantage of Python 3 only features: http://www.python3statement.org/
Oh nice! I wasn’t aware of this site, and will likely sign up to it soon. I’ve already mostly dropped Python 2 support for most of my personal projects, and of course GNU Mailman 3 Core has been Python 3-only for a while. (Some of the other components are not quite there yet, but we’re committed to it. Mailman 2 will likely never get ported. I suspect that fate will be shared by many projects, alas.)
In Fedora, we'd like to change /usr/bin/python to point to Python 3 by 2020 (when PEP 404 kicks in). However, we'd rather not do it without upstream's support in PEP 394. Hence we would like to drive the change of PEP 394 to either say that /usr/bin/python should point to Python 3, or to say it is up to the distribution maintainers. (And we would prefer the first option, so that the situation may eventually become consistent.)
Consistency across the Python ecosystem should be a top concern. The Python 3 transition has been a tumultuous one, and we’ve had successes and challenges. Overall, I think we’re in a good place right now, so let’s make sure we handle the end of the transition as well as possible! People may argue about whether such-and-such a decision is right or wrong, but uncertainty is much worse.
The message we want to send is: "everything that can be ported to Python 3 should be ported, and everything that can run on Python 3 should run on it.”
+1
However, having /usr/bin/python point to Python 2 is continuing the concrete mindset for Linux users that "the default" version of Python is still Python 2. Those mixed signals are especially problematic with maintainers that aren't familiar with the Python ecosystem (e.g. they only use a Python-based buildsystem).
IMHO, the reason why this is so is because `python` (at the prompt, a.k.a. typically /usr/bin/python) is the CLI API for *users*. It’s what they’re taught to type at the command line to run their scripts or to start the interpreter to play with things. PEP 394 already recommends `python2` and `python3` in the shebang line for distro-provided executables. I think we turn some SHOULDs into MUSTs so that any PEP-394 compliant *nix distro will be required to follow the same rules. Further, I’d add that PEP 394-compliant distros MUST NOT put /usr/bin/python in shebangs. (PEP 394 does not talk about `/usr/bin/env python` which is often used by developers, although highly discouraged for distro-provided scripts, at least in Debian. If we want to add a note about this, I think we make it clear that this usage relies solely on the user’s own $PATH environment, and therefore lies outside the scope of the PEP.) I’d like to discourage the use of naming schemes like `idle2`, `pydoc2`, etc. That’s a problematic approach, for which -m invocation is much more predictable, albeit less convenient. We still don’t have a good solution for that, although many have been discussed. That’s probably outside the scope of PEP 394 too. Lastly, we come to what /usr/bin/python points to, and that’s always been the sticking point. There are two use cases. * When people type that at the command line, which version of Python do they get? * When people use that in their own third-party (i.e. not distro-provided) scripts, what version of Python do they get? So, it’s a UI/UX problem now :) I would personally like to say /usr/bin/python will always point to python3 after such-and-such a date, although as I’ve said above, my timing would put that to later in 2020. However, I think we need to allow distributions to make their own decisions on that timing. Some may choose to never make it, others may be proactive. There are two ways to go here. Either we can make it a MUST to point /usr/bin/python to python3, and just label any distro that decides otherwise as “non-PEP-394-compliant”, or we make the choice more lax (possibly SHOULD, possibly MAY) and allow leeway for otherwise in-compliance distros. I suspect there will be some in Debian that will argue for the latter, and they will make good points. OTOH, I think we’re well past the tipping point, and will face the cliff pretty darn soon. I haven’t completely made my own mind up on this point. In the meantime, I think we can make a lot of progress on PEP 394 as I’ve outlined above. Cheers, -Barry
On 07/31/2017 04:25 PM, Barry Warsaw wrote: [...]
The message we want to send is: "everything that can be ported to Python 3 should be ported, and everything that can run on Python 3 should run on it.”
+1
However, having /usr/bin/python point to Python 2 is continuing the concrete mindset for Linux users that "the default" version of Python is still Python 2. Those mixed signals are especially problematic with maintainers that aren't familiar with the Python ecosystem (e.g. they only use a Python-based buildsystem).
IMHO, the reason why this is so is because `python` (at the prompt, a.k.a. typically /usr/bin/python) is the CLI API for *users*. It’s what they’re taught to type at the command line to run their scripts or to start the interpreter to play with things.
PEP 394 already recommends `python2` and `python3` in the shebang line for distro-provided executables. I think we turn some SHOULDs into MUSTs so that any PEP-394 compliant *nix distro will be required to follow the same rules. Further, I’d add that PEP 394-compliant distros MUST NOT put /usr/bin/python in shebangs.
+1 Ah, yes, this is missing from the "PEP 548" pull request.
(PEP 394 does not talk about `/usr/bin/env python` which is often used by developers, although highly discouraged for distro-provided scripts, at least in Debian. If we want to add a note about this, I think we make it clear that this usage relies solely on the user’s own $PATH environment, and therefore lies outside the scope of the PEP.)
+1
I’d like to discourage the use of naming schemes like `idle2`, `pydoc2`, etc. That’s a problematic approach, for which -m invocation is much more predictable, albeit less convenient. We still don’t have a good solution for that, although many have been discussed. That’s probably outside the scope of PEP 394 too.
Lastly, we come to what /usr/bin/python points to, and that’s always been the sticking point. There are two use cases.
* When people type that at the command line, which version of Python do they get? * When people use that in their own third-party (i.e. not distro-provided) scripts, what version of Python do they get?
So, it’s a UI/UX problem now :)
I would personally like to say /usr/bin/python will always point to
I think that ship has sailed. The unversioned names should switch along with /usr/bin/python, and the versioned ones should be allowed to die along with /usr/bin/python[23]. Improving the -m situation is out of scope. python3 after such-and-such a date, although as I’ve said above, my timing would put that to later in 2020. However, I think we need to allow distributions to make their own decisions on that timing. Some may choose to never make it, others may be proactive.
There are two ways to go here. Either we can make it a MUST to point /usr/bin/python to python3, and just label any distro that decides otherwise as “non-PEP-394-compliant”, or we make the choice more lax (possibly SHOULD, possibly MAY) and allow leeway for otherwise in-compliance distros. I suspect there will be some in Debian that will argue for the latter, and they will make good points. OTOH, I think we’re well past the tipping point, and will face the cliff pretty darn soon. I haven’t completely made my own mind up on this point. In the meantime, I think we can make a lot of progress on PEP 394 as I’ve outlined above.
The other recommendations proposed here are designed so they can be implemented right now, with no bad side effects, but switching /usr/bin/python will break things. I think here it's good to leave the timing up to distros, and to suggest that the distros allow users to override their decisions.
On Jul 31, 2017, at 12:34, Petr Viktorin
I’d like to discourage the use of naming schemes like `idle2`, `pydoc2`, etc. That’s a problematic approach, for which -m invocation is much more predictable, albeit less convenient. We still don’t have a good solution for that, although many have been discussed. That’s probably outside the scope of PEP 394 too.
I think that ship has sailed. The unversioned names should switch along with /usr/bin/python, and the versioned ones should be allowed to die along with /usr/bin/python[23]. Improving the -m situation is out of scope.
Definitely, improving -m is out of scope, but I’m not so sure about the location of that ship. Maybe that should be a distro-specific decision as well. E.g. some of Debian’s unversioned tools are already shebanged for Python 3, but that also causes problems sometimes. OTOH, it should only be a relatively small number of scripts that need to make the distinction in their /usr/bin name.
The other recommendations proposed here are designed so they can be implemented right now, with no bad side effects, but switching /usr/bin/python will break things. I think here it's good to leave the timing up to distros, and to suggest that the distros allow users to override their decisions.
Again, I’d like the recommendations to use MUST, SHOULD, MAY language so readers and implementers of the PEP will be clear in what’s expected of them and their distros. Cheers, -Barry
participants (4)
-
Barry Warsaw
-
Charalampos Stratakis
-
Nick Coghlan
-
Petr Viktorin