Another update for PEP 394 -- The "python" Command on Unix-Like Systems
![](https://secure.gravatar.com/avatar/870d613430249e453343efc9667ef636.jpg?s=120&d=mm&r=g)
PEP 394 says:
This recommendation will be periodically reviewed over the next few years, and updated when the core development team judges it appropriate. As a point of reference, regular maintenance releases for the Python 2.7 series will continue until at least 2020.
I think it's time for another review. I'm especially worried about the implication of these: - If the `python` command is installed, it should invoke the same version of Python as the `python2` command - scripts that are deliberately written to be source compatible with both Python 2.x and 3.x [...] may continue to use `python` on their shebang line. So, to support scripts that adhere to the recommendation, Python 2 needs to be installed :( Please see this PR for details and a suggested change: https://github.com/python/peps/pull/893
![](https://secure.gravatar.com/avatar/db5f70d2f2520ef725839f046bdc32fb.jpg?s=120&d=mm&r=g)
On Wed, 13 Feb 2019 16:24:48 +0100
Petr Viktorin
PEP 394 says:
This recommendation will be periodically reviewed over the next few years, and updated when the core development team judges it appropriate. As a point of reference, regular maintenance releases for the Python 2.7 series will continue until at least 2020.
I think it's time for another review. I'm especially worried about the implication of these:
- If the `python` command is installed, it should invoke the same version of Python as the `python2` command - scripts that are deliberately written to be source compatible with both Python 2.x and 3.x [...] may continue to use `python` on their shebang line.
So, to support scripts that adhere to the recommendation, Python 2 needs to be installed :(
I think PEP 394 should acknowledge that there are now years of established usage of `python` as Python 3 for many conda users. Regards Antoine.
![](https://secure.gravatar.com/avatar/870d613430249e453343efc9667ef636.jpg?s=120&d=mm&r=g)
On 2/13/19 4:46 PM, Antoine Pitrou wrote:
On Wed, 13 Feb 2019 16:24:48 +0100 Petr Viktorin
wrote: PEP 394 says:
This recommendation will be periodically reviewed over the next few years, and updated when the core development team judges it appropriate. As a point of reference, regular maintenance releases for the Python 2.7 series will continue until at least 2020.
I think it's time for another review. I'm especially worried about the implication of these:
- If the `python` command is installed, it should invoke the same version of Python as the `python2` command - scripts that are deliberately written to be source compatible with both Python 2.x and 3.x [...] may continue to use `python` on their shebang line.
So, to support scripts that adhere to the recommendation, Python 2 needs to be installed :(
I think PEP 394 should acknowledge that there are now years of established usage of `python` as Python 3 for many conda users.
The intention is that Conda environments are treated the same as venv environments, i.e.: When a virtual environment (created by the PEP 405 venv package or a similar tool) is active, the python command should refer to the virtual environment's interpreter. In other words, activating a virtual environment counts as deliberate user action to change the default python interpreter. Do you think conda should be listed explicitly along with venv?
![](https://secure.gravatar.com/avatar/4dc045274504f02e7a0b6264e96da643.jpg?s=120&d=mm&r=g)
Hi,
I'm a (strong) supporter of providing a "python" command which would
be the latest Python version!
As php does nowadays (after previous issues with "php4" vs "php5".) I
don't recall that perl had "perl4" vs "perl5", the command was always
"perl", no? Same for Ruby: it was still "ruby" after for Ruby 2, no?
Only Python and PHP used different program names depending on the
language version, no? And PHP now moved back to a single "php"
program.
In the container and virtualenv era, it's now easy to get your
favorite Python version for the "python" command.
On my Windows VM, "python" is Python 3.7 :-) In virtual environments,
"python" can also be Python 3 as well.
I recall that I saw commands using "python" rather than "python3" in
the *official* Python 3 documentation: see examples below (*).
Problem: On Windows, "python" is the right command. "python3" doesn't
work (doesn't exist) on Windows. Should we write the doc for Windows
or for Unix? Oooops.
There was an interesting discussion about the Python version following
Python 3.9: Python 3.10 or Python 4? And what are the issues which
would make us prefer 3.10 rather than 4.0?
https://mail.python.org/pipermail/python-committers/2018-September/006152.ht...
One practical issue is that right now, six.PY3 is defined by "PY3 =
sys.version_info[0] == 3" and so "if six.PY3:" will be false on Python
4.
Another interesting thing to mention is the Unix Python launcher
("py") written by Brett Cannon written in Rust:
https://github.com/brettcannon/python-launcher
(*) A few examples of "python" commands in the Python official documentation
"$ python prog.py -h"
https://docs.python.org/dev/library/argparse.html
"$ python logctx.py"
https://docs.python.org/dev/howto/logging-cookbook.html
"python setup.py install"
https://docs.python.org/dev/install/index.html
"python --help"
https://docs.python.org/dev/howto/argparse.html
"python setup.py build"
https://docs.python.org/dev/extending/building.html
"exec python $0 ${1+"$@"}"
https://docs.python.org/dev/faq/library.html
"python setup.py --help build_ext"
https://docs.python.org/dev/distutils/configfile.html
Victor
Le mer. 13 févr. 2019 à 16:49, Antoine Pitrou
On Wed, 13 Feb 2019 16:24:48 +0100 Petr Viktorin
wrote: PEP 394 says:
This recommendation will be periodically reviewed over the next few years, and updated when the core development team judges it appropriate. As a point of reference, regular maintenance releases for the Python 2.7 series will continue until at least 2020.
I think it's time for another review. I'm especially worried about the implication of these:
- If the `python` command is installed, it should invoke the same version of Python as the `python2` command - scripts that are deliberately written to be source compatible with both Python 2.x and 3.x [...] may continue to use `python` on their shebang line.
So, to support scripts that adhere to the recommendation, Python 2 needs to be installed :(
I think PEP 394 should acknowledge that there are now years of established usage of `python` as Python 3 for many conda users.
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
-- Night gathers, and now my watch begins. It shall not end until my death.
![](https://secure.gravatar.com/avatar/db5f70d2f2520ef725839f046bdc32fb.jpg?s=120&d=mm&r=g)
On Wed, 13 Feb 2019 17:18:15 +0100
Petr Viktorin
On 2/13/19 4:46 PM, Antoine Pitrou wrote:
On Wed, 13 Feb 2019 16:24:48 +0100 Petr Viktorin
wrote: PEP 394 says:
This recommendation will be periodically reviewed over the next few years, and updated when the core development team judges it appropriate. As a point of reference, regular maintenance releases for the Python 2.7 series will continue until at least 2020.
I think it's time for another review. I'm especially worried about the implication of these:
- If the `python` command is installed, it should invoke the same version of Python as the `python2` command - scripts that are deliberately written to be source compatible with both Python 2.x and 3.x [...] may continue to use `python` on their shebang line.
So, to support scripts that adhere to the recommendation, Python 2 needs to be installed :(
I think PEP 394 should acknowledge that there are now years of established usage of `python` as Python 3 for many conda users.
The intention is that Conda environments are treated the same as venv environments, i.e.:
When a virtual environment (created by the PEP 405 venv package or a similar tool) is active, the python command should refer to the virtual environment's interpreter. In other words, activating a virtual environment counts as deliberate user action to change the default python interpreter.
Anaconda is often used as providing, not only virtual environments, but the "main" user Python. At least it certainly is so on Windows and macOS, but I'm sure it is used like that as well on Linux, especially on ancient distributions such as RHEL 6 or Ubuntu 14.04. In any case, the fact that many people are used to "python" pointing to Python 3 is IMHO an important data point. Regards Antoine.
![](https://secure.gravatar.com/avatar/4dc045274504f02e7a0b6264e96da643.jpg?s=120&d=mm&r=g)
Some more context about Petr's change, Fedora, RHEL and Red Hat.
At the latest Language Summit (2018), Petr detailed the state of the
migration to Python 3 and how Python 2 is and will be handled at Red
Hat; "Linux distributions and Python 2" talk with Matthias Klose (who
works on Debian/Ubuntu):
https://lwn.net/Articles/756628/
Petr explained in a comment on this change that /usr/bin/python is
configurable in the incoming RHEL8:
https://github.com/python/peps/pull/893#issuecomment-463240453
"I'm responsible for how this is handled in RHEL 8 beta, where
`/usr/bin/python` is configurable (though configuring it is
discouraged). I don't recommend that in the PEP – I don't think it
needs to cover distros that need to lock in the behavior of
`/usr/bin/python` for a decade."
More details in his nice "Python in RHEL 8" article:
https://developers.redhat.com/blog/2018/11/14/python-in-rhel-8/
RHEL8 has specific challenges since it will be released around Python2
end-of-life with customers who are still running Python 2, but has
also to be prepared for the bright Python 3 world since RHEL is
usually supported for 10 years (if not longer).
Petr and me are working for Red Hat on Fedora and RHEL. My team is
helping on actively removing Python 2 from Fedora:
https://fedoraproject.org/wiki/Changes/Mass_Python_2_Package_Removal
"Python 2 will be deprecated in Fedora. Packagers can mark any other
Python 2 packages as deprecated as well."
See also:
* https://fedoraproject.org/wiki/FinalizingFedoraSwitchtoPython3 --
work-in-progress
* https://fedoraproject.org/wiki/Changes/Python_3_as_Default --
implemented in Fedora 23
The base installation of Fedora only provides "python3" (no "python"
nor "python2") since Fedora 23 (released in 2015), as does Ubuntu
nowadays. You get get "python" on Fedora by installing a special
"python-unversioned-command" package :-)
Victor
Le mer. 13 févr. 2019 à 16:28, Petr Viktorin
PEP 394 says:
This recommendation will be periodically reviewed over the next few years, and updated when the core development team judges it appropriate. As a point of reference, regular maintenance releases for the Python 2.7 series will continue until at least 2020.
I think it's time for another review. I'm especially worried about the implication of these:
- If the `python` command is installed, it should invoke the same version of Python as the `python2` command - scripts that are deliberately written to be source compatible with both Python 2.x and 3.x [...] may continue to use `python` on their shebang line.
So, to support scripts that adhere to the recommendation, Python 2 needs to be installed :(
Please see this PR for details and a suggested change: https://github.com/python/peps/pull/893 _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
-- Night gathers, and now my watch begins. It shall not end until my death.
![](https://secure.gravatar.com/avatar/be200d614c47b5a4dbb6be867080e835.jpg?s=120&d=mm&r=g)
On 13Feb2019 0820, Victor Stinner wrote:
On my Windows VM, "python" is Python 3.7 :-) In virtual environments, "python" can also be Python 3 as well.
I recall that I saw commands using "python" rather than "python3" in the *official* Python 3 documentation: see examples below (*). Problem: On Windows, "python" is the right command. "python3" doesn't work (doesn't exist) on Windows. Should we write the doc for Windows or for Unix? Oooops.
With the Windows Store package of Python, you get "python", "python3", and "python3.x" links added to your PATH, and I'm still thinking about ways to make this reasonable/reliable through the full installer as well (the difference is that the OS manages the links through the Store package, whereas each individual installer has to do it on their own otherwise). I'm inclined to view "python" as the default, official command, with the versioned ones being workarounds added by distributors. So: * our docs should say "python" consistently * we should recommend that distributors use the same workaround * our docs should describe the recommended workaround in any places people are likely to first encounter it (tutorial, sys.executable, etc.) (And maybe this isn't currently how things are done, but I'd rather hold up an ideal than pretend that the status quo can't be changed - this list is literally for discussing changing the status quo of anything in core Python ;) ) Cheers, Steve
![](https://secure.gravatar.com/avatar/5dde29b54a3f1b76b2541d0a4a9b232c.jpg?s=120&d=mm&r=g)
On Feb 13, 2019, at 9:13 AM, Steve Dower
I'm inclined to view "python" as the default, official command, with the versioned ones being workarounds added by distributors.
+1 — almost. I agree that “python” be the default, but it would be good to require (or at least highly encourage) that there be a “python3” as well. There will be folks wanting to run python3 on systems where there is still a “python” pointing to py2 — particularly since that is still the “correct” way to do it!
(And maybe this isn't currently how things are done, but I'd rather hold up an ideal than pretend that the status quo can't be changed -
Exactly. -CHB
this list is literally for discussing changing the status quo of anything in core Python ;) )
Cheers, Steve _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
![](https://secure.gravatar.com/avatar/01aa7d6d4db83982a2f6dd363d0ee0f3.jpg?s=120&d=mm&r=g)
On Feb 13, 2019, at 08:20, Victor Stinner
I'm a (strong) supporter of providing a "python" command which would be the latest Python version!
I think we should aspire for this to be the case too, eventually. When this has come up in the past, we’ve said that it’s not appropriate to change PEP 394 until Python 2 is officially deprecated. OTOH, I appreciate that distros and others have to make decisions on this now. I think it’s worth discussing where we eventually want to be as a community, even if we continue to recommend no official change until 2020. I personally would like for `python` to be the latest Python 3 version (or perhaps Brett’s launcher), `python2` to be Python 2.7 where installed (and not mandatory). `python3` would be an alias for the latest Python 3.
There was an interesting discussion about the Python version following Python 3.9: Python 3.10 or Python 4? And what are the issues which would make us prefer 3.10 rather than 4.0? https://mail.python.org/pipermail/python-committers/2018-September/006152.ht...
I don’t think this should be conflated with PEP 394. IMHO, 3.10 is just fine. Python 4 should be reserved for some future mythical GIL-less interpreter or other major API breaking change. It might never happen. -Barry
![](https://secure.gravatar.com/avatar/77945962d72696d9a536f2184b138d53.jpg?s=120&d=mm&r=g)
On 2019-02-13, Barry Warsaw wrote:
I personally would like for `python` to be the latest Python 3 version (or perhaps Brett’s launcher), `python2` to be Python 2.7 where installed (and not mandatory). `python3` would be an alias for the latest Python 3.
To me, having 'py' on Unix would be a good thing(tm). If we have that then I suppose we will encourage people to prefer it over 'python', 'python3', and 'python2'. At that point, where 'python' points would be less of an issue. I'm not opposed to making 'python' configurable or eventually pointing it to python3. However, if we do go with 'py' as the preferred command in the future, it seems to be some pain for little gain. If the OS already allows it to be re-directed, maybe that's good enough. Regards, Neil
![](https://secure.gravatar.com/avatar/d6b9415353e04ffa6de5a8f3aaea0553.jpg?s=120&d=mm&r=g)
On 2/13/2019 3:26 PM, Barry Warsaw wrote:
I personally would like for `python` to be the latest Python 3 version (or perhaps Brett’s launcher), `python2` to be Python 2.7 where installed (and not mandatory). `python3` would be an alias for the latest Python 3.
It appears python is already python3 for a large majority of human users (as opposed to machines). https://www.jetbrains.com/research/python-developers-survey-2018/ Nearly 20000 valid responses, Oct-Nov. "Which version of Python do you use the most" Python 3: 75% in 2017, 84% in 2018. The figure for other public surveys were 22%, 34%, 40% in Dec 2013, Dec 2014, Jan 2016. I expect at least 90% by next January. Py3 is already 90% among data scientists. -- Terry Jan Reedy
![](https://secure.gravatar.com/avatar/5615a372d9866f203a22b2c437527bbb.jpg?s=120&d=mm&r=g)
On Wed, Feb 13, 2019 at 05:16:54PM -0500, Terry Reedy wrote:
It appears python is already python3 for a large majority of human users (as opposed to machines).
https://www.jetbrains.com/research/python-developers-survey-2018/ Nearly 20000 valid responses, Oct-Nov.
They may be valid responses, but we don't know if they are representative of the broader Python community. Its a self-selected survey of people which always makes the results statistically suspect. (By definition, an Internet survey eliminates responses from people who don't fill out surveys on the Internet.) BUt even if representative, this survey only tells us what version people are using, now how they invoke it. We can't conclude that the command "python" means Python 3 for these users. We simply don't know one way or another (and I personally wouldn't want to hazard a guess.) -- Steven
![](https://secure.gravatar.com/avatar/4dc045274504f02e7a0b6264e96da643.jpg?s=120&d=mm&r=g)
Le mer. 13 févr. 2019 à 21:26, Barry Warsaw
I don’t think this should be conflated with PEP 394. IMHO, 3.10 is just fine. Python 4 should be reserved for some future mythical GIL-less interpreter or other major API breaking change. It might never happen.
My point is that changing the major version from 3 to 4 *will* break things. We have to prepare the community to support such change. For example, advice to replace "if major_version == 3: ... else: ..." with "if major_version >= 3: ... else: ...". Victor -- Night gathers, and now my watch begins. It shall not end until my death.
![](https://secure.gravatar.com/avatar/5dde29b54a3f1b76b2541d0a4a9b232c.jpg?s=120&d=mm&r=g)
On Wed, Feb 13, 2019 at 12:29 PM Barry Warsaw
I think we should aspire for this to be the case too, eventually. When this has come up in the past, we’ve said that it’s not appropriate to change PEP 394 until Python 2 is officially deprecated. OTOH, I appreciate that distros and others have to make decisions on this now. I think it’s worth discussing where we eventually want to be as a community, even if we continue to recommend no official change until 2020.
I don't think end-users will see any discontinuity in 2020. For quite a while now, more and more people have shifted from using python2 as the default to using python3 as the default. On the other hand, some folks are still using python2, and will continue to do so after 2020 (old, still supported versions RedHat anyone?) Hopefully, after 2020 no one will start anything new with py2, but it's going to be around for a long, long time. So as there will be no time for a "clear break", we might as well make changes when the time "seems" right. And has been pointed out in this thread, there are a lot of folks not following the PEP anyway (virtual environments, conda, ??). I myself have been telling my newbie students to make a link from "python" to python3 for a couple years (before I saw that PEP!). I personally would like for `python` to be the latest Python 3 version (or
perhaps Brett’s launcher), `python2` to be Python 2.7 where installed (and not mandatory). `python3` would be an alias for the latest Python 3.
+1 Starting now. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
![](https://secure.gravatar.com/avatar/01aa7d6d4db83982a2f6dd363d0ee0f3.jpg?s=120&d=mm&r=g)
On Feb 13, 2019, at 15:07, Victor Stinner
Le mer. 13 févr. 2019 à 21:26, Barry Warsaw
a écrit : I don’t think this should be conflated with PEP 394. IMHO, 3.10 is just fine. Python 4 should be reserved for some future mythical GIL-less interpreter or other major API breaking change. It might never happen.
My point is that changing the major version from 3 to 4 *will* break things. We have to prepare the community to support such change.
Perhaps. I just don’t think Python 4 is anything but distant vaporware. There’s a cost to freaking everyone out that Python 4 is coming and will be as disruptive as Python 3. Calling Python 3.9+1 Python 4 feeds into that FUD for no reason that I can tell except for an aversion to two digit minor version numbers. -Barry
![](https://secure.gravatar.com/avatar/8702771e2f72afdffc4fcb5527e46354.jpg?s=120&d=mm&r=g)
On 2019-02-13, Terry Reedy wrote:
It appears python is already python3 for a large majority of human users (as opposed to machines).
IMHO, the question about where /usr/bin/python points is more important for machines than for humans. Think about changing /bin/sh to some different version of the Borne Shell that changes 'echo'. Or changing 'awk' to some incompatible version. That's going to break a lot of scripts (cron jobs, etc). I experienced the bad old days when you couldn't rely on /bin/sh to be a proper POSIX shell. It was a mess and it wasted countless hours of human life to work around all the flavours. Python is not as fundamental as the Unix shell but it has replaced a lot of shell scripting. How can we avoid making a lot of work for people? I don't see an easy answer. We don't want Python to become frozen forever (whether it is called 'python', 'python3', or 'py'). OTOH, making /usr/bin/python point to the most recent X.Y release doesn't seem like a good solution either. For example, if I used 'async' as a variable in some of my scripts and then 3.7 broke them. Should we dust off PEP 407 "New release cycle and introducing long-term support versions"? Having /usr/bin/python point to a LTS release seems better to me. I don't know if the core developers are willing to support PEP 407 though. Maybe OS packagers like Red Hat and Debian will already do something like LTS releases and core developers don't need to. /usr/bin/python in Red Hat has behaved like that, as far as I know. Another idea is that we could adopt something like the Rust "language edition" system. Obviously lots of details to be worked out. If we had that, the 'py' command could take an argument to specify the Python edition. OTOH, perhaps deprecation warnings and __future__ achieves most of the same benefits. Maintaining different editions sounds like a lot of work. More work than doing LTS releases. Maybe the solution is just that we become a lot more careful about making incompatible changes. To me, that would seem to reduce the rate that Python is improving. However, a less evolved but more stable Python could actually have a higher value to society. We could create an experimental branch of Python, e.g. python-ng. Then, all the crazy new ideas go in there. Only after they are stable would we merge them into the stable version of Python. I'm not sure how well that works in practice though. That's similar to what Linux did with the even/odd version numbering. It turned into a mess because the experimental (next) version quickly outran the stable version and merging fixes between them was difficult. They abandoned that and now use something like PEP 407 for LTS releases. Regards, Neil
![](https://secure.gravatar.com/avatar/e2371bef92eb40cd7c586e9f2cc75cd8.jpg?s=120&d=mm&r=g)
Steven D'Aprano writes:
But even if representative, this survey only tells us what version people are using, now how they invoke it. We can't conclude that the command "python" means Python 3 for these users. We simply don't know one way or another (and I personally wouldn't want to hazard a guess.)
Agreed on "can't tell invocation". I've been using "pythonX.Y" since the last time I used Red Hat a lot (which was when Red Hat required Python 1.5.2 or it almost wouldn't boot, and before several core developers were born, I suspect). We should also remember that Python is often invoked implicitly in scripts that may be even older than that. I don't think that Perl and PHP experience are sufficiently analogous. As far as I can tell, they're pretty much backward compatible, except that errors became valid code. The unicode -> str, str -> bytes upgrade in Python 3 means that an awful lot of scripts break if you use the wrong one. I think in the spirit of saving keystrokes ;-), we should encourage the use of the "py" wrapper. Yet another Steve -- Associate Professor Division of Policy and Planning Science http://turnbull.sk.tsukuba.ac.jp/ Faculty of Systems and Information Email: turnbull@sk.tsukuba.ac.jp University of Tsukuba Tel: 029-853-5175 Tennodai 1-1-1, Tsukuba 305-8573 JAPAN
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Wed, Feb 13, 2019 at 3:02 PM Steven D'Aprano
On Wed, Feb 13, 2019 at 05:16:54PM -0500, Terry Reedy wrote:
It appears python is already python3 for a large majority of human users (as opposed to machines).
https://www.jetbrains.com/research/python-developers-survey-2018/ Nearly 20000 valid responses, Oct-Nov.
They may be valid responses, but we don't know if they are representative of the broader Python community. Its a self-selected survey of people which always makes the results statistically suspect.
(By definition, an Internet survey eliminates responses from people who don't fill out surveys on the Internet.)
BUt even if representative, this survey only tells us what version people are using, now how they invoke it. We can't conclude that the command "python" means Python 3 for these users. We simply don't know one way or another (and I personally wouldn't want to hazard a guess.)
Can we gather data? What if pip started reporting info on how it was run when contacting pypi? What info would be useful? I guess whether it's pip/pip3/python -m pip/python3 -m pip would be nice to know. I don't know if sys.executable would tell us anything useful or not. pip knows where the current python's script directory is; maybe it should report whether it contains 'python2', 'python3', 'python', and perhaps which ones are the same as each other? -n -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/5615a372d9866f203a22b2c437527bbb.jpg?s=120&d=mm&r=g)
On Wed, Feb 13, 2019 at 03:33:21PM -0800, Barry Warsaw wrote:
I just don’t think Python 4 is anything but distant vaporware.
If Python 4 follows 3.9, that could be as little as 3-4 years away :-)
There’s a cost to freaking everyone out that Python 4 is coming and will be as disruptive as Python 3.
Indeed. I do my bit to combat that in two ways: - remind people that Guido has pronounced that Python 4 will not be a disruptive, backwards-incompatible change like Python 3 was; - and use "Python 5000" to refer to any such hypothetical and very unlikely incompatible version.
Calling Python 3.9+1 Python 4 feeds into that FUD for no reason that I can tell except for an aversion to two digit minor version numbers.
I haven't come across this FUD about Python 4, so I wonder whether it exists more in our fears than the reality. I daresay there are a few people out there who will instantly react to even a casual mention of "Python 4" as if it were a concrete upgrade that just broke their servers, but I would hope the average Python coder had more sense. I know that we have to plan for the community we have rather the community we want, but I would be very sad if we had decisions forced on us by the most ignorant, Dunning-Kruger, unteachable and proud of it segment of the community. Any such hypothetical Python 3.10/4.0 version is at least three or four years away. Let's not limit our options until we know whether or not this FUD is widespread. Whatever we plan, we should allow for *both* a Python 3.10 and a Python 4, and then we'll be good even if 4.0 follows 3.12 :-) -- Steven
![](https://secure.gravatar.com/avatar/7faf63886805e94b38a583a03af239bb.jpg?s=120&d=mm&r=g)
On Wed, Feb 13, 2019 at 10:26 AM Petr Viktorin
PEP 394 says:
This recommendation will be periodically reviewed over the next few years, and updated when the core development team judges it appropriate. As a point of reference, regular maintenance releases for the Python 2.7 series will continue until at least 2020.
I think it's time for another review. I'm especially worried about the implication of these:
- If the `python` command is installed, it should invoke the same version of Python as the `python2` command - scripts that are deliberately written to be source compatible with both Python 2.x and 3.x [...] may continue to use `python` on their shebang line.
So, to support scripts that adhere to the recommendation, Python 2 needs to be installed :(
I literally just ran into this problem now. Part of a software suite I've written uses Python to fetch updates during the installation process. Due to the target audience, it needs to access the system Python (only), and support systems as old as RHEL 5 (Python 2.4 and later, including Python 3.x in the same code base, using nothing but the stdlib). The shebang line was "#!/usr/bin/env python" It's been working for years, but was only now reported broken by a user that upgraded their Ubuntu distribution and suddenly had no "python" executable anywhere. But they had python3. I suspect suddenly not having any "python" executable in a Linux system will screw up a lot more people than just me. The workaround was ugly. I'd like to see there always be a `python` executable available if any version of Python is installed. Thanks, Jason -- Jason M. Swails
![](https://secure.gravatar.com/avatar/d3a20bec8951854eb1db68a111438a2a.jpg?s=120&d=mm&r=g)
On 2019-02-13, 23:33 GMT, Barry Warsaw wrote:
Perhaps. I just don’t think Python 4 is anything but distant vaporware. There’s a cost to freaking everyone out that Python 4 is coming and will be as disruptive as Python 3. Calling Python 3.9+1 Python 4 feeds into that FUD for no reason that I can tell except for an aversion to two digit minor version numbers.
Is this relevant to the discussion at hand? We are talking about the binary /usr/bin/python3 which will be surely be provided even by Python 4, won't it? Matěj -- https://matej.ceplovi.cz/blog/, Jabber: mcepl@ceplovi.cz GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8 Reading after a certain age diverts the mind too much from its creative pursuits. Any man who reads too much and uses his own brain too little falls into lazy habits of thinking, just as the man who spends too much time in the theater is tempted to be content with living vicariously instead of living his own life. -- Albert Einstein to The Saturday Evening Post, October 1929
![](https://secure.gravatar.com/avatar/db5f70d2f2520ef725839f046bdc32fb.jpg?s=120&d=mm&r=g)
On Wed, 13 Feb 2019 17:32:44 -0800
Nathaniel Smith
On Wed, Feb 13, 2019 at 3:02 PM Steven D'Aprano
wrote: On Wed, Feb 13, 2019 at 05:16:54PM -0500, Terry Reedy wrote:
It appears python is already python3 for a large majority of human users (as opposed to machines).
https://www.jetbrains.com/research/python-developers-survey-2018/ Nearly 20000 valid responses, Oct-Nov.
They may be valid responses, but we don't know if they are representative of the broader Python community. Its a self-selected survey of people which always makes the results statistically suspect.
(By definition, an Internet survey eliminates responses from people who don't fill out surveys on the Internet.)
BUt even if representative, this survey only tells us what version people are using, now how they invoke it. We can't conclude that the command "python" means Python 3 for these users. We simply don't know one way or another (and I personally wouldn't want to hazard a guess.)
Can we gather data? What if pip started reporting info on how it was run when contacting pypi?
The most important information pip should report is whether it's running on a CI platform (should be doable by looking at a few environment variables, at least for the most popular platforms). Currently nobody knows what the PyPI download stats mean, because they could be 99% human or 99% CI. Regards Antoine.
![](https://secure.gravatar.com/avatar/db5f70d2f2520ef725839f046bdc32fb.jpg?s=120&d=mm&r=g)
On Thu, 14 Feb 2019 00:57:36 -0500
Jason Swails
I literally just ran into this problem now. Part of a software suite I've written uses Python to fetch updates during the installation process. Due to the target audience, it needs to access the system Python (only), and support systems as old as RHEL 5 (Python 2.4 and later, including Python 3.x in the same code base, using nothing but the stdlib). The shebang line was "#!/usr/bin/env python"
It's been working for years, but was only now reported broken by a user that upgraded their Ubuntu distribution and suddenly had no "python" executable anywhere. But they had python3.
I suspect suddenly not having any "python" executable in a Linux system will screw up a lot more people than just me. The workaround was ugly.
I'm not sure what you mean. Isn't the workaround to install Python 2 in this case? Regards Antoine.
![](https://secure.gravatar.com/avatar/97c543aca1ac7bbcfb5279d0300c8330.jpg?s=120&d=mm&r=g)
On Thu, Feb 14, 2019 at 12:43 AM Antoine Pitrou
On Wed, 13 Feb 2019 17:32:44 -0800 Nathaniel Smith
wrote: On Wed, Feb 13, 2019 at 3:02 PM Steven D'Aprano
wrote: On Wed, Feb 13, 2019 at 05:16:54PM -0500, Terry Reedy wrote:
It appears python is already python3 for a large majority of human users (as opposed to machines).
https://www.jetbrains.com/research/python-developers-survey-2018/ Nearly 20000 valid responses, Oct-Nov.
They may be valid responses, but we don't know if they are representative of the broader Python community. Its a self-selected survey of people which always makes the results statistically suspect.
(By definition, an Internet survey eliminates responses from people who don't fill out surveys on the Internet.)
BUt even if representative, this survey only tells us what version people are using, now how they invoke it. We can't conclude that the command "python" means Python 3 for these users. We simply don't know one way or another (and I personally wouldn't want to hazard a guess.)
Can we gather data? What if pip started reporting info on how it was run when contacting pypi?
The most important information pip should report is whether it's running on a CI platform (should be doable by looking at a few environment variables, at least for the most popular platforms). Currently nobody knows what the PyPI download stats mean, because they could be 99% human or 99% CI.
I agree :-) https://github.com/pypa/pip/issues/5499#issuecomment-406840712 That's kind of orthogonal to this discussion though. -n -- Nathaniel J. Smith -- https://vorpus.org
![](https://secure.gravatar.com/avatar/870d613430249e453343efc9667ef636.jpg?s=120&d=mm&r=g)
On 2/13/19 4:24 PM, Petr Viktorin wrote:
I think it's time for another review. [...] Please see this PR for details and a suggested change: https://github.com/python/peps/pull/893
Summary of the thread so far. Antoine Pitrou noted that the PEP should acknowledge that there are now years of established usage of `python` as Python 3 for many conda users, often as the "main" Python. Victor Stinner expressed support for "python" being the latest Python version, citing PHP, Ruby, Perl; containers; mentions of "python" in our docs. Steve Dower later proposed concrete points how to make "python" the default command: * our docs should say "python" consistently * we should recommend that distributors use the same workaround * our docs should describe the recommended workaround in any places people are likely to first encounter it (tutorial, sys.executable, etc.) Chris Barker added that "python3" should still be available, even if "python" is default. Barry Warsaw gave a +1 to making "python" default, noting that there were plans to change this when Python 2 is officially deprecated. But distros need to make decisions about 2020 now. Chris Barker noted that users won't see any discuntinuity in 2020. That's just a date support from CPython devs ends. Victor pointed to discussions on 4.0 vs. 3.10. (I'll ignore discussions on 4.0 in this summary.) Victor also posted some interesting info and links on Fedora and RHEL. There was a discussion on the PSF survey about how many people use Python 3. (I'll ignore this sub-thread, it's not really about the "python" command.) Steve noted that the Windows Store package of Python 3 provides "python", but he is still thinking how to make this reasonable/reliable in the full installer. Several people think "py" on Unix would be a good thing. Neil Schemenauer supposes we would encourage people to use it over "python"/"python2"/"python3", so "python" would be less of an issue. Neil Schemenauer is not opposed to making "python" configurable or eventually pointing it to Python 3. Jason Swails shared experience from running software with a "#!/usr/bin/env python" shebang on a system that didn't have Python 2 (and followed the PEP, so no "python" either). The workaround was ugly. ------------- Since this list is public, I'd like to remind all readers that it is full of people who work extensively with Python 3, and tend to drive it forward at any opportunity. (Myself included, but on this thread I'll leave the arguments to someone else – they're covered adequately.) Thoughts of Python developers are important, but we're not hearing any other voices. Perhaps everyone with a different opinion has already self-selected out. I don't know of a good place for this discussion, and I'm not a good person to give arguments to support the original "python" should be Python 2 direction. (But if I did, I imagine posting them here would feel a bit scary.) But I would not be surprised, or annoyed, if the Council had a private discussion and pronounced "No, sorry, not yet". Anyway, how should this be decided? Where should it be discussed?
![](https://secure.gravatar.com/avatar/6a7c690adf3a7cff20e3c7c52f0160ad.jpg?s=120&d=mm&r=g)
Hi Petr, I would like to add this issue from the devguide where I ask if we need to use python or python3 in the documentation. https://github.com/python/devguide/issues/208 -- Stéphane Wirtel - https://wirtel.be - @matrixise
![](https://secure.gravatar.com/avatar/870d613430249e453343efc9667ef636.jpg?s=120&d=mm&r=g)
On 2/13/19 5:45 PM, Victor Stinner wrote:
Some more context about Petr's change, Fedora, RHEL and Red Hat. [...]
Fedora could switch "python" to Python 3 now*, if the PEP changes to allow it. * "now" has a release date of around October 2019. The next release after that should then be around May 2020.
![](https://secure.gravatar.com/avatar/7faf63886805e94b38a583a03af239bb.jpg?s=120&d=mm&r=g)
On Feb 14, 2019, at 3:44 AM, Antoine Pitrou
wrote: On Thu, 14 Feb 2019 00:57:36 -0500 Jason Swails
wrote: I literally just ran into this problem now. Part of a software suite I've written uses Python to fetch updates during the installation process. Due to the target audience, it needs to access the system Python (only), and support systems as old as RHEL 5 (Python 2.4 and later, including Python 3.x in the same code base, using nothing but the stdlib). The shebang line was "#!/usr/bin/env python"
It's been working for years, but was only now reported broken by a user that upgraded their Ubuntu distribution and suddenly had no "python" executable anywhere. But they had python3.
I suspect suddenly not having any "python" executable in a Linux system will screw up a lot more people than just me. The workaround was ugly.
I'm not sure what you mean. Isn't the workaround to install Python 2 in this case?
I release the software, so the problem is not my machine, it’s others’. The installation process also fetches a local miniconda distribution for the Python utilities that are part of the program suite (and the python programs are optional and typically not installed when this suite is deployed on a supercomputer, for instance). But the software needs to check for updates before it does any of that (hence my concern — this script needs to be able to run before the user does *anything* else, including installing dependencies). This would also be the first time we’d have to give different installation instructions for different versions of the same Linux distro. The workaround from a users perspective is simple for me, but I can’t make that same assumption for all of my users. This is an impediment to keeping the user experience as simple as possible. Thanks, Jason
![](https://secure.gravatar.com/avatar/63c50004c4f6eaf3235b9ea836f4b6cf.jpg?s=120&d=mm&r=g)
I am glad this resurfaced as back in September I proposed updated that very old PEP but I got rejected. https://github.com/python/peps/pull/785 https://github.com/python/peps/pull/785 The main issue is that most distros will not fix it until PEP is refreshed because most of them do want to follow PEPs. There is still hope. Cheers Sorin
On 13 Feb 2019, at 16:20, Victor Stinner
wrote: Hi,
I'm a (strong) supporter of providing a "python" command which would be the latest Python version!
As php does nowadays (after previous issues with "php4" vs "php5".) I don't recall that perl had "perl4" vs "perl5", the command was always "perl", no? Same for Ruby: it was still "ruby" after for Ruby 2, no? Only Python and PHP used different program names depending on the language version, no? And PHP now moved back to a single "php" program.
In the container and virtualenv era, it's now easy to get your favorite Python version for the "python" command.
On my Windows VM, "python" is Python 3.7 :-) In virtual environments, "python" can also be Python 3 as well.
I recall that I saw commands using "python" rather than "python3" in the *official* Python 3 documentation: see examples below (*). Problem: On Windows, "python" is the right command. "python3" doesn't work (doesn't exist) on Windows. Should we write the doc for Windows or for Unix? Oooops.
There was an interesting discussion about the Python version following Python 3.9: Python 3.10 or Python 4? And what are the issues which would make us prefer 3.10 rather than 4.0? https://mail.python.org/pipermail/python-committers/2018-September/006152.ht...
One practical issue is that right now, six.PY3 is defined by "PY3 = sys.version_info[0] == 3" and so "if six.PY3:" will be false on Python 4.
Another interesting thing to mention is the Unix Python launcher ("py") written by Brett Cannon written in Rust: https://github.com/brettcannon/python-launcher
(*) A few examples of "python" commands in the Python official documentation
"$ python prog.py -h" https://docs.python.org/dev/library/argparse.html
"$ python logctx.py" https://docs.python.org/dev/howto/logging-cookbook.html
"python setup.py install" https://docs.python.org/dev/install/index.html
"python --help" https://docs.python.org/dev/howto/argparse.html
"python setup.py build" https://docs.python.org/dev/extending/building.html
"exec python $0 ${1+"$@"}" https://docs.python.org/dev/faq/library.html
"python setup.py --help build_ext" https://docs.python.org/dev/distutils/configfile.html
Victor
Le mer. 13 févr. 2019 à 16:49, Antoine Pitrou
a écrit : On Wed, 13 Feb 2019 16:24:48 +0100 Petr Viktorin
wrote: PEP 394 says:
This recommendation will be periodically reviewed over the next few years, and updated when the core development team judges it appropriate. As a point of reference, regular maintenance releases for the Python 2.7 series will continue until at least 2020.
I think it's time for another review. I'm especially worried about the implication of these:
- If the `python` command is installed, it should invoke the same version of Python as the `python2` command - scripts that are deliberately written to be source compatible with both Python 2.x and 3.x [...] may continue to use `python` on their shebang line.
So, to support scripts that adhere to the recommendation, Python 2 needs to be installed :(
I think PEP 394 should acknowledge that there are now years of established usage of `python` as Python 3 for many conda users.
Regards
Antoine.
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com
-- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/sorin.sbarnea%40gmail.com
![](https://secure.gravatar.com/avatar/7557037e8ed553e4231f355c39f97970.jpg?s=120&d=mm&r=g)
On 13.02.19 17:20, Victor Stinner wrote:
Hi,
I'm a (strong) supporter of providing a "python" command which would be the latest Python version!
This very much depends on what is working with the latest Python version, and what amount of third party packages your distro has to support. It doesn't have to be the newest version.
As php does nowadays (after previous issues with "php4" vs "php5".) I don't recall that perl had "perl4" vs "perl5", the command was always "perl", no? Same for Ruby: it was still "ruby" after for Ruby 2, no? Only Python and PHP used different program names depending on the language version, no? And PHP now moved back to a single "php" program.
it's not only upstreams doing that kind of versioned names; distros are doing that to ease the pain for larger transitions.
In the container and virtualenv era, it's now easy to get your favorite Python version for the "python" command.
On my Windows VM, "python" is Python 3.7 :-) In virtual environments, "python" can also be Python 3 as well.
maybe the PEP should recommend to have python3 in virtual environments as well? Matthias
![](https://secure.gravatar.com/avatar/7557037e8ed553e4231f355c39f97970.jpg?s=120&d=mm&r=g)
On 13.02.19 16:24, Petr Viktorin wrote:
PEP 394 says:
This recommendation will be periodically reviewed over the next few years, and updated when the core development team judges it appropriate. As a point of reference, regular maintenance releases for the Python 2.7 series will continue until at least 2020.
I think it's time for another review. I'm especially worried about the implication of these:
- If the `python` command is installed, it should invoke the same version of Python as the `python2` command - scripts that are deliberately written to be source compatible with both Python 2.x and 3.x [...] may continue to use `python` on their shebang line.
So, to support scripts that adhere to the recommendation, Python 2 needs to be installed :(
Debian's concern about pointing python to python3 is that it will break software after an upgrade. The current state seems is still the same that Debian doesn't want to ship a python symlink after the Python2 removal. For Ubuntu, I'm not sure if I want a python executable at all, because there is not much progress in handling more than one python installation, so just using python3 for the distro sounds fine. pypi.org now recommends unconditionally installing with pip, and pip is still happy to modify system installed packages when asked, messing around with the distro packages. But probably that kind of users then install their own python symlink anyway. For the Ubuntu 20.04 LTS release and the Debian bullseye release (maybe 2021), I am trying to make sure that the python shebang isn't used by distro packages anymore (either by removing python2 altogether, or by using the python2/python2.7 shebangs). Matthias
![](https://secure.gravatar.com/avatar/4dc045274504f02e7a0b6264e96da643.jpg?s=120&d=mm&r=g)
Le jeu. 14 févr. 2019 à 14:38, Matthias Klose
Debian's concern about pointing python to python3 is that it will break software after an upgrade. The current state seems is still the same that Debian doesn't want to ship a python symlink after the Python2 removal.
The other safer alternative is to start to provide "py" launcher on Unix as well. Since it's something new, it's perfectly fine to decide from the start to make it point to the latest Python version by default. Victor -- Night gathers, and now my watch begins. It shall not end until my death.
![](https://secure.gravatar.com/avatar/01aa7d6d4db83982a2f6dd363d0ee0f3.jpg?s=120&d=mm&r=g)
On Feb 13, 2019, at 23:08, Matěj Cepl
Is this relevant to the discussion at hand? We are talking about the binary /usr/bin/python3 which will be surely be provided even by Python 4, won't it?
Why would it be? Since this is all hypothetical anyway <wink>, I’d more likely expect to only ship /usr/bin/python. -Barry
![](https://secure.gravatar.com/avatar/e8600d16ba667cc8d7f00ddc9f254340.jpg?s=120&d=mm&r=g)
On Thu, Feb 14, 2019 at 7:50 AM Victor Stinner
Le jeu. 14 févr. 2019 à 14:38, Matthias Klose
a écrit : Debian's concern about pointing python to python3 is that it will break software after an upgrade. The current state seems is still the same that Debian doesn't want to ship a python symlink after the Python2 removal.
The other safer alternative is to start to provide "py" launcher on Unix as well. Since it's something new, it's perfectly fine to decide from the start to make it point to the latest Python version by default.
Since it has come up a couple of times and in case people are curious, the Python Launcher for UNIX is currently available at https://crates.io/crates/python-launcher and the basics are there. I have one more key feature to implement -- `py --list` -- before I view it as having all the basics in place. Once I have --list done it will be trying to tackle the hard issue of how to tie in things like PyPy or non-PATH-installed interpreters into the launcher (which, since it is configuration, people will bikeshed on forever about, so maybe I should ignore people and solve it quickly ;) .
![](https://secure.gravatar.com/avatar/06f67632bd8f7d257a9ee873f83c550a.jpg?s=120&d=mm&r=g)
On Thu, 14 Feb 2019 at 15:52, Victor Stinner
Le jeu. 14 févr. 2019 à 14:38, Matthias Klose
a écrit : Debian's concern about pointing python to python3 is that it will break software after an upgrade. The current state seems is still the same that Debian doesn't want to ship a python symlink after the Python2 removal.
The other safer alternative is to start to provide "py" launcher on Unix as well. Since it's something new, it's perfectly fine to decide from the start to make it point to the latest Python version by default.
While I like very much the idea of having `py` as command, does it really need to be a wrapper command? Why can't it simply be a symlink? /usr/bin/py -> /usr/bin/python3 I worry about (1) startup time overhead of starting another process, (2) added complexity of learning about py's additional command-line options, we don't really need them, imho.
Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com
-- Gustavo J. A. M. Carneiro Gambit Research "The universe is always one step beyond logic." -- Frank Herbert
![](https://secure.gravatar.com/avatar/e8600d16ba667cc8d7f00ddc9f254340.jpg?s=120&d=mm&r=g)
On Thu, Feb 14, 2019 at 10:21 AM Gustavo Carneiro
On Thu, 14 Feb 2019 at 15:52, Victor Stinner
wrote: Le jeu. 14 févr. 2019 à 14:38, Matthias Klose
a écrit : Debian's concern about pointing python to python3 is that it will break software after an upgrade. The current state seems is still the same that Debian doesn't want to ship a python symlink after the Python2 removal.
The other safer alternative is to start to provide "py" launcher on Unix as well. Since it's something new, it's perfectly fine to decide from the start to make it point to the latest Python version by default.
While I like very much the idea of having `py` as command, does it really need to be a wrapper command? Why can't it simply be a symlink?
/usr/bin/py -> /usr/bin/python3
Because that is not guaranteed to be the *latest* version of Python 3, just the *last* version installed or the *first* one that happens to be on PATH.
I worry about (1) startup time overhead of starting another process,
It's being implemented in Rust, uses execv(), etc. The initial design is such that it is meant to minimize overhead such that you should worry more about what you import at startup than using the Python launcher if you're that concerned with startup performance. :) But honestly, you don't *have* to use the launcher; it's just for convenience.
(2) added complexity of learning about py's additional command-line options, we don't really need them, imho.
There's only 2 more and they only work in the first position, so the cognitive overhead is extremely low. In my experience after using 'py' on Windows I consistently miss it on UNIX now, so to me there is enough of a benefit that I will continue to chip away at the project until it's done regardless of whether anyone else uses it. :) -Brett
Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com
-- Gustavo J. A. M. Carneiro Gambit Research "The universe is always one step beyond logic." -- Frank Herbert _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
![](https://secure.gravatar.com/avatar/5dde29b54a3f1b76b2541d0a4a9b232c.jpg?s=120&d=mm&r=g)
On Fri, Feb 15, 2019 at 2:39 PM Brett Cannon
In my experience after using 'py' on Windows I consistently miss it on UNIX now, so to me there is enough of a benefit that I will continue to chip away at the project until it's done regardless of whether anyone else uses it. :)
And I would REALLY like it if as much was the same as possible on all platforms... -CHB
-Brett
Victor -- Night gathers, and now my watch begins. It shall not end until my death. _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/gjcarneiro%40gmail.com
-- Gustavo J. A. M. Carneiro Gambit Research "The universe is always one step beyond logic." -- Frank Herbert _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov
-- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
![](https://secure.gravatar.com/avatar/7f37d34f3bb0e91890c01450f8321524.jpg?s=120&d=mm&r=g)
On Thu, Feb 14, 2019 at 9:29 AM Barry Warsaw
On Feb 13, 2019, at 23:08, Matěj Cepl
wrote: Is this relevant to the discussion at hand? We are talking about the binary /usr/bin/python3 which will be surely be provided even by Python 4, won't it?
Why would it be? Since this is all hypothetical anyway <wink>, I’d more likely expect to only ship /usr/bin/python.
Because nobody can use 'python' and expect that to be anything but a 2and3
compatible interpreter for the next 5+ years given we live in a world where
people routinely have a very real need to write #! lines that works with
obsolete distributions. python3 implies >=3.x, thus python 4, 5, 6, 2069,
3001, and 90210 should all have python3 point to them. realistically
people will stop referring to python3 by 2069 so we could consider removing
the recommendation at that point.
2020 is not the end of use or end of importance for Python 2. It is merely
the end of bugfixes applied by python-dev.
A thing I want to make sure we _don't_ do in the future is allow future
pythonN binaries. python4, python90210, etc. those should never exist.
python, python3, and pythonX.Y only. If we were ever to go back on our
promise and create another world-breaking python version, it could get its
own canonical binary name. But we're specifically planning _not_ to make
that mistake again.
I suspect most of my opining below will be contentious to multiple people
because I describe a state of the world that is at conflict with decisions
multiple independent distros have already made. Accept their mistakes and
move on past it to the hack in that case:
A new "py" launcher isn't going to solve this problem - it is separate and
should be its own PEP as it has its own set of requirements and reasons to
be considered (especially on platforms with no concept of a #!). Recommend
"py" today-ish and nobody can rely on it for at least 10+ years in a wide
support cross platform scripting type of situation because it won't be
present on the obsolete or long term supported things that people have a
need for such #!s to run on.
Not our problem? Well, actually, it is. Matthias speaking for Debian
suggesting they don't want to have "python" at all if it isn't a synonym
for "python2" because it'll break software is... wrong. If software is not
3 compatible and uses "python", it'll also break when python is python3.
Just in a different manner. "python" should point to python3 when a distro
does not require python2 for its core. It should not _vary_ as to which of
2.7 or 3.7 it will point to within a given stable distribution (installing
python2.7 should never suddenly redirect it back to python2). But "python"
itself should always exist when any python interpreter is core to the OS.
That means if a distro no longer requires python2 as part of its base/core
but does require python3... "python" must point to "python3". If a posixy
OS no longer requires python at all (surely there are some by now?) the
question of what python should point to when an OS distro supplied optional
python package gets installed is likely either "nothing at all" or ">=3.x"
but should never be decided as "2.7" (which sadly may be what macOS does).
Do we already have LTS _stable_ distributions making that mistake today?
If so they've done something undesirable for the world at large and we're
already screwed if that distro release is deemed important by masses of
users: There is no way to write a *direct* #! line that works out of the
box to launch a working latest version Python interpreter across all
platforms.
The hack to make that work otherwise involves:
```sh
#!/bin/sh # (or bash if that much is guaranteed)
... some shell logic to find _an_ acceptible interpreter ...
exec "${DISCOVERED_PYTHON}" - <<
![](https://secure.gravatar.com/avatar/f3ba3ecffd20251d73749afbfa636786.jpg?s=120&d=mm&r=g)
On Sat, 16 Feb 2019 at 09:19, Gregory P. Smith
Not our problem? Well, actually, it is. Matthias speaking for Debian suggesting they don't want to have "python" at all if it isn't a synonym for "python2" because it'll break software is... wrong. If software is not 3 compatible and uses "python", it'll also break when python is python3. Just in a different manner. "python" should point to python3 when a distro does not require python2 for its core. It should not _vary_ as to which of 2.7 or 3.7 it will point to within a given stable distribution (installing python2.7 should never suddenly redirect it back to python2). But "python" itself should always exist when any python interpreter is core to the OS. That means if a distro no longer requires python2 as part of its base/core but does require python3... "python" must point to "python3". If a posixy OS no longer requires python at all (surely there are some by now?) the question of what python should point to when an OS distro supplied optional python package gets installed is likely either "nothing at all" or ">=3.x" but should never be decided as "2.7" (which sadly may be what macOS does).
Do we already have LTS _stable_ distributions making that mistake today? If so they've done something undesirable for the world at large and we're already screwed if that distro release is deemed important by masses of users: There is no way to write a direct #! line that works out of the box to launch a working latest version Python interpreter across all platforms.
This is exactly why we want to change Fedora et al to have /usr/bin/python aliased to /usr/bin/python3 by default, and yes, having /usr/bin/python missing by default does indeed break the world (search for Fedora 28 and Ubuntu 16.04 Ansible Python issues for more). While Matthias is still personally reluctant to add the alias for Debian/Ubuntu, the *only* thing preventing aliasing /usr/bin/python to /usr/bin/python3 right now on the Fedora & RHEL side of things is PEP 394, and Guido objected strongly when Petr last tried to get the PEP to even acknowledge that it was reasonable for distros to make that setting configurable on a system-wide basis: https://github.com/python/peps/pull/630 For RHEL 8, the resolution was "Well, we'll ignore the upstream PEP, then" and make the setting configurable anyway, but Fedora tries to work more closely with upstream than that - if we think upstream are giving people bad or outdated advice, then we'll aim to get the advice changed rather than ignoring it. In this case, the advice is outdated: there have been a couple of years of releases with /usr/bin/python missing, so it's time to move to the "/usr/bin/python3" side of source compatibility, and go back to having "just run python" be the way you start a modern Python interpreter, even when you're using the system Python on a Linux distro. Cheers, Nick. P.S. Note that we're not asking for the PEP to say "You should do this..." - just for the PEP to acknowledge it as a reasonable choice for distros to make given the looming Python 2 End of Life. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
![](https://secure.gravatar.com/avatar/01aa7d6d4db83982a2f6dd363d0ee0f3.jpg?s=120&d=mm&r=g)
On Feb 16, 2019, at 09:25, Nick Coghlan
While Matthias is still personally reluctant to add the alias for Debian/Ubuntu, the *only* thing preventing aliasing /usr/bin/python to /usr/bin/python3 right now on the Fedora & RHEL side of things is PEP 394, and Guido objected strongly when Petr last tried to get the PEP to even acknowledge that it was reasonable for distros to make that setting configurable on a system-wide basis: https://github.com/python/peps/pull/630
P.S. Note that we're not asking for the PEP to say "You should do this..." - just for the PEP to acknowledge it as a reasonable choice for distros to make given the looming Python 2 End of Life.
I think this is a reasonable ask. PEP 394 shouldn’t *prevent* distros from doing what they believe is in the best interest of their users. While we do want consistency in the user experience across Linux distros (and more broadly, across all supported platforms), I think we also have to acknowledge that we’re still in a time of transition (maybe more so right now), so we should find ways to allow for experimentation within that context. I’m not sure that I agree with all the proposed changes to PEP 394, but those are the guidelines I think I’ll re-evaluate the PR by. -Barry
![](https://secure.gravatar.com/avatar/7557037e8ed553e4231f355c39f97970.jpg?s=120&d=mm&r=g)
On 16.02.19 00:15, Gregory P. Smith wrote:
On Thu, Feb 14, 2019 at 9:29 AM Barry Warsaw
wrote: On Feb 13, 2019, at 23:08, Matěj Cepl
wrote: Is this relevant to the discussion at hand? We are talking about the binary /usr/bin/python3 which will be surely be provided even by Python 4, won't it?
Why would it be? Since this is all hypothetical anyway <wink>, I’d more likely expect to only ship /usr/bin/python.
Because nobody can use 'python' and expect that to be anything but a 2and3 compatible interpreter for the next 5+ years given we live in a world where people routinely have a very real need to write #! lines that works with obsolete distributions. python3 implies >=3.x, thus python 4, 5, 6, 2069, 3001, and 90210 should all have python3 point to them. realistically people will stop referring to python3 by 2069 so we could consider removing the recommendation at that point.
2020 is not the end of use or end of importance for Python 2. It is merely the end of bugfixes applied by python-dev.
A thing I want to make sure we _don't_ do in the future is allow future pythonN binaries. python4, python90210, etc. those should never exist. python, python3, and pythonX.Y only. If we were ever to go back on our promise and create another world-breaking python version, it could get its own canonical binary name. But we're specifically planning _not_ to make that mistake again.
I suspect most of my opining below will be contentious to multiple people because I describe a state of the world that is at conflict with decisions multiple independent distros have already made. Accept their mistakes and move on past it to the hack in that case:
A new "py" launcher isn't going to solve this problem - it is separate and should be its own PEP as it has its own set of requirements and reasons to be considered (especially on platforms with no concept of a #!). Recommend "py" today-ish and nobody can rely on it for at least 10+ years in a wide support cross platform scripting type of situation because it won't be present on the obsolete or long term supported things that people have a need for such #!s to run on.
Not our problem? Well, actually, it is. Matthias speaking for Debian suggesting they don't want to have "python" at all if it isn't a synonym for "python2" because it'll break software is... wrong. If software is not 3 compatible and uses "python", it'll also break when python is python3. Just in a different manner. "python" should point to python3 when a distro does not require python2 for its core. It should not _vary_ as to which of 2.7 or 3.7 it will point to within a given stable distribution (installing python2.7 should never suddenly redirect it back to python2). But "python" itself should always exist when any python interpreter is core to the OS. That means if a distro no longer requires python2 as part of its base/core but does require python3... "python" must point to "python3". If a posixy OS no longer requires python at all (surely there are some by now?) the question of what python should point to when an OS distro supplied optional python package gets installed is likely either "nothing at all" or ">=3.x" but should never be decided as "2.7" (which sadly may be what macOS does).
There is no notion of a "core" for Debian. So "core" applies to the whole distro, as long as there are python shebangs found. For Ubuntu, you don't have a python command on the default desktop install, just python3. Trying to invoke python, command-not-found tells you: $ python Command 'python' not found, but can be installed with: [...] You also have python3 installed, you can run 'python3' instead. That tells you which way to go.
Do we already have LTS _stable_ distributions making that mistake today? If so they've done something undesirable for the world at large and we're already screwed if that distro release is deemed important by masses of users: There is no way to write a *direct* #! line that works out of the box to launch a working latest version Python interpreter across all platforms.
If you count the above example towards this "mistake", probably yes. But there is *no* way to have a sane way to have what you want. Matthias
![](https://secure.gravatar.com/avatar/7557037e8ed553e4231f355c39f97970.jpg?s=120&d=mm&r=g)
On 16.02.19 18:25, Nick Coghlan wrote:
While Matthias is still personally reluctant to add the alias for Debian/Ubuntu, the *only* thing preventing aliasing /usr/bin/python to /usr/bin/python3 right now on the Fedora & RHEL side of things is PEP 394, and Guido objected strongly when Petr last tried to get the PEP to even acknowledge that it was reasonable for distros to make that setting configurable on a system-wide basis: https://github.com/python/peps/pull/630
No, I'm not "personally reluctant" about this, it's the current majority view of people on the debian-python ML. Barry stepped back as a Debian maintainer, so there are not many people supporting your view. Matthias
![](https://secure.gravatar.com/avatar/7f37d34f3bb0e91890c01450f8321524.jpg?s=120&d=mm&r=g)
On Mon, Feb 18, 2019, 7:34 AM Matthias Klose On Thu, Feb 14, 2019 at 9:29 AM Barry Warsaw On Feb 13, 2019, at 23:08, Matěj Cepl Is this relevant to the discussion at hand? We are talking about
the binary /usr/bin/python3 which will be surely be provided
even by Python 4, won't it? Why would it be? Since this is all hypothetical anyway <wink>, I’d more
likely expect to only ship /usr/bin/python. Because nobody can use 'python' and expect that to be anything but a
2and3
compatible interpreter for the next 5+ years given we live in a world
where
people routinely have a very real need to write #! lines that works with
obsolete distributions. python3 implies >=3.x, thus python 4, 5, 6,
2069,
3001, and 90210 should all have python3 point to them. realistically
people will stop referring to python3 by 2069 so we could consider
removing
the recommendation at that point. 2020 is not the end of use or end of importance for Python 2. It is
merely
the end of bugfixes applied by python-dev. A thing I want to make sure we _don't_ do in the future is allow future
pythonN binaries. python4, python90210, etc. those should never exist.
python, python3, and pythonX.Y only. If we were ever to go back on our
promise and create another world-breaking python version, it could get
its
own canonical binary name. But we're specifically planning _not_ to make
that mistake again. I suspect most of my opining below will be contentious to multiple people
because I describe a state of the world that is at conflict with
decisions
multiple independent distros have already made. Accept their mistakes
and
move on past it to the hack in that case: A new "py" launcher isn't going to solve this problem - it is separate
and
should be its own PEP as it has its own set of requirements and reasons
to
be considered (especially on platforms with no concept of a #!).
Recommend
"py" today-ish and nobody can rely on it for at least 10+ years in a wide
support cross platform scripting type of situation because it won't be
present on the obsolete or long term supported things that people have a
need for such #!s to run on. Not our problem? Well, actually, it is. Matthias speaking for Debian
suggesting they don't want to have "python" at all if it isn't a synonym
for "python2" because it'll break software is... wrong. If software is
not
3 compatible and uses "python", it'll also break when python is python3.
Just in a different manner. "python" should point to python3 when a
distro
does not require python2 for its core. It should not _vary_ as to which
of
2.7 or 3.7 it will point to within a given stable distribution
(installing
python2.7 should never suddenly redirect it back to python2). But
"python"
itself should always exist when any python interpreter is core to the OS.
That means if a distro no longer requires python2 as part of its but does require python3... "python" must point to "python3". If a OS no longer requires python at all (surely there are some by now?) the
question of what python should point to when an OS distro supplied On 16.02.19 00:15, Gregory P. Smith wrote:
base/core
posixy
optional python package gets installed is likely either "nothing at all" or
">=3.x"
but should never be decided as "2.7" (which sadly may be what macOS
does). There is no notion of a "core" for Debian. So "core" applies to the whole
distro, as long as there are python shebangs found. For Ubuntu, you don't have a python command on the default desktop
install, just
python3. Trying to invoke python, command-not-found tells you: $ python Command 'python' not found, but can be installed with: [...] You also have python3 installed, you can run 'python3' instead. That tells you which way to go. "Core" just means part of the minimal install, needed by startup scripts
and the package manager perhaps. This would be a default install with no
package groups selected or perhaps the netinst image for Debian. If
packages in that set don't need a python interpreter, Debian is in great
shape! :) Do we already have LTS _stable_ distributions making that mistake today?
If so they've done something undesirable for the world at large and we're
already screwed if that distro release is deemed important by masses of
users: There is no way to write a *direct* #! line that works out of the
box to launch a working latest version Python interpreter across all
platforms. If you count the above example towards this "mistake", probably yes. But
there
is *no* way to have a sane way to have what you want. Agreed.
In the long run, expecting python 2 to exist is not sane. But given our
pep394 text of "for the time being, all distributions *should* ensure that
python, if installed, refers to the same target as python2," What Debian
has done is still unfortunately encouraged by us.
We've created a world where #! lines cannot be used to invoke an
intentionally compatible script across a wide variety of platforms over
time. But our decision to do that was the decision to have an incompatible
release in the first place.
Too late now. :)
![](https://secure.gravatar.com/avatar/870d613430249e453343efc9667ef636.jpg?s=120&d=mm&r=g)
On 2/14/19 9:56 AM, Petr Viktorin wrote:
On 2/13/19 4:24 PM, Petr Viktorin wrote:
I think it's time for another review. [...] Please see this PR for details and a suggested change: https://github.com/python/peps/pull/893
Summary of the thread so far.
Antoine Pitrou noted that the PEP should acknowledge that there are now years of established usage of `python` as Python 3 for many conda users, often as the "main" Python.
Victor Stinner expressed support for "python" being the latest Python version, citing PHP, Ruby, Perl; containers; mentions of "python" in our docs.
Steve Dower later proposed concrete points how to make "python" the default command: * our docs should say "python" consistently * we should recommend that distributors use the same workaround * our docs should describe the recommended workaround in any places people are likely to first encounter it (tutorial, sys.executable, etc.)
Chris Barker added that "python3" should still be available, even if "python" is default.
Barry Warsaw gave a +1 to making "python" default, noting that there were plans to change this when Python 2 is officially deprecated. But distros need to make decisions about 2020 now.
Chris Barker noted that users won't see any discuntinuity in 2020. That's just a date support from CPython devs ends.
Victor pointed to discussions on 4.0 vs. 3.10. (I'll ignore discussions on 4.0 in this summary.) Victor also posted some interesting info and links on Fedora and RHEL. There was a discussion on the PSF survey about how many people use Python 3. (I'll ignore this sub-thread, it's not really about the "python" command.)
Steve noted that the Windows Store package of Python 3 provides "python", but he is still thinking how to make this reasonable/reliable in the full installer.
Several people think "py" on Unix would be a good thing. Neil Schemenauer supposes we would encourage people to use it over "python"/"python2"/"python3", so "python" would be less of an issue.
Neil Schemenauer is not opposed to making "python" configurable or eventually pointing it to Python 3.
Jason Swails shared experience from running software with a "#!/usr/bin/env python" shebang on a system that didn't have Python 2 (and followed the PEP, so no "python" either). The workaround was ugly.
There haven't been many new ideas since this summary – mostly it was explaining and re-hashing what's been mentioned before. Matthias Klose pointed out some Debian/Ubuntu points, to which I'll add the situation in other distros I know of. *Debian* is concerned that python → python3 will break software after an upgrade. Debian appears to not want to ship the unversioned command after py2 removal. For *Ubuntu*, Matthias is not sure if he wants a python executable at all. He notes that pypi.org recommends pip, and pip still breaks system-installed packages when asked to. For both Ubuntu 20.04 LTS and Debian bullseye, the goal is that distro packages don't use the unversioned shebang. *Fedora* packages don't use the unversioned shebang. If it was up to me, the unversioned command would be removed in F31 (released in the first half of 2019) and then pointed to python3 in F32 (second half). But we'd rather happy to follow upstream consensus. (And the PEP, if it reflects the consensus.) In *RHEL*, the unversioned command is missing by default. Sysadmins can change it, but are advised to use python2/python3 instead. RHEL decision makers don't give the PEP much weight. *Arch* did the switch to python3 a long time ago (and the resulting fire wasn't all that bright). With *Homebrew*, `python` points to Homebrew’s Python 2.7.x (if installed) otherwise the macOS system Python. That's exactly according to the PEP. They tried to switch python to python3 before, and got rather nasty backlash citing PEP 394. I assume they will follow the PEP quite strictly from now on. The *macOS* system Python is out of our hands; Apple ignores upstream recommendations.
![](https://secure.gravatar.com/avatar/351a10f392414345ed67a05e986dc4dd.jpg?s=120&d=mm&r=g)
With *Homebrew*, `python` points to Homebrew’s Python 2.7.x (if installed) otherwise the macOS system Python. That's exactly according to the PEP. They tried to switch python to python3 before, and got rather nasty backlash citing PEP 394. I assume they will follow the PEP quite strictly from now on.
I want to add note here.
When homebrew switched to python -> python3, node-gyp is broken.
It is very widely used tool for web developers.
Since Google was very lazy about adding Python 3 to gyp, node-gyp
can't support Python 3 for a long time.
But this situation is changing. Google added Python 3 support to gyp.
node-gyp project is working on Python 3 support for now.
I think keeping PEP 394 as-is until node-gyp officially support Python 3
would helps many web developers.
--
INADA Naoki
![](https://secure.gravatar.com/avatar/01aa7d6d4db83982a2f6dd363d0ee0f3.jpg?s=120&d=mm&r=g)
There haven't been many new ideas since this summary – mostly it was explaining and re-hashing what's been mentioned before.
Thanks for the summary Petr. Here’s another way to think about the problem. I know Nick and I have talked about this before, but I don’t think any distros have actually done this, though I’ve been out of that business a while now so correct me if I’m wrong. I see this question as having several parts, and the conflation of them is part of the reason why the unversioned `python` command is so problematic. Python is used for: * OS functionality * to run applications that aren’t critical to the OS but are delivered on the OS * as the entry point to the interactive interpreter * to run applications written and deployed on the OS but completely outside of it Which `python` are we trying to change? All of them? For OS functionality, there should probably be a separate command not conflated with /usr/bin/python. The OS can make any adjustments it needs, calling it `spython` (as I think Nick once suggested), or whatever. Nobody but OS maintainers cares what this is called or what version of Python it exposes. I strongly believe that (eventually) the interactive interpreter command should be /usr/bin/python and that this should point to Python 3, since this provides the best experience for beginners, dabblers, etc. So what about the other two use cases? Well, for applications packages within the OS but aren’t critical to it, I think they should always use the versioned shebang, never the unversioned shebang. Distros can control this, so that transition should be easier. The tricky part then seems to me what to do for 3rd parties which are using the distro Python in their shebangs? Nobody sees their code but them, and changing the shebang out from under them could cause their code to break. But don’t they already take lots of precautions and planning for any OS upgrade? Changing the shebang for Python 2 would be just one of the things they’d have to worry about in an OS upgrade. I don’t know whether this analysis is complete or correct, but perhaps it helps inform a way forward on PEP 394. Cheers, -Barry
![](https://secure.gravatar.com/avatar/7f37d34f3bb0e91890c01450f8321524.jpg?s=120&d=mm&r=g)
On Tue, Feb 26, 2019 at 8:59 AM INADA Naoki
With *Homebrew*, `python` points to Homebrew’s Python 2.7.x (if installed) otherwise the macOS system Python. That's exactly according to the PEP. They tried to switch python to python3 before, and got rather nasty backlash citing PEP 394. I assume they will follow the PEP quite strictly from now on.
I want to add note here. When homebrew switched to python -> python3, node-gyp is broken. It is very widely used tool for web developers.
Since Google was very lazy about adding Python 3 to gyp, node-gyp can't support Python 3 for a long time.
But this situation is changing. Google added Python 3 support to gyp. node-gyp project is working on Python 3 support for now.
I think keeping PEP 394 as-is until node-gyp officially support Python 3 would helps many web developers.
In practice, does what /usr/bin/python is even matter to node-gyp? I'd *hope* that it would refer to /usr/bin/python2.7... (does anyone use something as modern as node.js on a system with python 2 but without /usr/bin/python2.7? [i probably don't want to know the answer to that...]) node-gyp's got a great issue number for this: https://github.com/nodejs/node-gyp/issues/1337 -gps
![](https://secure.gravatar.com/avatar/7f37d34f3bb0e91890c01450f8321524.jpg?s=120&d=mm&r=g)
On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw
There haven't been many new ideas since this summary – mostly it was explaining and re-hashing what's been mentioned before.
Thanks for the summary Petr.
Here’s another way to think about the problem. I know Nick and I have talked about this before, but I don’t think any distros have actually done this, though I’ve been out of that business a while now so correct me if I’m wrong.
I see this question as having several parts, and the conflation of them is part of the reason why the unversioned `python` command is so problematic. Python is used for:
* OS functionality * to run applications that aren’t critical to the OS but are delivered on the OS * as the entry point to the interactive interpreter * to run applications written and deployed on the OS but completely outside of it
Which `python` are we trying to change? All of them?
For OS functionality, there should probably be a separate command not conflated with /usr/bin/python. The OS can make any adjustments it needs, calling it `spython` (as I think Nick once suggested), or whatever. Nobody but OS maintainers cares what this is called or what version of Python it exposes.
I strongly believe that (eventually) the interactive interpreter command should be /usr/bin/python and that this should point to Python 3, since this provides the best experience for beginners, dabblers, etc.
So what about the other two use cases? Well, for applications packages within the OS but aren’t critical to it, I think they should always use the versioned shebang, never the unversioned shebang. Distros can control this, so that transition should be easier.
The tricky part then seems to me what to do for 3rd parties which are using the distro Python in their shebangs? Nobody sees their code but them, and changing the shebang out from under them could cause their code to break. But don’t they already take lots of precautions and planning for any OS upgrade? Changing the shebang for Python 2 would be just one of the things they’d have to worry about in an OS upgrade.
A feature that *I* find missing from posix-y OSes that support #! lines is an ability to restrict what can use a given interpreter. For an OS distro provided interpreter, being able to restrict its use to only OS distro provided software would be ideal (so ideal that people who haven't learned the hard distro maintenance lessons may hate me for it). Such a restriction could be implemented within the interpreter itself. For example: Say that only this set of fully qualified path whitelisted .py files are allowed to invoke it, with no interactive, stdin, or command line "-c" use allowed. I'm not aware of anyone actually having done that. It's hard to see how to do that in a *maintainable* manner that people using many distros wouldn't just naively work around by adding themselves to the whitelist rather than providing their own interpreter for their own software stack. It feels more doable without workarounds for something like macOS or any other distro wholly controlled and maintained as a single set of software rather than a widely varying packages. Solving that is way outside the scope of PEP 394. Just food for thought that I'd like to leave as an earworm for the future for distro minded folks. I some people to hate this idea. -gps
I don’t know whether this analysis is complete or correct, but perhaps it helps inform a way forward on PEP 394.
Cheers, -Barry
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/greg%40krypto.org
![](https://secure.gravatar.com/avatar/870d613430249e453343efc9667ef636.jpg?s=120&d=mm&r=g)
On 2/26/19 6:54 PM, Barry Warsaw wrote:
There haven't been many new ideas since this summary – mostly it was explaining and re-hashing what's been mentioned before.
Thanks for the summary Petr.
Here’s another way to think about the problem. I know Nick and I have talked about this before, but I don’t think any distros have actually done this, though I’ve been out of that business a while now so correct me if I’m wrong.
I see this question as having several parts, and the conflation of them is part of the reason why the unversioned `python` command is so problematic. Python is used for:
* OS functionality * to run applications that aren’t critical to the OS but are delivered on the OS * as the entry point to the interactive interpreter * to run applications written and deployed on the OS but completely outside of it
Which `python` are we trying to change? All of them?
For OS functionality, there should probably be a separate command not conflated with /usr/bin/python. The OS can make any adjustments it needs, calling it `spython` (as I think Nick once suggested), or whatever. Nobody but OS maintainers cares what this is called or what version of Python it exposes.
Yup. RHEL 8 actually has exactly that. (It's called /usr/libexec/platform-python; please don't use it!) Fedora (and most other distros) makes this the same as the interpreter for other packaged software. For Fedora the main reason is that don't want to maintain two full separate Python stacks.
I strongly believe that (eventually) the interactive interpreter command should be /usr/bin/python and that this should point to Python 3, since this provides the best experience for beginners, dabblers, etc.
+1
So what about the other two use cases? Well, for applications packages within the OS but aren’t critical to it, I think they should always use the versioned shebang, never the unversioned shebang. Distros can control this, so that transition should be easier.
+1
The tricky part then seems to me what to do for 3rd parties which are using the distro Python in their shebangs? Nobody sees their code but them, and changing the shebang out from under them could cause their code to break. But don’t they already take lots of precautions and planning for any OS upgrade? Changing the shebang for Python 2 would be just one of the things they’d have to worry about in an OS upgrade.
Also, things will break for them anyway, it's just a matter of time. Python 2 *is* going away, eventually. (Right?) I don't think we're doing that many people a favor by keeping /usr/bin/python → python2 around. Instead, we're *hiding* the problem from them. Too many people think python2 is still the "default". Making /usr/bin/python be missing for some time, rather than pointing it to python3 now, is the more explicit way to do the transition.
I don’t know whether this analysis is complete or correct, but perhaps it helps inform a way forward on PEP 394.
I have two very different questions in mind for moving this forward. Who gets to decide on PEP 394 changes? Since so many people on python-dev are in agreement, where do I go for opposing voices?
![](https://secure.gravatar.com/avatar/8702771e2f72afdffc4fcb5527e46354.jpg?s=120&d=mm&r=g)
On 2019-02-26, Gregory P. Smith wrote:
On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw
wrote: For an OS distro provided interpreter, being able to restrict its use to only OS distro provided software would be ideal (so ideal that people who haven't learned the hard distro maintenance lessons may hate me for it).
Interesting idea. I remember when I was helping develop Debian packaging guides for Python software. I had to fight with people to convince them that Debian packages should use #!/usr/bin/pythonX.Y rather than #!/usr/bin/env python The situtation is much better now but I still sometimes have packaged software fail because it picks up my version of /usr/local/bin/python. I don't understand how people can believe grabbing /usr/local/bin/python is going to be a way to build a reliable system.
Such a restriction could be implemented within the interpreter itself. For example: Say that only this set of fully qualified path whitelisted .py files are allowed to invoke it, with no interactive, stdin, or command line "-c" use allowed.
I think this is related to an idea I was tinkering with on the weekend. Why shouldn't we do more compile time linkage of Python packages? At least, I think we give people the option to do it. Obviously you still need to also support run-time import search (interactive REPL, support __import__(unknown_at_compiletime)__). Here is the sketch of the idea (probably half-baked, as most of my ideas are): - add PYTHONPACKAGES envvar and -p options to 'python' - the argument for these options would be a colon separated list of Python package archives (crates, bales, bundles?). The -p option could be a colon separated list or provided multiple times to specify more packages. - the modules/packages contained in those archives become the preferred bytecode code source when those names are imported. We look there first. The crawling around behavor (dynamic import based on sys.path) happens only if a module is not found and could be turned off. - the linking of the modules could be computed when the code is compiled and the package archive created, rather than when the 'import' statement gets executed. This would provide a number of advantages. It would be faster. Code analysis tools could statically determine which modules imported code corresponds too. E.g. if your code calls module.foo, assuming no monkey patching, you know what code 'foo' actually is. - to get extra fancy, the package archives could be dynamic link libraries containing "frozen modules" like this FB experiment: https://github.com/python/cpython/pull/9320 That way, you avoid the unmarshal step and just execute the module bytecode directly. On startup, Python would dlopen all of the package archives specified by PYTHONPACKAGES. On init, it would build an index of the package tree and it would have the memory location for the code object for each module. That would seem like quite a useful thing. For an application like Mercurial, they could build all the modules/packages required into a single package archive. Or, there would be a small number of archives (one for standard Python library, one for everything else that Mercurial needs). Now that I write this, it sounds a lot like the debate between static linking and dynamic linking. Golang does static linking and people seem to like the single executable distribution. Regards, Neil
![](https://secure.gravatar.com/avatar/4dc045274504f02e7a0b6264e96da643.jpg?s=120&d=mm&r=g)
Le mar. 26 févr. 2019 à 22:24, Gregory P. Smith
A feature that I find missing from posix-y OSes that support #! lines is an ability to restrict what can use a given interpreter.
Fedora runs system tools (like "/usr/bin/semanage", tool to manager SELinux) with "python3 -Es": $ head /usr/sbin/semanage #! /usr/bin/python3 -Es -E: ignore PYTHON* environment variables (such as PYTHONPATH) -s: don't add user site directory to sys.path Is it what you mean?
Such a restriction could be implemented within the interpreter itself. For example: Say that only this set of fully qualified path whitelisted .py files are allowed to invoke it, with no interactive, stdin, or command line "-c" use allowed. I'm not aware of anyone actually having done that. It's hard to see how to do that in a maintainable manner that people using many distros wouldn't just naively work around by adding themselves to the whitelist rather than providing their own interpreter for their own software stack. It feels more doable without workarounds for something like macOS or any other distro wholly controlled and maintained as a single set of software rather than a widely varying packages.
Technically, Python initialization is highly customizable: see _PyCoreConfig in Include/coreconfig.h. But we lack a public API for that :-) https://www.python.org/dev/peps/pep-0432/ is a work-in-progress. With a proper public API, building your own interpreter would take a few lines of C to give you fine control on what Python can do or not. Extract of Programs/_freeze_importlib.c (give you an idea of what can be done): --- _PyCoreConfig config = _PyCoreConfig_INIT; config.user_site_directory = 0; config.site_import = 0; config.use_environment = 0; config.program_name = L"./_freeze_importlib"; /* Don't install importlib, since it could execute outdated bytecode. */ config._install_importlib = 0; config._frozen = 1; _PyInitError err = _Py_InitializeFromConfig(&config); --- As Petr wrote below, RHEL 8 has a private /usr/libexec/platform-python which is the Python used to run system tools (written in Python). But this Python isn't customized. I'm not sure that there is a strong need to customize Python default configuration for this interpreter. Note: Sorry to hijack again this thread with unrelated discussions :-( Victor -- Night gathers, and now my watch begins. It shall not end until my death.
![](https://secure.gravatar.com/avatar/be200d614c47b5a4dbb6be867080e835.jpg?s=120&d=mm&r=g)
On 2/26/2019 1:20 PM, Gregory P. Smith wrote:
For an OS distro provided interpreter, being able to restrict its use to only OS distro provided software would be ideal (so ideal that people who haven't learned the hard distro maintenance lessons may hate me for it).
Such a restriction could be implemented within the interpreter itself. For example: Say that only this set of fully qualified path whitelisted .py files are allowed to invoke it, with no interactive, stdin, or command line "-c" use allowed. I'm not aware of anyone actually having done that. It's hard to see how to do that in a /maintainable/ manner that people using many distros wouldn't just naively work around by adding themselves to the whitelist rather than providing their own interpreter for their own software stack. It feels more doable without workarounds for something like macOS or any other distro wholly controlled and maintained as a single set of software rather than a widely varying packages.
Solving that is way outside the scope of PEP 394. Just food for thought that I'd like to leave as an earworm for the future for distro minded folks. I some people to hate this idea.
I haven't caught up on this thread yet, but this sounds a lot like the "Restricting the entry point" section of https://www.python.org/dev/peps/pep-0551/ (which is still a draft, so if anyone wants to help make it more like what they want, I'm happy to have contributors). So I'm in favour of making this easy (since I'm already having to deal with it being difficult ;) ), as it's extremely valuable for security-conscious deployments as well as the distro package cases mentioned by Gregory. Cheers, Steve
![](https://secure.gravatar.com/avatar/5dde29b54a3f1b76b2541d0a4a9b232c.jpg?s=120&d=mm&r=g)
On Tue, Feb 26, 2019 at 9:58 AM Barry Warsaw
I see this question as having several parts, and the conflation of them is part of the reason why the unversioned `python` command is so problematic. Python is used for:
* OS functionality * to run applications that aren’t critical to the OS but are delivered on the OS * as the entry point to the interactive interpreter * to run applications written and deployed on the OS but completely outside of it
For OS functionality, there should probably be a separate command not conflated with /usr/bin/python. The OS can make any adjustments it needs, calling it `spython` (as I think Nick once suggested), or whatever. Nobody but OS maintainers cares what this is called or what version of Python it exposes.
I'm not sure that's necessary at all -- the OS should simply use an unambiguous path! I was a RedHat user way back when when in the midst of the python1.5 => 2.0 transition. RedHat had a bunch of system scripts that had (I think): #!/usr/bin/env python In them (or maybe /usr/bin/python, but I'm pretty sure it was the env version). In any case, when you installed python2 (to local) your system scripts would ll break (even if they were python2 compatible, there wasn't a lot of difference, but RedHat also depended on extra packages...) So what we had to do was install python 2, remove the "python" command that came with it, and use "python2" in all our scripts. This was simply broken, and it was RedHat's fault. If they had used: /usr/bin/python1.5 in their shebang lines, there would have been no problem. And users could still use the system python1.5 if they wanted, or install an update, or whatever. My pint is: Any OS that ships OS scripts that expect "python" to be a specific version (or worse, a specific install) is broken. Some distros are going to ignore the PEP anyway, so there is no harm (and some good) in specifying in teh PEP the way we think it SHOULD be done, and then see what happens -- we don't need to make the PEP match current practice. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
![](https://secure.gravatar.com/avatar/01aa7d6d4db83982a2f6dd363d0ee0f3.jpg?s=120&d=mm&r=g)
On Feb 26, 2019, at 13:34, Petr Viktorin
I have two very different questions in mind for moving this forward.
Who gets to decide on PEP 394 changes?
Honestly, I think it’s the active distro maintainers who need to make this decision. They have the pulse of their own communities and users, and can make the best decisions and compromises for their constituents. I personally am not part of that any more, so I have no problem having no say (despite still having opinions :).
Since so many people on python-dev are in agreement, where do I go for opposing voices?
Well, why look for more dissent? If you can align Homebrew, Fedora-et-al, Debian-et-al, and we already know what Arch has done, and the PEP authors are in agreement, isn't that enough to JFDI? It couldn’t hurt to reach out to a few other distros, but do you think they will have substantially different opinions than what you’ve gathered already? -Barry
![](https://secure.gravatar.com/avatar/01aa7d6d4db83982a2f6dd363d0ee0f3.jpg?s=120&d=mm&r=g)
On Feb 26, 2019, at 14:04, Neil Schemenauer
Interesting idea. I remember when I was helping develop Debian packaging guides for Python software. I had to fight with people to convince them that Debian packages should use
#!/usr/bin/pythonX.Y
rather than
#!/usr/bin/env python
Indeed. I used to fight that battle quite a bit, although at least in my circles that lesson has by now been learned. `/usr/bin/env python` is great for development and terrible for deployment. -Barry
![](https://secure.gravatar.com/avatar/5dde29b54a3f1b76b2541d0a4a9b232c.jpg?s=120&d=mm&r=g)
On Tue, Feb 26, 2019 at 3:25 PM Barry Warsaw
Who gets to decide on PEP 394 changes?
Honestly, I think it’s the active distro maintainers who need to make this decision. They have the pulse of their own communities and users, and can make the best decisions and compromises for their constituents. I personally am not part of that any more, so I have no problem having no say (despite still having opinions :).
The PEP is what the Python community recommends. The distro maintainers can (and will) do whatever they want. IF we are going to let the distros drive it, then there is no point to a the PEP. Well, why look for more dissent? If you can align Homebrew, Fedora-et-al,
Debian-et-al, and we already know what Arch has done, and the PEP authors are in agreement, isn't that enough to JFDI?
more than enough :-) -CHB ----- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker@noaa.gov
![](https://secure.gravatar.com/avatar/2c69ccb9eb83c7ef2ba155a850df68a9.jpg?s=120&d=mm&r=g)
While these are interesting ideas, wouldn't it be better to leave this kind
of packaging to snap and similar utilities that bundle the language support
and libraries to allow simple isolated installation.
Kind regards
Steve Holden
On Tue, Feb 26, 2019 at 10:05 PM Neil Schemenauer
On 2019-02-26, Gregory P. Smith wrote:
On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw
wrote: For an OS distro provided interpreter, being able to restrict its use to only OS distro provided software would be ideal (so ideal that people who haven't learned the hard distro maintenance lessons may hate me for it). Interesting idea. I remember when I was helping develop Debian packaging guides for Python software. I had to fight with people to convince them that Debian packages should use
#!/usr/bin/pythonX.Y
rather than
#!/usr/bin/env python
The situtation is much better now but I still sometimes have packaged software fail because it picks up my version of /usr/local/bin/python. I don't understand how people can believe grabbing /usr/local/bin/python is going to be a way to build a reliable system.
Such a restriction could be implemented within the interpreter itself. For example: Say that only this set of fully qualified path whitelisted .py files are allowed to invoke it, with no interactive, stdin, or command line "-c" use allowed.
I think this is related to an idea I was tinkering with on the weekend. Why shouldn't we do more compile time linkage of Python packages? At least, I think we give people the option to do it. Obviously you still need to also support run-time import search (interactive REPL, support __import__(unknown_at_compiletime)__).
Here is the sketch of the idea (probably half-baked, as most of my ideas are):
- add PYTHONPACKAGES envvar and -p options to 'python'
- the argument for these options would be a colon separated list of Python package archives (crates, bales, bundles?). The -p option could be a colon separated list or provided multiple times to specify more packages.
- the modules/packages contained in those archives become the preferred bytecode code source when those names are imported. We look there first. The crawling around behavor (dynamic import based on sys.path) happens only if a module is not found and could be turned off.
- the linking of the modules could be computed when the code is compiled and the package archive created, rather than when the 'import' statement gets executed. This would provide a number of advantages. It would be faster. Code analysis tools could statically determine which modules imported code corresponds too. E.g. if your code calls module.foo, assuming no monkey patching, you know what code 'foo' actually is.
- to get extra fancy, the package archives could be dynamic link libraries containing "frozen modules" like this FB experiment: https://github.com/python/cpython/pull/9320 That way, you avoid the unmarshal step and just execute the module bytecode directly. On startup, Python would dlopen all of the package archives specified by PYTHONPACKAGES. On init, it would build an index of the package tree and it would have the memory location for the code object for each module.
That would seem like quite a useful thing. For an application like Mercurial, they could build all the modules/packages required into a single package archive. Or, there would be a small number of archives (one for standard Python library, one for everything else that Mercurial needs).
Now that I write this, it sounds a lot like the debate between static linking and dynamic linking. Golang does static linking and people seem to like the single executable distribution.
Regards,
Neil _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com
![](https://secure.gravatar.com/avatar/7f37d34f3bb0e91890c01450f8321524.jpg?s=120&d=mm&r=g)
On Tue, Feb 26, 2019 at 2:28 PM Victor Stinner
Le mar. 26 févr. 2019 à 22:24, Gregory P. Smith
a écrit : A feature that I find missing from posix-y OSes that support #! lines is an ability to restrict what can use a given interpreter.
Fedora runs system tools (like "/usr/bin/semanage", tool to manager SELinux) with "python3 -Es":
$ head /usr/sbin/semanage #! /usr/bin/python3 -Es
-E: ignore PYTHON* environment variables (such as PYTHONPATH) -s: don't add user site directory to sys.path
Is it what you mean?
Not quite. I meant that python interpreter would need to decide /usr/sbin/semanage is allowed to use it as an interpreter. -gps
Such a restriction could be implemented within the interpreter itself. For example: Say that only this set of fully qualified path whitelisted .py files are allowed to invoke it, with no interactive, stdin, or command line "-c" use allowed. I'm not aware of anyone actually having done that. It's hard to see how to do that in a maintainable manner that people using many distros wouldn't just naively work around by adding themselves to the whitelist rather than providing their own interpreter for their own software stack. It feels more doable without workarounds for something like macOS or any other distro wholly controlled and maintained as a single set of software rather than a widely varying packages.
Technically, Python initialization is highly customizable: see _PyCoreConfig in Include/coreconfig.h.
But we lack a public API for that :-) https://www.python.org/dev/peps/pep-0432/ is a work-in-progress.
With a proper public API, building your own interpreter would take a few lines of C to give you fine control on what Python can do or not.
Extract of Programs/_freeze_importlib.c (give you an idea of what can be done): --- _PyCoreConfig config = _PyCoreConfig_INIT; config.user_site_directory = 0; config.site_import = 0; config.use_environment = 0; config.program_name = L"./_freeze_importlib"; /* Don't install importlib, since it could execute outdated bytecode. */ config._install_importlib = 0; config._frozen = 1;
_PyInitError err = _Py_InitializeFromConfig(&config); ---
As Petr wrote below, RHEL 8 has a private /usr/libexec/platform-python which is the Python used to run system tools (written in Python). But this Python isn't customized. I'm not sure that there is a strong need to customize Python default configuration for this interpreter.
Note: Sorry to hijack again this thread with unrelated discussions :-(
Victor -- Night gathers, and now my watch begins. It shall not end until my death.
![](https://secure.gravatar.com/avatar/7f37d34f3bb0e91890c01450f8321524.jpg?s=120&d=mm&r=g)
On Tue, Feb 26, 2019 at 2:31 PM Steve Dower
On 2/26/2019 1:20 PM, Gregory P. Smith wrote:
For an OS distro provided interpreter, being able to restrict its use to only OS distro provided software would be ideal (so ideal that people who haven't learned the hard distro maintenance lessons may hate me for it).
Such a restriction could be implemented within the interpreter itself. For example: Say that only this set of fully qualified path whitelisted .py files are allowed to invoke it, with no interactive, stdin, or command line "-c" use allowed. I'm not aware of anyone actually having done that. It's hard to see how to do that in a /maintainable/ manner that people using many distros wouldn't just naively work around by adding themselves to the whitelist rather than providing their own interpreter for their own software stack. It feels more doable without workarounds for something like macOS or any other distro wholly controlled and maintained as a single set of software rather than a widely varying packages.
Solving that is way outside the scope of PEP 394. Just food for thought that I'd like to leave as an earworm for the future for distro minded folks. I some people to hate this idea.
I haven't caught up on this thread yet, but this sounds a lot like the "Restricting the entry point" section of https://www.python.org/dev/peps/pep-0551/ (which is still a draft, so if anyone wants to help make it more like what they want, I'm happy to have contributors).
So I'm in favour of making this easy (since I'm already having to deal with it being difficult ;) ), as it's extremely valuable for security-conscious deployments as well as the distro package cases mentioned by Gregory.
Similar. What I'm talking about has nothing to do with _security_ (pep-0551's focus) and only to do with installed interpreter maintainability. But an implementation of the concept of deciding what is allowed to use an entry point in what manner might be the same. :) As for 551, I'm not a fan of pretending you can aid security by restricting use of an interpreter; when an adversary has the ability to trigger an exec of new process with args or input of its desire, their opponent lost the game several moves ago. Defense in depth minded people may disagree and still desire the feature. -gps
![](https://secure.gravatar.com/avatar/7a9c1d88f484c9806bceca0d6d91e948.jpg?s=120&d=mm&r=g)
On Tue, Feb 26, 2019 at 2:07 PM Neil Schemenauer
On 2019-02-26, Gregory P. Smith wrote:
On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw
wrote: For an OS distro provided interpreter, being able to restrict its use to only OS distro provided software would be ideal (so ideal that people who haven't learned the hard distro maintenance lessons may hate me for it). This idea has some definite problems. I think enforcing it via convention is about as much as would be good to do. Anything more and you make it hard for people who really need to use the vendor provided interpreter from being able to do so.
Why might someone need to use the distro provided interpreter? * Vendor provides some python modules in their system packages which are not installable from pip (possibly even a proprietary extension module, so not even buildable from source or copyable from the system location) which the end user needs to use to do something to their system. * End user writes a python module which is a plugin to a system tool which has to be installed into the system python to from which that system tool runs. The user then wants to write a script which uses the system tool with the plugin in order to do something to their system outside of the system tool (perhaps the system tool is GUI-driven and the user wants to automate a part of it via the python module). They need their script to use the system python so that they are using the same code as the system tool itself would use. There's probably other scenarios where the benefits of locking the user out of the system python outweigh the benefits but these are the ones that I've run across lately. -Toshio
![](https://secure.gravatar.com/avatar/7f37d34f3bb0e91890c01450f8321524.jpg?s=120&d=mm&r=g)
On Wed, Feb 27, 2019 at 5:12 PM Toshio Kuratomi
On Tue, Feb 26, 2019 at 2:07 PM Neil Schemenauer
wrote: On 2019-02-26, Gregory P. Smith wrote:
On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw
wrote: For an OS distro provided interpreter, being able to restrict its use to only OS distro provided software would be ideal (so ideal that people who haven't learned the hard distro maintenance lessons may hate me for it). This idea has some definite problems. I think enforcing it via convention is about as much as would be good to do. Anything more and you make it hard for people who really need to use the vendor provided interpreter from being able to do so.
Why might someone need to use the distro provided interpreter?
* Vendor provides some python modules in their system packages which are not installable from pip (possibly even a proprietary extension module, so not even buildable from source or copyable from the system location) which the end user needs to use to do something to their system. * End user writes a python module which is a plugin to a system tool which has to be installed into the system python to from which that system tool runs. The user then wants to write a script which uses the system tool with the plugin in order to do something to their system outside of the system tool (perhaps the system tool is GUI-driven and the user wants to automate a part of it via the python module). They need their script to use the system python so that they are using the same code as the system tool itself would use.
There's probably other scenarios where the benefits of locking the user out of the system python outweigh the benefits but these are the ones that I've run across lately.
Agreed. The convention approach as someone said RHEL 8 has apparently done with an os distro reserved interpreter (yay) is likely good enough for most situations. I'd go a *little* further than that and suggest such an os distro reserved interpreter attempt to prevent installation of packages (ie: remove pip/ensurepip/distutils) via any other means than the OS package manager (rpms, debs). Obviously that can't actually prevent someone from figuring out how to run getpip or manually installing trees of packages within its sys.path, but it acts as a deterrent suggesting that this interpreter is not intended for arbitrary software installation. -gps
![](https://secure.gravatar.com/avatar/870d613430249e453343efc9667ef636.jpg?s=120&d=mm&r=g)
On 2/28/19 6:56 PM, Gregory P. Smith wrote:
On Wed, Feb 27, 2019 at 5:12 PM Toshio Kuratomi
mailto:a.badger@gmail.com> wrote: On Tue, Feb 26, 2019 at 2:07 PM Neil Schemenauer
mailto:nas-python@arctrix.com> wrote: On 2019-02-26, Gregory P. Smith wrote: > On Tue, Feb 26, 2019 at 9:55 AM Barry Warsaw
mailto:barry@python.org> wrote: > For an OS distro provided interpreter, being able to restrict its use to > only OS distro provided software would be ideal (so ideal that people who > haven't learned the hard distro maintenance lessons may hate me for it). This idea has some definite problems. I think enforcing it via convention is about as much as would be good to do. Anything more and you make it hard for people who really need to use the vendor provided interpreter from being able to do so.
Why might someone need to use the distro provided interpreter?
* Vendor provides some python modules in their system packages which are not installable from pip (possibly even a proprietary extension module, so not even buildable from source or copyable from the system location) which the end user needs to use to do something to their system. * End user writes a python module which is a plugin to a system tool which has to be installed into the system python to from which that system tool runs. The user then wants to write a script which uses the system tool with the plugin in order to do something to their system outside of the system tool (perhaps the system tool is GUI-driven and the user wants to automate a part of it via the python module). They need their script to use the system python so that they are using the same code as the system tool itself would use.
There's probably other scenarios where the benefits of locking the user out of the system python outweigh the benefits but these are the ones that I've run across lately.
Agreed. The convention approach as someone said RHEL 8 has apparently done with an os distro reserved interpreter (yay) is likely good enough for most situations.
I'd go a /little/ further than that and suggest such an os distro reserved interpreter attempt to prevent installation of packages (ie: remove pip/ensurepip/distutils) via any other means than the OS package manager (rpms, debs). Obviously that can't actually prevent someone from figuring out how to run getpip or manually installing trees of packages within its sys.path, but it acts as a deterrent suggesting that this interpreter is not intended for arbitrary software installation.
Currently, in RHEL/Fedora: - "sudo pip" installs to /usr/local/, RPM packages install to /usr/ - with "-I", the interpreter does not look into /usr/local/. AFAIK, Debian/Ubuntu have something very similar, and were first to do it. What remains to tie this together is making "platform-python" always run with -I. This is PEP 432's "system-python" use case / design goal. (The RHEL/Fedora platform-python was initially called system-python. We renamed to it so it doesn't clash with the PEP. It's the same use case, but we don't want to assign semantics to the name prematurely. Cutrrently, system-python is waiting for the larger-scale restructuring, and hopefully wider/longer discussion.)
![](https://secure.gravatar.com/avatar/01aa7d6d4db83982a2f6dd363d0ee0f3.jpg?s=120&d=mm&r=g)
On Mar 1, 2019, at 02:41, Petr Viktorin
Currently, in RHEL/Fedora: - "sudo pip" installs to /usr/local/, RPM packages install to /usr/ - with "-I", the interpreter does not look into /usr/local/. AFAIK, Debian/Ubuntu have something very similar, and were first to do it.
Debuntu’s pip installs to the user path by default, and you have to do something explicit to install into the system Python. -Barry
participants (25)
-
Antoine Pitrou
-
Barry Warsaw
-
Brett Cannon
-
Chris Barker
-
Chris Barker - NOAA Federal
-
Gregory P. Smith
-
Gustavo Carneiro
-
INADA Naoki
-
Jason Swails
-
Matthias Klose
-
Matěj Cepl
-
Nathaniel Smith
-
Neil Schemenauer
-
Neil Schemenauer
-
Nick Coghlan
-
Petr Viktorin
-
Sorin Sbarnea
-
Stephane Wirtel
-
Stephen J. Turnbull
-
Steve Dower
-
Steve Holden
-
Steven D'Aprano
-
Terry Reedy
-
Toshio Kuratomi
-
Victor Stinner