Re: [Python-ideas] Looking for input to help with the pip situation
General summary so far ####################### This debate has been very civil so far and quite productive, so thanks to everyone involved. Several issues and solution proposals have been mentioned, so I will make a summary here. Issue 1: running Python is a different command different OS when several versions are installed ================================================================================================== Proposal A: ----------- Suffix Python executable on Windows like on Unix, so that people will type pythonX.X if they want a specify version. Pros: easy and discoverable. Cons: you need a lot of stuff in the system path. Proposal B: ------------ Provide the "py" command on Unix, so that people will type "py -x.x" Pros: no more issues with system path. Just add this command, and register python installations to be listed by "py". Cons: very few people know about "py", even on windows. This would need lots of communication. Harder to change Mac and majors Linux distros setup than just the next windows installer. Issue 2: there is no one and only one obvious way to do venv / pip =================================================================== Because pip and venv are tied to a Python version, there are numerous ways to call them. Just for pip, you have "pip", "py -x.x -m pip", "pythonx.x -m pip" and "pipx". Proposal A: ----------- Decide on one form, and promote it everywhere. Pros: simple. Cons: long and tedious process, with a huge inertia before all 3rd party to it. Would need a "PEP 8 for Python command". Requires "issue 1" to be solved before. Proposal B: ------------ Provide a pipenv-like behavior pipenv is a tool created by the author of "requests" that automatically "do the right thing" by default. The first time you install a package, it creates a virtualenv for the current directory (but don't store it in the current directory). It will then install all packages in there. "pipenv shell" starts a shell in the virtualenv. Dependencies are automatically dumped in a requirement files separating dev and prod. Pros: bypass the multiple installers issue by prompting you what version you wish the first time it creates a virtualenv. Solve the problem of admin rights and system path. Cons: lots of work. Either a new tool or breaking compat with pip. If debian didn't install pip by default, this may not help. Issue 3: pip and virtualenv are not available everywhere =================================================================== Proposal A: ------------ Make a mock "pip" commands for when it's not installed. venv already does that in some cases (see attachements), alhought it does give you the wrong command so it should be fixed. The gist of it: if the users try to run pip install while it's missing, it prints a warning explaining it's not installed and the command you can run to solve the problem plus a link to more information in the documentation. Pros: low friction to implement. Cons: requires to put up a hook somewhere so that each linux distro can input "the proper commands to use" for their system when they package python. More a workaround than a solution. Proposal B: ------------ Make pip and venv part of the standard and request debian that they provide it. Pros: straight forward. Cons: holy war in the making. ................. Can I have +1 and -1 from each of you on each point ? Should I start a new thread for each of them ? Favorite personnal combination ================================ 1 - So far, I like "providing py on unix" better, but I think it's too hard to do. So provide suffix on windows seems more achievable. So +1 for having pythonX.X on windows. 2 - Decide on one form, and promote it everywhere is again the simplest IMO. I love pipenv but the strings to pull to get something like this means it probably would not happen. If we stick to suffix for issue 1, this mean we can promote: pythonX.X -m pip pythonX.X -m venv Everywhere. In the docs. In the tutorials. In teaching classes. This will help a lot to remove all confusions and make the Python start up experience way easier. 3 - getting angry debian devs is probably not wise. The "mock" pip/venv command also plays very well with the 2 other solutions. It means we can promote everywhere: pythonX.X -m cmd It will work most of the time. And when it doesn't, give clear contextual steps on
On Tue, Nov 14, 2017 at 5:47 PM, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
If this would really cause such back-lash, how about: Make pip and venv part of the standard, but permit them to be omitted from "python3" as long as there is a "python3-full" package that will drag them in. Several other packages are split up like that. Pros: Only slightly less straight-forward, and hopefully incites only a small war. Cons: What does "full" really mean? Does it require a GUI subsystem, for instance (as a dep of tkinter)? Might just shift the problem. OTOH, the sorts of people who run non-graphical Linuxes are usually going to be comfortable installing python3-pip explicitly, so maybe it's doable. ChrisA
On Tue, Nov 14, 2017 at 6:38 PM, Chris Angelico <rosuav@gmail.com> wrote:
Oh, forgot to give a clear opinion: Even if the compromise isn't viable, I am still +1 on auto-installing pip. There's already a python3-minimal package for people who want "just the interpreter, please" installs. ChrisA
On Nov 14, 2017 02:39, "Chris Angelico" <rosuav@gmail.com> wrote: On Tue, Nov 14, 2017 at 5:47 PM, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Cons: What does "full" really mean? Does it require a GUI subsystem, for instance (as a dep of tkinter)? Might just shift the problem. The solution a lot of other Linux distributions use is to have a "python(2|3)-base" package that pulls in just the minimal runtime environment and have the regular "python(2|3)" package gives the full upstream python install. Things like tk bindings and pip tend to be separate packages the normal install depends on. "Minimal", if I understand it correctly, is the python(2|3) executable and any module not needing external dependencies, and "full" is everything upstream provides.
On 14 November 2017 at 16:47, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Con: we hope to have the problem resolved on the Linux distro side such that "python" typically means "python" by the time community support for Python 2 ends in 2020. Since Windows has gone the better part of two decades without version Python commands, adding them because we're impatient with the pace of change at the Linux distro level doesn't really make sense (especially when Linux holds such a small fraction of the non-phone client device market). Proposal B:
Proposal C: Make sure 'python' just works on a default Window user install, and treat any system where that doesn't work by default as a special case that needs troubleshooting. Pro: a solid troubleshooting guide will be useful regardless Con: mainstream Linux distros will still be a consistent exception for at least the next couple of years (and probably longer for the LTS variants)
This downside doesn't apply as long as the form we standardise on is "pip install module". That already works in a lot of contexts, and the cases where it doesn't work are typically ones where "python -m pip" may also be a problem. Given a solid troubleshooting guide for issue 1, we may also be able to add some implicit checks to pip that provide alerts for problematic configurations, like checking: >>> user_scheme = "posix_user" # or "nt_user" or "osx_framework_user" >>> sysconfig.get_path("scripts", scheme="posix_user") in os.getenv("PATH").split(os.pathsep) True The latter check returning false is a sign that the `PATH` configuration might be a problem.
This is essentially the path that conda takes, and it works well for folks that are willing to adopt conda's specific approach to managing your Python runtimes.
I don't think it's *that* bad. Debian does have support for weak dependencies, so the resolution here may be as simple as asking Matthias to add a Recommends from python3 to python3-venv so that ensurepip is present by default on typical workstation configurations, but can still be easily omitted from server configurations.
I think this is actually similar to the situation with 'py' on *nix systems: with Windows having never had these before, adding and promoting them would be at best ineffective, and at worst outright confusing (especially if "python" starts meaning Python 3 by default on Linux as well within the next few years).
We've been evaluating a potential pipenv tutorial for the Python Packaging User Guide, and while we're likely going to add it soon, it will be as a new "application dependency management" tutorial alongside the existing ones, rather than as a direct replacement for the pip-based "package installation" tutorial: https://github.com/pypa/python-packaging-user-guide/issues/394#issuecomment-...
Except it won't, because now you can't seamlessly update your class instructions to new versions of Python - you'll have to go through and change all the "X.Y" references to refer to the right version. Writing cross-platform scripts will also get even harder, as you wouldn't be able to say "I just need a version of Python, I don't really care which one" any more. Python feature releases are already disruptive enough due to filesystem layout changes and file format changes - we don't need to make them worse by encouraging people to embed the exact target version in their documentation and helper scripts.
3 - getting angry debian devs is probably not wise. The "mock" pip/venv command also plays very well with the 2 other solutions.
While I was a bit snarky about "python -m venv" being broken by default earlier in the thread, *please* don't take that as meaning that we can't collaborate constructively with the Debian folks. We can, and do, but the friction between language level package management and system level package management is one with a *long* history that we're all working towards resolving together (See [1] & [2] for write-ups of a couple of linux.conf.au talks I've given about the problem), as is the question of how best to handle the `/usr/bin/python` symlink.
The combination of: pip install package python -m venv already works in most cases, *except* apparently the critical one of "New Python user on Windows downloads the python.org installer and clicks through all the buttons without changing any settings". So I think the main near term step forward would be to convince Steve Dower (as the Windows installer maintainer) to change that default behaviour yet again for 3.7, and then work towards coming up with a solid environment troubleshooting guide to include on packaging.python.org. Cheers, Nick. [1] https://lwn.net/Articles/580399/ [2] https://lwn.net/Articles/711906/ -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi Nick, 2017-11-14 11:07 GMT+01:00 Nick Coghlan <ncoghlan@gmail.com>:
Perhaps I could sell you on the idea of a Windows "python3" executable, not as the New Official Way to do things, but rather as a pragmatic measure to make code from those Linux weirdos ;-) more likely to work on Windows. I would like to compare it with the strings/bytes-for-pathnames issue: the official recommendation is to use strings everywhere, problems with round-tripping arbitrary not-valid-UTF8 filenames on POSIX have been solved now. Still, POSIX people continued to use bytes, so a pragmatic change was to make so that bytes now also work reliably as pathnames under Windows. Similarly, even when all Linux distributions have switched to python==python3, people will probably still read and write tutorials with python3 in it, and perhaps we should accommodate that. Otherwise I am +1 on your proposal C: if anything this thread has made it clear that there is so much variety in third-party Python installers, not just Linux distributions, but also things like Anaconda, that it seems unreasonable to require that the official Python documentation covers all those situations. Stephan
On 15 November 2017 at 01:23, Stephan Houben <stephanh42@gmail.com> wrote:
Aye, I'm not opposed to adding pythonX and pythonX.Y binaries or symlinks to the Windows installers as a way to make Linux-assuming tutorials slightly more likely to work for Windows users. I'm only opposed to promoting that as the new preferred way of launching Python from the Windows command line, if for no other reason than if the release after Python 3.9 actually does end up being Python 4.0, then we only have around 5 more years of "the latest Python" and "the latest Python 3.x" being the same thing :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Tue, Nov 14, 2017 at 5:47 PM, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
If this would really cause such back-lash, how about: Make pip and venv part of the standard, but permit them to be omitted from "python3" as long as there is a "python3-full" package that will drag them in. Several other packages are split up like that. Pros: Only slightly less straight-forward, and hopefully incites only a small war. Cons: What does "full" really mean? Does it require a GUI subsystem, for instance (as a dep of tkinter)? Might just shift the problem. OTOH, the sorts of people who run non-graphical Linuxes are usually going to be comfortable installing python3-pip explicitly, so maybe it's doable. ChrisA
On Tue, Nov 14, 2017 at 6:38 PM, Chris Angelico <rosuav@gmail.com> wrote:
Oh, forgot to give a clear opinion: Even if the compromise isn't viable, I am still +1 on auto-installing pip. There's already a python3-minimal package for people who want "just the interpreter, please" installs. ChrisA
On Nov 14, 2017 02:39, "Chris Angelico" <rosuav@gmail.com> wrote: On Tue, Nov 14, 2017 at 5:47 PM, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Cons: What does "full" really mean? Does it require a GUI subsystem, for instance (as a dep of tkinter)? Might just shift the problem. The solution a lot of other Linux distributions use is to have a "python(2|3)-base" package that pulls in just the minimal runtime environment and have the regular "python(2|3)" package gives the full upstream python install. Things like tk bindings and pip tend to be separate packages the normal install depends on. "Minimal", if I understand it correctly, is the python(2|3) executable and any module not needing external dependencies, and "full" is everything upstream provides.
On 14 November 2017 at 16:47, Michel Desmoulin <desmoulinmichel@gmail.com> wrote:
Con: we hope to have the problem resolved on the Linux distro side such that "python" typically means "python" by the time community support for Python 2 ends in 2020. Since Windows has gone the better part of two decades without version Python commands, adding them because we're impatient with the pace of change at the Linux distro level doesn't really make sense (especially when Linux holds such a small fraction of the non-phone client device market). Proposal B:
Proposal C: Make sure 'python' just works on a default Window user install, and treat any system where that doesn't work by default as a special case that needs troubleshooting. Pro: a solid troubleshooting guide will be useful regardless Con: mainstream Linux distros will still be a consistent exception for at least the next couple of years (and probably longer for the LTS variants)
This downside doesn't apply as long as the form we standardise on is "pip install module". That already works in a lot of contexts, and the cases where it doesn't work are typically ones where "python -m pip" may also be a problem. Given a solid troubleshooting guide for issue 1, we may also be able to add some implicit checks to pip that provide alerts for problematic configurations, like checking: >>> user_scheme = "posix_user" # or "nt_user" or "osx_framework_user" >>> sysconfig.get_path("scripts", scheme="posix_user") in os.getenv("PATH").split(os.pathsep) True The latter check returning false is a sign that the `PATH` configuration might be a problem.
This is essentially the path that conda takes, and it works well for folks that are willing to adopt conda's specific approach to managing your Python runtimes.
I don't think it's *that* bad. Debian does have support for weak dependencies, so the resolution here may be as simple as asking Matthias to add a Recommends from python3 to python3-venv so that ensurepip is present by default on typical workstation configurations, but can still be easily omitted from server configurations.
I think this is actually similar to the situation with 'py' on *nix systems: with Windows having never had these before, adding and promoting them would be at best ineffective, and at worst outright confusing (especially if "python" starts meaning Python 3 by default on Linux as well within the next few years).
We've been evaluating a potential pipenv tutorial for the Python Packaging User Guide, and while we're likely going to add it soon, it will be as a new "application dependency management" tutorial alongside the existing ones, rather than as a direct replacement for the pip-based "package installation" tutorial: https://github.com/pypa/python-packaging-user-guide/issues/394#issuecomment-...
Except it won't, because now you can't seamlessly update your class instructions to new versions of Python - you'll have to go through and change all the "X.Y" references to refer to the right version. Writing cross-platform scripts will also get even harder, as you wouldn't be able to say "I just need a version of Python, I don't really care which one" any more. Python feature releases are already disruptive enough due to filesystem layout changes and file format changes - we don't need to make them worse by encouraging people to embed the exact target version in their documentation and helper scripts.
3 - getting angry debian devs is probably not wise. The "mock" pip/venv command also plays very well with the 2 other solutions.
While I was a bit snarky about "python -m venv" being broken by default earlier in the thread, *please* don't take that as meaning that we can't collaborate constructively with the Debian folks. We can, and do, but the friction between language level package management and system level package management is one with a *long* history that we're all working towards resolving together (See [1] & [2] for write-ups of a couple of linux.conf.au talks I've given about the problem), as is the question of how best to handle the `/usr/bin/python` symlink.
The combination of: pip install package python -m venv already works in most cases, *except* apparently the critical one of "New Python user on Windows downloads the python.org installer and clicks through all the buttons without changing any settings". So I think the main near term step forward would be to convince Steve Dower (as the Windows installer maintainer) to change that default behaviour yet again for 3.7, and then work towards coming up with a solid environment troubleshooting guide to include on packaging.python.org. Cheers, Nick. [1] https://lwn.net/Articles/580399/ [2] https://lwn.net/Articles/711906/ -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
Hi Nick, 2017-11-14 11:07 GMT+01:00 Nick Coghlan <ncoghlan@gmail.com>:
Perhaps I could sell you on the idea of a Windows "python3" executable, not as the New Official Way to do things, but rather as a pragmatic measure to make code from those Linux weirdos ;-) more likely to work on Windows. I would like to compare it with the strings/bytes-for-pathnames issue: the official recommendation is to use strings everywhere, problems with round-tripping arbitrary not-valid-UTF8 filenames on POSIX have been solved now. Still, POSIX people continued to use bytes, so a pragmatic change was to make so that bytes now also work reliably as pathnames under Windows. Similarly, even when all Linux distributions have switched to python==python3, people will probably still read and write tutorials with python3 in it, and perhaps we should accommodate that. Otherwise I am +1 on your proposal C: if anything this thread has made it clear that there is so much variety in third-party Python installers, not just Linux distributions, but also things like Anaconda, that it seems unreasonable to require that the official Python documentation covers all those situations. Stephan
On 15 November 2017 at 01:23, Stephan Houben <stephanh42@gmail.com> wrote:
Aye, I'm not opposed to adding pythonX and pythonX.Y binaries or symlinks to the Windows installers as a way to make Linux-assuming tutorials slightly more likely to work for Windows users. I'm only opposed to promoting that as the new preferred way of launching Python from the Windows command line, if for no other reason than if the release after Python 3.9 actually does end up being Python 4.0, then we only have around 5 more years of "the latest Python" and "the latest Python 3.x" being the same thing :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
participants (5)
-
Chris Angelico -
Michel Desmoulin -
Nick Coghlan -
Stephan Houben -
Todd