sudo pip install: install pip files into /usr/local on Linux?
Hi, pip is currently not well integrated on Linux: it conflicts with the system package manager like apt or rpm. When pip writes files into /usr, it can replace files written by the system package manager and so create different kind of issues. For example, if you check the system integry, you will likely see that some Python files have been modified. I would like to open a discussion to see how each Linux vendor handles the issue, and see if a common solution can be designed. Debian uses /usr for apt-get install and /usr/local for distutils and "sudo pip". Fedora decided to change pip to install files into /usr/local by default, instead of /usr, so "sudo pip install" doesn't replace files installed by dnf (Fedora package manager): https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe It gives you 3 main places to install Python code: /usr (managed by dnf), /usr/local (managed by sudo pip), $HOME/.local (managed by pip --user). Would it make sense to make the Fedora/Debian change upstream? At least, give an opt-in option for Linux vendors to use /usr/local? I propose to make the change upstream because there are still issues, and I don't want to be alone to have to fix them :-) It should be easier if we agree on a filesystem layout and an implementation, so we can collaborate on issues! Issues with the current Fedora implementation: (1) When Python is embedded in an application, there is an issue with the current heuristic to decide if /usr/local should be added to sys.path: https://bugzilla.redhat.com/show_bug.cgi?id=1532287 (2) On Fedora, "sudo pip install -U" currently removes old code from /usr and install the new one in /usr/local. We should leave /usr unchanged, since only dnf should touch /usr. https://bugzilla.redhat.com/show_bug.cgi?id=1550368#c24 The implementation is made of a single patch on the Python site module: https://src.fedoraproject.org/rpms/python3/blob/master/f/00251-change-user-i... -- There are two issues related to the "sudo pip" change, but they already exist when pip is installed in $HOME/.local: (3) Priority issue between PATH and PYTHONPATH directories. When the user runs "pip", the pip binary may come from /usr, /usr/local or $HOME/.local/bin, but the Python pip module ("import pip") may come from a different path. Which binary and which module should be used? Obvisouly, users can replace these two environment variables... (4) Related to (3). Running "pip" may run pip binary of one pip version, but pick the "pip" Python module of another pip version. For example, pip9 binary from /usr/bin/pip, but pip10 module from /usr/local. Fedora works around issue (4) with a downstream patch on pip: https://src.fedoraproject.org/rpms/python-pip/blob/master/f/pip9-allow-pip10... -- I don't well well how Linux distributions handle the issue with "sudo pip". So don't hesitate to correct me if I'm wrong :-) My goal is just to start a discussion about a common "upstream" solution. Victor
I think the obvious, if socially hard solution, is to make pip panic when it sees its being run as root (without, perhaps, a flag to tell pip "No, I really mean it, run as root"), and default to --user. It is not a good idea to install packages system wide with pip for reasons more than just clobbering apt/dnf installed packages. I still think the best idea for getting a python program to run system wide is either A: symlink from a inside a venv into something on $PATH, B: just set a shebang to the python in a venv, or C: bundle your application into a .deb or .rpm and use the system package manager to install it.
-----Original Message----- From: Victor Stinner <vstinner@redhat.com> Sent: Wednesday, May 23, 2018 11:22 AM To: distutils-sig@python.org Subject: [Distutils] sudo pip install: install pip files into /usr/local on Linux?
Hi,
pip is currently not well integrated on Linux: it conflicts with the system package manager like apt or rpm. When pip writes files into /usr, it can replace files written by the system package manager and so create different kind of issues. For example, if you check the system integry, you will likely see that some Python files have been modified.
I would like to open a discussion to see how each Linux vendor handles the issue, and see if a common solution can be designed.
Debian uses /usr for apt-get install and /usr/local for distutils and "sudo pip".
Fedora decided to change pip to install files into /usr/local by default, instead of /usr, so "sudo pip install" doesn't replace files installed by dnf (Fedora package manager): https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
It gives you 3 main places to install Python code: /usr (managed by dnf), /usr/local (managed by sudo pip), $HOME/.local (managed by pip --user).
Would it make sense to make the Fedora/Debian change upstream? At least, give an opt-in option for Linux vendors to use /usr/local?
I propose to make the change upstream because there are still issues, and I don't want to be alone to have to fix them :-) It should be easier if we agree on a filesystem layout and an implementation, so we can collaborate on issues!
Issues with the current Fedora implementation:
(1) When Python is embedded in an application, there is an issue with the current heuristic to decide if /usr/local should be added to sys.path:
https://bugzilla.redhat.com/show_bug.cgi?id=1532287
(2) On Fedora, "sudo pip install -U" currently removes old code from /usr and install the new one in /usr/local. We should leave /usr unchanged, since only dnf should touch /usr.
https://bugzilla.redhat.com/show_bug.cgi?id=1550368#c24
The implementation is made of a single patch on the Python site module:
https://src.fedoraproject.org/rpms/python3/blob/master/f/00251-change- user-install-location.patch
--
There are two issues related to the "sudo pip" change, but they already exist when pip is installed in $HOME/.local:
(3) Priority issue between PATH and PYTHONPATH directories.
When the user runs "pip", the pip binary may come from /usr, /usr/local or $HOME/.local/bin, but the Python pip module ("import pip") may come from a different path. Which binary and which module should be used?
Obvisouly, users can replace these two environment variables...
(4) Related to (3). Running "pip" may run pip binary of one pip version, but pick the "pip" Python module of another pip version.
For example, pip9 binary from /usr/bin/pip, but pip10 module from /usr/local.
Fedora works around issue (4) with a downstream patch on pip:
https://src.fedoraproject.org/rpms/python-pip/blob/master/f/pip9-allow- pip10-import.patch
--
I don't well well how Linux distributions handle the issue with "sudo pip". So don't hesitate to correct me if I'm wrong :-) My goal is just to start a discussion about a common "upstream" solution.
Victor -- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils- sig@python.org/message/OLGLHTSHLEPLHUTTVNU6L5QFTMNFIB6Z/
On Wednesday, May 23, 2018, Alex Walters <tritium-list@sdamon.com> wrote:
I think the obvious, if socially hard solution, is to make pip panic when it sees its being run as root (without, perhaps, a flag to tell pip "No, I really mean it, run as root"), and default to --user.
Maybe default to --user if not in a VIRTUAL_ENV (or a conda env).
It is not a good idea to install packages system wide with pip for reasons more than just clobbering apt/dnf installed packages. I still think the best idea for getting a python program to run system wide is either A: symlink from a inside a venv into something on $PATH, B: just set a shebang to the python in a venv, or C: bundle your application into a .deb or .rpm and use the system package manager to install it.
Pip shouldn't be run as root. That being said, installing files as user:group root is less insecure than making them user-writeable; so long as nothing is setuid is root. In most cases, it's ill-advised to allow apps to overwrite themselves and/or their dependencies.
-----Original Message----- From: Victor Stinner <vstinner@redhat.com> Sent: Wednesday, May 23, 2018 11:22 AM To: distutils-sig@python.org Subject: [Distutils] sudo pip install: install pip files into /usr/local on Linux?
Hi,
pip is currently not well integrated on Linux: it conflicts with the system package manager like apt or rpm. When pip writes files into /usr, it can replace files written by the system package manager and so create different kind of issues. For example, if you check the system integry, you will likely see that some Python files have been modified.
I would like to open a discussion to see how each Linux vendor handles the issue, and see if a common solution can be designed.
Debian uses /usr for apt-get install and /usr/local for distutils and "sudo pip".
Fedora decided to change pip to install files into /usr/local by default, instead of /usr, so "sudo pip install" doesn't replace files installed by dnf (Fedora package manager): https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
It gives you 3 main places to install Python code: /usr (managed by dnf), /usr/local (managed by sudo pip), $HOME/.local (managed by pip --user).
Would it make sense to make the Fedora/Debian change upstream? At least, give an opt-in option for Linux vendors to use /usr/local?
I propose to make the change upstream because there are still issues, and I don't want to be alone to have to fix them :-) It should be easier if we agree on a filesystem layout and an implementation, so we can collaborate on issues!
Issues with the current Fedora implementation:
(1) When Python is embedded in an application, there is an issue with the current heuristic to decide if /usr/local should be added to sys.path:
https://bugzilla.redhat.com/show_bug.cgi?id=1532287
(2) On Fedora, "sudo pip install -U" currently removes old code from /usr and install the new one in /usr/local. We should leave /usr unchanged, since only dnf should touch /usr.
https://bugzilla.redhat.com/show_bug.cgi?id=1550368#c24
The implementation is made of a single patch on the Python site module:
https://src.fedoraproject.org/rpms/python3/blob/master/f/00251-change- user-install-location.patch
--
There are two issues related to the "sudo pip" change, but they already exist when pip is installed in $HOME/.local:
(3) Priority issue between PATH and PYTHONPATH directories.
When the user runs "pip", the pip binary may come from /usr, /usr/local or $HOME/.local/bin, but the Python pip module ("import pip") may come from a different path. Which binary and which module should be used?
Obvisouly, users can replace these two environment variables...
(4) Related to (3). Running "pip" may run pip binary of one pip version, but pick the "pip" Python module of another pip version.
For example, pip9 binary from /usr/bin/pip, but pip10 module from /usr/local.
Fedora works around issue (4) with a downstream patch on pip:
https://src.fedoraproject.org/rpms/python-pip/blob/master/f/pip9-allow- pip10-import.patch
--
I don't well well how Linux distributions handle the issue with "sudo pip". So don't hesitate to correct me if I'm wrong :-) My goal is just to start a discussion about a common "upstream" solution.
Victor -- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils- sig@python.org/message/OLGLHTSHLEPLHUTTVNU6L5QFTMNFIB6Z/ -- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/ archives/list/distutils-sig@python.org/message/ I2F6SW6MVFTJFQNTQCB2HTZC7QOT7GBE/
From: Wes Turner <wes.turner@gmail.com> Sent: Wednesday, May 23, 2018 4:37 PM To: Alex Walters <tritium-list@sdamon.com> Cc: Victor Stinner <vstinner@redhat.com>; distutils-sig@python.org Subject: Re: [Distutils] sudo pip install: install pip files into /usr/local on Linux? On Wednesday, May 23, 2018, Alex Walters <tritium-list@sdamon.com <mailto:tritium-list@sdamon.com> > wrote: I think the obvious, if socially hard solution, is to make pip panic when it sees its being run as root (without, perhaps, a flag to tell pip "No, I really mean it, run as root"), and default to --user. Maybe default to --user if not in a VIRTUAL_ENV (or a conda env). Yes, of course. The assumption I made is that the behavior being discussed is when pip is not being run in a virtualenv. Pip should default to the most isolated environment available to it (venv, followed by user) It is not a good idea to install packages system wide with pip for reasons more than just clobbering apt/dnf installed packages. I still think the best idea for getting a python program to run system wide is either A: symlink from a inside a venv into something on $PATH, B: just set a shebang to the python in a venv, or C: bundle your application into a .deb or .rpm and use the system package manager to install it. Pip shouldn't be run as root. That being said, installing files as user:group root is less insecure than making them user-writeable; so long as nothing is setuid is root. Until there is something stopping me from uploading an sdist with malicious code in setup.py to pypi, running pip as a root is a bad idea even without user-writeable or setuid. In most cases, it's ill-advised to allow apps to overwrite themselves and/or their dependencies.
-----Original Message----- From: Victor Stinner <vstinner@redhat.com <mailto:vstinner@redhat.com> > Sent: Wednesday, May 23, 2018 11:22 AM To: distutils-sig@python.org <mailto:distutils-sig@python.org> Subject: [Distutils] sudo pip install: install pip files into /usr/local on Linux?
Hi,
pip is currently not well integrated on Linux: it conflicts with the system package manager like apt or rpm. When pip writes files into /usr, it can replace files written by the system package manager and so create different kind of issues. For example, if you check the system integry, you will likely see that some Python files have been modified.
I would like to open a discussion to see how each Linux vendor handles the issue, and see if a common solution can be designed.
Debian uses /usr for apt-get install and /usr/local for distutils and "sudo pip".
Fedora decided to change pip to install files into /usr/local by default, instead of /usr, so "sudo pip install" doesn't replace files installed by dnf (Fedora package manager): https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
It gives you 3 main places to install Python code: /usr (managed by dnf), /usr/local (managed by sudo pip), $HOME/.local (managed by pip --user).
Would it make sense to make the Fedora/Debian change upstream? At least, give an opt-in option for Linux vendors to use /usr/local?
I propose to make the change upstream because there are still issues, and I don't want to be alone to have to fix them :-) It should be easier if we agree on a filesystem layout and an implementation, so we can collaborate on issues!
Issues with the current Fedora implementation:
(1) When Python is embedded in an application, there is an issue with the current heuristic to decide if /usr/local should be added to sys.path:
https://bugzilla.redhat.com/show_bug.cgi?id=1532287
(2) On Fedora, "sudo pip install -U" currently removes old code from /usr and install the new one in /usr/local. We should leave /usr unchanged, since only dnf should touch /usr.
https://bugzilla.redhat.com/show_bug.cgi?id=1550368#c24
The implementation is made of a single patch on the Python site module:
https://src.fedoraproject.org/rpms/python3/blob/master/f/00251-change- user-install-location.patch
--
There are two issues related to the "sudo pip" change, but they already exist when pip is installed in $HOME/.local:
(3) Priority issue between PATH and PYTHONPATH directories.
When the user runs "pip", the pip binary may come from /usr, /usr/local or $HOME/.local/bin, but the Python pip module ("import pip") may come from a different path. Which binary and which module should be used?
Obvisouly, users can replace these two environment variables...
(4) Related to (3). Running "pip" may run pip binary of one pip version, but pick the "pip" Python module of another pip version.
For example, pip9 binary from /usr/bin/pip, but pip10 module from /usr/local.
Fedora works around issue (4) with a downstream patch on pip:
https://src.fedoraproject.org/rpms/python-pip/blob/master/f/pip9-allow- pip10-import.patch
--
I don't well well how Linux distributions handle the issue with "sudo pip". So don't hesitate to correct me if I'm wrong :-) My goal is just to start a discussion about a common "upstream" solution.
Victor -- Distutils-SIG mailing list distutils-sig@python.org <mailto:distutils-sig@python.org> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils- sig@python.org/message/OLGLHTSHLEPLHUTTVNU6L5QFTMNFIB6Z/ <http://sig@python.org/message/OLGLHTSHLEPLHUTTVNU6L5QFTMNFIB6Z/> -- Distutils-SIG mailing list distutils-sig@python.org <mailto:distutils-sig@python.org> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/I...
As an user, I want to use "sudo pip install" because packages installed in /usr (or /usr/local) are accessible without having to touch PYTHONPATH: the install directory is part of the default sys.path. Steve Dower also proposed the idea of a "default virtual environment" somewhere in the $HOME directory which would be in the default sys.path. Victor 2018-05-23 21:47 GMT+02:00 Alex Walters <tritium-list@sdamon.com>:
I think the obvious, if socially hard solution, is to make pip panic when it sees its being run as root (without, perhaps, a flag to tell pip "No, I really mean it, run as root"), and default to --user. It is not a good idea to install packages system wide with pip for reasons more than just clobbering apt/dnf installed packages. I still think the best idea for getting a python program to run system wide is either A: symlink from a inside a venv into something on $PATH, B: just set a shebang to the python in a venv, or C: bundle your application into a .deb or .rpm and use the system package manager to install it.
-----Original Message----- From: Victor Stinner <vstinner@redhat.com> Sent: Wednesday, May 23, 2018 11:22 AM To: distutils-sig@python.org Subject: [Distutils] sudo pip install: install pip files into /usr/local on Linux?
Hi,
pip is currently not well integrated on Linux: it conflicts with the system package manager like apt or rpm. When pip writes files into /usr, it can replace files written by the system package manager and so create different kind of issues. For example, if you check the system integry, you will likely see that some Python files have been modified.
I would like to open a discussion to see how each Linux vendor handles the issue, and see if a common solution can be designed.
Debian uses /usr for apt-get install and /usr/local for distutils and "sudo pip".
Fedora decided to change pip to install files into /usr/local by default, instead of /usr, so "sudo pip install" doesn't replace files installed by dnf (Fedora package manager): https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
It gives you 3 main places to install Python code: /usr (managed by dnf), /usr/local (managed by sudo pip), $HOME/.local (managed by pip --user).
Would it make sense to make the Fedora/Debian change upstream? At least, give an opt-in option for Linux vendors to use /usr/local?
I propose to make the change upstream because there are still issues, and I don't want to be alone to have to fix them :-) It should be easier if we agree on a filesystem layout and an implementation, so we can collaborate on issues!
Issues with the current Fedora implementation:
(1) When Python is embedded in an application, there is an issue with the current heuristic to decide if /usr/local should be added to sys.path:
https://bugzilla.redhat.com/show_bug.cgi?id=1532287
(2) On Fedora, "sudo pip install -U" currently removes old code from /usr and install the new one in /usr/local. We should leave /usr unchanged, since only dnf should touch /usr.
https://bugzilla.redhat.com/show_bug.cgi?id=1550368#c24
The implementation is made of a single patch on the Python site module:
https://src.fedoraproject.org/rpms/python3/blob/master/f/00251-change- user-install-location.patch
--
There are two issues related to the "sudo pip" change, but they already exist when pip is installed in $HOME/.local:
(3) Priority issue between PATH and PYTHONPATH directories.
When the user runs "pip", the pip binary may come from /usr, /usr/local or $HOME/.local/bin, but the Python pip module ("import pip") may come from a different path. Which binary and which module should be used?
Obvisouly, users can replace these two environment variables...
(4) Related to (3). Running "pip" may run pip binary of one pip version, but pick the "pip" Python module of another pip version.
For example, pip9 binary from /usr/bin/pip, but pip10 module from /usr/local.
Fedora works around issue (4) with a downstream patch on pip:
https://src.fedoraproject.org/rpms/python-pip/blob/master/f/pip9-allow- pip10-import.patch
--
I don't well well how Linux distributions handle the issue with "sudo pip". So don't hesitate to correct me if I'm wrong :-) My goal is just to start a discussion about a common "upstream" solution.
Victor -- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/archives/list/distutils- sig@python.org/message/OLGLHTSHLEPLHUTTVNU6L5QFTMNFIB6Z/
On Fri, May 25, 2018, at 5:11 PM, Victor Stinner wrote:
As an user, I want to use "sudo pip install" because packages installed in /usr (or /usr/local) are accessible without having to touch PYTHONPATH: the install directory is part of the default sys.path.
This is also true for "pip install --user", at least on the systems I'm familiar with. There are options to disable 'user site packages', but it's enabled by default. It's more annoying for scripts - on common Linux distributions, the user scripts location ~/.local/bin is not on PATH by default. Thomas
On Friday, May 25, 2018, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
On Fri, May 25, 2018, at 5:11 PM, Victor Stinner wrote:
As an user, I want to use "sudo pip install" because packages installed in /usr (or /usr/local) are accessible without having to touch PYTHONPATH: the install directory is part of the default sys.path.
This is also true for "pip install --user", at least on the systems I'm familiar with. There are options to disable 'user site packages', but it's enabled by default.
It's more annoying for scripts - on common Linux distributions, the user scripts location ~/.local/bin is not on PATH by default.
~/.local/bin is user-writeable. If ~/.local was on PATH or by default, it could potentially preempt/modify the behavior of system libraries and binaries; which is a security risk. ~/.local/bin/bash could be wrapper script that logs commands, for example. If it's first in the path (as e.g. Homebrew does, IIRC), it's run when bare `bash` or `/usr/bin/env bash` are executed. pipsi creates isolated virtualenvs per-install which are isolated from other library installs, but each env then must be upgraded separately.
Thomas -- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/ archives/list/distutils-sig@python.org/message/ 2XEH7R56Y63D72BZBAWRRJ33HPIKCWZH/
On Friday, May 25, 2018, Wes Turner <wes.turner@gmail.com> wrote:
On Friday, May 25, 2018, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
On Fri, May 25, 2018, at 5:11 PM, Victor Stinner wrote:
As an user, I want to use "sudo pip install" because packages installed in /usr (or /usr/local) are accessible without having to touch PYTHONPATH: the install directory is part of the default sys.path.
This is also true for "pip install --user", at least on the systems I'm familiar with. There are options to disable 'user site packages', but it's enabled by default.
It's more annoying for scripts - on common Linux distributions, the user scripts location ~/.local/bin is not on PATH by default.
~/.local/bin is user-writeable. If ~/.local was on PATH or by default, it could potentially preempt/modify the behavior of system libraries and binaries; which is a security risk.
*If ~/.local was on $PATH or sys.path by default
~/.local/bin/bash could be wrapper script that logs commands, for example. If it's first in the path (as e.g. Homebrew does, IIRC), it's run when bare `bash` or `/usr/bin/env bash` are executed.
pipsi creates isolated virtualenvs per-install which are isolated from other library installs, but each env then must be upgraded separately.
Thomas -- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/ar chives/list/distutils-sig@python.org/message/2XEH7R56Y63D72B ZBAWRRJ33HPIKCWZH/
~/.local/bin is user-writeable. If ~/.local was on PATH or by default, it could potentially preempt/modify the behavior of system libraries and binaries; which is a security risk. I've heard this argument before, and it doesn't stand up, because files
On Fri, May 25, 2018, at 6:58 PM, Wes Turner wrote: like .profile and .bashrc are user writable, and you can use those to add a directory to PATH (among many other things). You may be able to come up with some corner case where it's possible to modify ~/.local/bin but not ~/.profile, but it's pretty clear that this is a post-hoc rationalisation, not a real reason. It's like that, I strongly suspect, just because that's how it's been forever, and the people who are inconvenienced by it know how to work around it. Thomas
On Fri, May 25, 2018 at 1:07 PM, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
On Fri, May 25, 2018, at 6:58 PM, Wes Turner wrote:
~/.local/bin is user-writeable. If ~/.local was on PATH or by default, it could potentially preempt/modify the behavior of system libraries and binaries; which is a security risk.
I've heard this argument before, and it doesn't stand up, because files like .profile and .bashrc are user writable, and you can use those to add a directory to PATH (among many other things). You may be able to come up with some corner case where it's possible to modify ~/.local/bin but not ~/.profile, but it's pretty clear that this is a post-hoc rationalisation, not a real reason.
I think there's a critical distinction to make here. If some tool modifies .profile or .bashrc, it's a big deal. You're playing with the user's experience in a major way. Because of that, tools (such as Anaconda's installer) generally make it an explicitly confirmed change. If .local is on PATH by default (especially up front), people may not understand the shadowing implications of installing some package with pip. One other point: users of Anaconda already have trouble with --user installs with pip. Users who run into trouble with their installation usually try to remove and reinstall Anaconda. Because .local is not managed by Anaconda, those users get very confused why the uninstall/reinstall doesn't fix their issue. Having out-of-sysroot locations for packages, such as /usr/local or .local definitely requires more from users. Is it worth it? Maybe. We should be mindful of ways that we can help users understand the locations that they may need to search/clean to resolve problems. Not everyone is advanced enough to look at sys.path or to do something like "troublesome_package.__file__" to figure out why things are misbehaving.
It's like that, I strongly suspect, just because that's how it's been forever, and the people who are inconvenienced by it know how to work around it.
Thomas
-- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/ archives/list/distutils-sig@python.org/message/ YAIWB2P3UUBMYU2LJYU7A72P4RTDP64T/
For secured production applications, the user running the app should not be able to preempt system binaries or overwrite user-writeable config in $HOME. We tend to compromise on the side of developer-friendliness over secure by default. Is pip a tool for development or a tool for production deployments? Pip is definitely a tool for development. There are lots of packaging systems for production deployments which can handle e.g. file permissions and modifying /etc config. Pip is sometimes a tool used for production deployment. On Friday, May 25, 2018, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
On Fri, May 25, 2018, at 6:58 PM, Wes Turner wrote:
~/.local/bin is user-writeable. If ~/.local was on PATH or by default, it could potentially preempt/modify the behavior of system libraries and binaries; which is a security risk.
I've heard this argument before, and it doesn't stand up, because files like .profile and .bashrc are user writable, and you can use those to add a directory to PATH (among many other things). You may be able to come up with some corner case where it's possible to modify ~/.local/bin but not ~/.profile, but it's pretty clear that this is a post-hoc rationalisation, not a real reason.
It's like that, I strongly suspect, just because that's how it's been forever, and the people who are inconvenienced by it know how to work around it.
Thomas
On Sat., 26 May 2018, 4:25 am Donald Stufft, <donald@stufft.io> wrote:
On May 25, 2018, at 12:44 PM, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
It's more annoying for scripts - on common Linux distributions, the user scripts location ~/.local/bin is not on PATH by default.
It’s on $PATH by default in Fedora I think.
Yep, after all the other entries. I actually think Debian/Ubuntu may have changed their default set up as well somewhere along the line, but even if they did, it potentially wouldn't change the settings for existing user profiles (depending on exactly how they implemented it). Cheers, Nick.
On 26.05.2018 14:59, Nick Coghlan wrote:
On Sat., 26 May 2018, 4:25 am Donald Stufft, <donald@stufft.io> wrote:
On May 25, 2018, at 12:44 PM, Thomas Kluyver <thomas@kluyver.me.uk> wrote:
It's more annoying for scripts - on common Linux distributions, the user scripts location ~/.local/bin is not on PATH by default.
It’s on $PATH by default in Fedora I think.
Yep, after all the other entries. I actually think Debian/Ubuntu may have changed their default set up as well somewhere along the line, but even if they did, it potentially wouldn't change the settings for existing user profiles (depending on exactly how they implemented it).
It's on the path by default in Debian and Ubuntu, only for new users (~/.profile).
On Thursday, May 31, 2018, Matthias Klose <doko@debian.org> wrote:
On 26.05.2018 14:59, Nick Coghlan wrote:
On Sat., 26 May 2018, 4:25 am Donald Stufft, <donald@stufft.io> wrote:
On May 25, 2018, at 12:44 PM, Thomas Kluyver <thomas@kluyver.me.uk>
wrote:
It's more annoying for scripts - on common Linux distributions, the user scripts location ~/.local/bin is not on PATH by default.
It’s on $PATH by default in Fedora I think.
Yep, after all the other entries. I actually think Debian/Ubuntu may have changed their default set up as well somewhere along the line, but even if they did, it potentially wouldn't change the settings for existing user profiles (depending on exactly how they implemented it).
It's on the path by default in Debian and Ubuntu, only for new users (~/.profile).
I believe ~/.profile is copied from /etc/skel/.profile on most systems.
-- Distutils-SIG mailing list -- distutils-sig@python.org To unsubscribe send an email to distutils-sig-leave@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/ archives/list/distutils-sig@python.org/message/ GCDA3XQSGF5PFZXR3KTMTPUB67UBFUFW/
On 1 June 2018 at 02:11, Wes Turner <wes.turner@gmail.com> wrote:
On Thursday, May 31, 2018, Matthias Klose <doko@debian.org> wrote:
On 26.05.2018 14:59, Nick Coghlan wrote:
Yep, after all the other entries. I actually think Debian/Ubuntu may have changed their default set up as well somewhere along the line, but even if they did, it potentially wouldn't change the settings for existing user profiles (depending on exactly how they implemented it).
It's on the path by default in Debian and Ubuntu, only for new users (~/.profile).
I believe ~/.profile is copied from /etc/skel/.profile on most systems.
Right, but distro upgrades are now regularly reliable enough that folks may go for years without creating a fresh user profile for themselves. For me personally, even when I do set up a fresh machine with a clean Fedora install, I'll rsync my old home directory over the top of the new one. This kind of thing means that even when distros change their default settings, a *lot* of users will still have the old default. Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
I like to be able to diff the dotfiles in my home directory. Sometimes, just copying from one machine to another clobbers useful/necessary machine defaults (eg. $PATH). I don't want to add my entire home directory to version control because slow, so I wrote a shell script to copy existing files out of the way and then symlink from e.g. ~/.bashrc to ~/-dotfiles/etc/bash. That way, I can 'cdd[otfiles]' (cd $__DOTFILES) and 'git status etc/.bashrc' and 'git diff'. It's a shell script in order to avoid having a dependency on python on eg mobile/embedded systems, but it does the job. I'm sure there are various other approaches to managing dotfiles within version control. https://github.com/westurner/dotfiles/blob/develop/scripts/bootstrap_dotfile... Is being able to diff shell configuration a security advantage? You'd need to be pretty sophisticated to be making changes on disk; such as dropping a script or a binary with execute permissions into a directory at the top of the $PATH. On Friday, June 1, 2018, Nick Coghlan <ncoghlan@gmail.com> wrote:
On 1 June 2018 at 02:11, Wes Turner <wes.turner@gmail.com> wrote:
On Thursday, May 31, 2018, Matthias Klose <doko@debian.org> wrote:
On 26.05.2018 14:59, Nick Coghlan wrote:
Yep, after all the other entries. I actually think Debian/Ubuntu may have changed their default set up as well somewhere along the line, but even if they did, it potentially wouldn't change the settings for existing user profiles (depending on exactly how they implemented it).
It's on the path by default in Debian and Ubuntu, only for new users (~/.profile).
I believe ~/.profile is copied from /etc/skel/.profile on most systems.
Right, but distro upgrades are now regularly reliable enough that folks may go for years without creating a fresh user profile for themselves. For me personally, even when I do set up a fresh machine with a clean Fedora install, I'll rsync my old home directory over the top of the new one.
This kind of thing means that even when distros change their default settings, a *lot* of users will still have the old default.
Cheers, Nick.
-- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia
On Thu, May 31, 2018, at 4:39 PM, Matthias Klose wrote:
It's on the path by default in Debian and Ubuntu, only for new users (~/.profile).
Thanks, that's good to hear. I realise it's been a couple of years since I set up a fresh Linux installation. Upgrades are working too well!
Maybe `sudo pip install` should: - create a chroot && mkdir --prefix - drop privileges* - pip install - chown -R root:root - mv chroot/prefix/* prefix/ In most cases, the user does not need to run the (unreviewed, unsigned) code as root; neither should they run the (unreviewed, unsigned) setup.py as root. All a wheel needs to be installed is an unzip.* On Friday, May 25, 2018, Victor Stinner <vstinner@redhat.com> wrote:
As an user, I want to use "sudo pip install" because packages installed in /usr (or /usr/local) are accessible without having to touch PYTHONPATH: the install directory is part of the default sys.path.
Steve Dower also proposed the idea of a "default virtual environment" somewhere in the $HOME directory which would be in the default sys.path.
Victor
I think the obvious, if socially hard solution, is to make pip panic when it sees its being run as root (without, perhaps, a flag to tell pip "No, I really mean it, run as root"), and default to --user. It is not a good idea to install packages system wide with pip for reasons more than just clobbering apt/dnf installed packages. I still think the best idea for getting a python program to run system wide is either A: symlink from a inside a venv into something on $PATH, B: just set a shebang to the
2018-05-23 21:47 GMT+02:00 Alex Walters <tritium-list@sdamon.com>: python
in a venv, or C: bundle your application into a .deb or .rpm and use the system package manager to install it.
-----Original Message----- From: Victor Stinner <vstinner@redhat.com> Sent: Wednesday, May 23, 2018 11:22 AM To: distutils-sig@python.org Subject: [Distutils] sudo pip install: install pip files into /usr/local on Linux?
Hi,
pip is currently not well integrated on Linux: it conflicts with the system package manager like apt or rpm. When pip writes files into /usr, it can replace files written by the system package manager and so create different kind of issues. For example, if you check the system integry, you will likely see that some Python files have been modified.
I would like to open a discussion to see how each Linux vendor handles the issue, and see if a common solution can be designed.
Debian uses /usr for apt-get install and /usr/local for distutils and "sudo pip".
Fedora decided to change pip to install files into /usr/local by default, instead of /usr, so "sudo pip install" doesn't replace files installed by dnf (Fedora package manager): https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
It gives you 3 main places to install Python code: /usr (managed by dnf), /usr/local (managed by sudo pip), $HOME/.local (managed by pip --user).
Would it make sense to make the Fedora/Debian change upstream? At least, give an opt-in option for Linux vendors to use /usr/local?
I propose to make the change upstream because there are still issues, and I don't want to be alone to have to fix them :-) It should be easier if we agree on a filesystem layout and an implementation, so we can collaborate on issues!
Issues with the current Fedora implementation:
(1) When Python is embedded in an application, there is an issue with the current heuristic to decide if /usr/local should be added to sys.path:
https://bugzilla.redhat.com/show_bug.cgi?id=1532287
(2) On Fedora, "sudo pip install -U" currently removes old code from /usr and install the new one in /usr/local. We should leave /usr unchanged, since only dnf should touch /usr.
https://bugzilla.redhat.com/show_bug.cgi?id=1550368#c24
The implementation is made of a single patch on the Python site module:
https://src.fedoraproject.org/rpms/python3/blob/master/f/00251-change- user-install-location.patch
--
There are two issues related to the "sudo pip" change, but they already exist when pip is installed in $HOME/.local:
(3) Priority issue between PATH and PYTHONPATH directories.
When the user runs "pip", the pip binary may come from /usr, /usr/local or $HOME/.local/bin, but the Python pip module ("import pip") may come from a different path. Which binary and which module should be used?
Obvisouly, users can replace these two environment variables...
(4) Related to (3). Running "pip" may run pip binary of one pip version, but pick the "pip" Python module of another pip version.
For example, pip9 binary from /usr/bin/pip, but pip10 module from /usr/local.
Fedora works around issue (4) with a downstream patch on pip:
https://src.fedoraproject.org/rpms/python-pip/blob/master/f/pip9-allow- pip10-import.patch
--
I don't well well how Linux distributions handle the issue with "sudo pip". So don't hesitate to correct me if I'm wrong :-) My goal is just to start a discussion about a common "upstream" solution.
Victor -- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/ archives/list/distutils- sig@python.org/message/OLGLHTSHLEPLHUTTVNU6L5QFTMNFIB6Z/
-- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/ archives/list/distutils-sig@python.org/message/ 3FIFTWKLXC6Y6QI5XI7ZA2J2XEYS4A6J/
On 25May2018 0911, Victor Stinner wrote:
As an user, I want to use "sudo pip install" because packages installed in /usr (or /usr/local) are accessible without having to touch PYTHONPATH: the install directory is part of the default sys.path.
Steve Dower also proposed the idea of a "default virtual environment" somewhere in the $HOME directory which would be in the default sys.path.
For anyone who wants to follow along with this proposal (which barely resembles the brief description here :) ), we've been working on it on Kushal's github repo: https://github.com/kushaldas/peps/pull/1/files Clarifying comments/questions are welcomed. (Apologies to anyone who doesn't have a github account yet, but it seems like it will be getting increasingly harder to contribute to Python in any way without one, so you may not have much choice...) Cheers, Steve
Hi,
pip is currently not well integrated on Linux: it conflicts with the system package manager like apt or rpm. When pip writes files into /usr, it can replace files written by the system package manager and so create different kind of issues. For example, if you check the system integry, you will likely see that some Python files have been modified.
I would like to open a discussion to see how each Linux vendor handles the issue, and see if a common solution can be designed.
Debian uses /usr for apt-get install and /usr/local for distutils and "sudo pip".
Fedora decided to change pip to install files into /usr/local by default, instead of /usr, so "sudo pip install" doesn't replace files installed by dnf (Fedora package manager): https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
It gives you 3 main places to install Python code: /usr (managed by dnf), /usr/local (managed by sudo pip), $HOME/.local (managed by pip --user).
Would it make sense to make the Fedora/Debian change upstream? At least, give an opt-in option for Linux vendors to use /usr/local?
I propose to make the change upstream because there are still issues, and I don't want to be alone to have to fix them :-) It should be easier if we agree on a filesystem layout and an implementation, so we can collaborate on issues!
Issues with the current Fedora implementation:
(1) When Python is embedded in an application, there is an issue with the current heuristic to decide if /usr/local should be added to sys.path:
https://bugzilla.redhat.com/show_bug.cgi?id=1532287
(2) On Fedora, "sudo pip install -U" currently removes old code from /usr and install the new one in /usr/local. We should leave /usr unchanged, since only dnf should touch /usr.
https://bugzilla.redhat.com/show_bug.cgi?id=1550368#c24
The implementation is made of a single patch on the Python site module:
https://src.fedoraproject.org/rpms/python3/blob/master/f/ 00251-change-user-install-location.patch
--
There are two issues related to the "sudo pip" change, but they already exist when pip is installed in $HOME/.local:
(3) Priority issue between PATH and PYTHONPATH directories.
When the user runs "pip", the pip binary may come from /usr, /usr/local or $HOME/.local/bin, but the Python pip module ("import pip") may come from a different path. Which binary and which module should be used?
Obvisouly, users can replace these two environment variables...
(4) Related to (3). Running "pip" may run pip binary of one pip version, but pick the "pip" Python module of another pip version.
For example, pip9 binary from /usr/bin/pip, but pip10 module from /usr/local.
Fedora works around issue (4) with a downstream patch on pip:
https://src.fedoraproject.org/rpms/python-pip/blob/master/f/
Thanks for starting this discussion, Victor! This topic is something we're very interested in at Anaconda. I'd like to generalize the problem statement to the question of "how can we make pip behave well when it is sharing package management with something else? Similarly, how can we make the something else behave well with pip?" We all share the pain of trying to have two package managers effectively manage the same space. The alternate-folder-for-pip approach is a good idea, but ultimately has issues that you pointed out. After several great conversations with many people at PyCon last week, I came to the conclusion that conda and pip probably won't ever interoperate very well. In order for them to do so, conda must respect all of pip's constraints during its solving of dependencies, and likewise, pip must respect conda's constraints. We are investigating the first of those options and having some promising initial results, but the inverse is not something that seems feasible. It amounts to pip having a solver that is at least as good as the best supported package manager, and pip learning about *all* of any other package managers that it claims to be compatible with. That doesn't seem like a viable project. Ultimately, I believe a better approach is for the PyPA to define a minimal set of functionality and interfaces to PyPI that any package manager claiming to manage python packages must implement. Pip can be a reference implementation of that specification. Any distributor (Red Hat, Canonical, Homebrew, Anaconda) could then have their own implementations that use their solvers, but also can install software from PyPI at user's request, or as a fall-through when a native package is unavailable. User interface could be unified by having "pip" on distributions be a wrapper around the native package manager, matching the exact minimal behavior of pip. The same kind of approach may also be good for virtual environments, but it seems like there's less contention there. Conda is different enough from virtualenv that we get some friction, but I think and hope we can smooth that out over time. Best, Michael On Wednesday, May 23, 2018, Victor Stinner <vstinner@redhat.com> wrote: pip9-allow-pip10-import.patch
--
I don't well well how Linux distributions handle the issue with "sudo pip". So don't hesitate to correct me if I'm wrong :-) My goal is just to start a discussion about a common "upstream" solution.
Victor -- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/
archives/list/distutils-sig@python.org/message/ OLGLHTSHLEPLHUTTVNU6L5QFTMNFIB6Z/
On Wednesday, May 23, 2018, Michael Sarahan <msarahan@anaconda.com> wrote:
Thanks for starting this discussion, Victor! This topic is something we're very interested in at Anaconda.
I'd like to generalize the problem statement to the question of "how can we make pip behave well when it is sharing package management with something else? Similarly, how can we make the something else behave well with pip?"
We all share the pain of trying to have two package managers effectively manage the same space. The alternate-folder-for-pip approach is a good idea, but ultimately has issues that you pointed out.
After several great conversations with many people at PyCon last week, I came to the conclusion that conda and pip probably won't ever interoperate very well. In order for them to do so, conda must respect all of pip's constraints during its solving of dependencies, and likewise, pip must respect conda's constraints. We are investigating the first of those options and having some promising initial results, but the inverse is not something that seems feasible. It amounts to pip having a solver that is at least as good as the best supported package manager, and pip learning about *all* of any other package managers that it claims to be compatible with. That doesn't seem like a viable project.
What's the status on this? AFAIU, depsolving setup.py packages is blocked because it's necessary to execute setup.py to get the conditional requirements? https://www.pypa.io/en/latest/roadmap/#pip-dependency-resolution
Ultimately, I believe a better approach is for the PyPA to define a minimal set of functionality and interfaces to PyPI that any package manager claiming to manage python packages must implement. Pip can be a reference implementation of that specification. Any distributor (Red Hat, Canonical, Homebrew, Anaconda) could then have their own implementations that use their solvers, but also can install software from PyPI at user's request, or as a fall-through when a native package is unavailable.
Metadata compatibility and adapter registration could help solve for this; though there's no money and not much demand. #PEP426JSONLD
User interface could be unified by having "pip" on distributions be a wrapper around the native package manager, matching the exact minimal behavior of pip.
Would `sudo pip` then be the only way?
The same kind of approach may also be good for virtual environments, but it seems like there's less contention there. Conda is different enough from virtualenv that we get some friction, but I think and hope we can smooth that out over time.
`conda install pip` and `conda export -f environment.yml` seem to work okay (for Python, R, nodejs but not yet npm,)? Solving dependencies in a container is still the correct way to avoid cruft, IMHO.
Best,
Michael
Hi,
pip is currently not well integrated on Linux: it conflicts with the system package manager like apt or rpm. When pip writes files into /usr, it can replace files written by the system package manager and so create different kind of issues. For example, if you check the system integry, you will likely see that some Python files have been modified.
I would like to open a discussion to see how each Linux vendor handles the issue, and see if a common solution can be designed.
Debian uses /usr for apt-get install and /usr/local for distutils and "sudo pip".
Fedora decided to change pip to install files into /usr/local by default, instead of /usr, so "sudo pip install" doesn't replace files installed by dnf (Fedora package manager): https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
It gives you 3 main places to install Python code: /usr (managed by dnf), /usr/local (managed by sudo pip), $HOME/.local (managed by pip --user).
Would it make sense to make the Fedora/Debian change upstream? At least, give an opt-in option for Linux vendors to use /usr/local?
I propose to make the change upstream because there are still issues, and I don't want to be alone to have to fix them :-) It should be easier if we agree on a filesystem layout and an implementation, so we can collaborate on issues!
Issues with the current Fedora implementation:
(1) When Python is embedded in an application, there is an issue with the current heuristic to decide if /usr/local should be added to sys.path:
https://bugzilla.redhat.com/show_bug.cgi?id=1532287
(2) On Fedora, "sudo pip install -U" currently removes old code from /usr and install the new one in /usr/local. We should leave /usr unchanged, since only dnf should touch /usr.
https://bugzilla.redhat.com/show_bug.cgi?id=1550368#c24
The implementation is made of a single patch on the Python site module:
https://src.fedoraproject.org/rpms/python3/blob/master/f/002 51-change-user-install-location.patch
--
There are two issues related to the "sudo pip" change, but they already exist when pip is installed in $HOME/.local:
(3) Priority issue between PATH and PYTHONPATH directories.
When the user runs "pip", the pip binary may come from /usr, /usr/local or $HOME/.local/bin, but the Python pip module ("import pip") may come from a different path. Which binary and which module should be used?
Obvisouly, users can replace these two environment variables...
(4) Related to (3). Running "pip" may run pip binary of one pip version, but pick the "pip" Python module of another pip version.
For example, pip9 binary from /usr/bin/pip, but pip10 module from /usr/local.
Fedora works around issue (4) with a downstream patch on pip:
https://src.fedoraproject.org/rpms/python-pip/blob/master/f/
On Wednesday, May 23, 2018, Victor Stinner <vstinner@redhat.com> wrote: pip9-allow-pip10-import.patch
--
I don't well well how Linux distributions handle the issue with "sudo pip". So don't hesitate to correct me if I'm wrong :-) My goal is just to start a discussion about a common "upstream" solution.
Victor -- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/ar
chives/list/distutils-sig@python.org/message/OLGLHTSHLEPLHUT TVNU6L5QFTMNFIB6Z/
On Wed, May 23, 2018 at 3:45 PM, Wes Turner <wes.turner@gmail.com> wrote:
On Wednesday, May 23, 2018, Michael Sarahan <msarahan@anaconda.com> wrote:
Thanks for starting this discussion, Victor! This topic is something we're very interested in at Anaconda.
I'd like to generalize the problem statement to the question of "how can we make pip behave well when it is sharing package management with something else? Similarly, how can we make the something else behave well with pip?"
We all share the pain of trying to have two package managers effectively manage the same space. The alternate-folder-for-pip approach is a good idea, but ultimately has issues that you pointed out.
After several great conversations with many people at PyCon last week, I came to the conclusion that conda and pip probably won't ever interoperate very well. In order for them to do so, conda must respect all of pip's constraints during its solving of dependencies, and likewise, pip must respect conda's constraints. We are investigating the first of those options and having some promising initial results, but the inverse is not something that seems feasible. It amounts to pip having a solver that is at least as good as the best supported package manager, and pip learning about *all* of any other package managers that it claims to be compatible with. That doesn't seem like a viable project.
What's the status on this? AFAIU, depsolving setup.py packages is blocked because it's necessary to execute setup.py to get the conditional requirements?
Yes, executing setup.py is a blocker. Wheels are improving this. The main complicating factor is conditional or optional dependencies, as you say, and how to express or execute the branch logic required. setup.py need not be executed all the time - only enough to gather the metadata to feed an index. It's much more of a security concern than a time concern. A first rough approach that didn't handle optional or conditional requirements seems reasonable as a proof of concept.
https://www.pypa.io/en/latest/roadmap/#pip-dependency-resolution
Ultimately, I believe a better approach is for the PyPA to define a minimal set of functionality and interfaces to PyPI that any package manager claiming to manage python packages must implement. Pip can be a reference implementation of that specification. Any distributor (Red Hat, Canonical, Homebrew, Anaconda) could then have their own implementations that use their solvers, but also can install software from PyPI at user's request, or as a fall-through when a native package is unavailable.
Metadata compatibility and adapter registration could help solve for this; though there's no money and not much demand. #PEP426JSONLD
With adapter registration, what would the adapters be? I'm just not sure what pip's role should be. If you define interfaces for solving and installing/uninstalling/upgrading the solved-for set of packages, maybe that's enough? #PEP426JSONLD looks great, but do you see that as a glue between PyPI and pip, pip and other tools, or other tools and PyPI? I am impressed and encouraged by your research into the topic, and I'd like to know if there's a way that we can help with it if it would further our goals of having conda be able to either interop with pip happily or install directly from PyPI.
User interface could be unified by having "pip" on distributions be a wrapper around the native package manager, matching the exact minimal behavior of pip.
Would `sudo pip` then be the only way?
Heavens, no. I'm only concerned about the myriad blog posts out there that tell people to [sudo] pip install something. That is the user interface that exists and is most common. It is the universal command that might not be the right option, but by golly it's always an option. There is great value in having only one way to tell people to do things. I'm proposing that we make the underlying implementation of that user interface be vendor-specific. Vendors can and should keep their own interfaces to package management, too, because those are broader in scope than pip.
The same kind of approach may also be good for virtual environments, but it seems like there's less contention there. Conda is different enough from virtualenv that we get some friction, but I think and hope we can smooth that out over time.
`conda install pip` and `conda export -f environment.yml` seem to work okay (for Python, R, nodejs but not yet npm,)?
Solving dependencies in a container is still the correct way to avoid cruft, IMHO.
I agree, but sadly I can't force that on users. We (Anaconda and other distribution managers) still have to support people who are happily shooting themselves in the foot with "sudo pip" even when sudo is in no way necessary for a normal conda installation. Until we (the Python community) solve the package problems we have, we'll lose mindshare to less useful (IMHO) tools that are easier to manage packages with.
Best,
Michael
Hi,
pip is currently not well integrated on Linux: it conflicts with the system package manager like apt or rpm. When pip writes files into /usr, it can replace files written by the system package manager and so create different kind of issues. For example, if you check the system integry, you will likely see that some Python files have been modified.
I would like to open a discussion to see how each Linux vendor handles the issue, and see if a common solution can be designed.
Debian uses /usr for apt-get install and /usr/local for distutils and "sudo pip".
Fedora decided to change pip to install files into /usr/local by default, instead of /usr, so "sudo pip install" doesn't replace files installed by dnf (Fedora package manager): https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
It gives you 3 main places to install Python code: /usr (managed by dnf), /usr/local (managed by sudo pip), $HOME/.local (managed by pip --user).
Would it make sense to make the Fedora/Debian change upstream? At least, give an opt-in option for Linux vendors to use /usr/local?
I propose to make the change upstream because there are still issues, and I don't want to be alone to have to fix them :-) It should be easier if we agree on a filesystem layout and an implementation, so we can collaborate on issues!
Issues with the current Fedora implementation:
(1) When Python is embedded in an application, there is an issue with the current heuristic to decide if /usr/local should be added to sys.path:
https://bugzilla.redhat.com/show_bug.cgi?id=1532287
(2) On Fedora, "sudo pip install -U" currently removes old code from /usr and install the new one in /usr/local. We should leave /usr unchanged, since only dnf should touch /usr.
https://bugzilla.redhat.com/show_bug.cgi?id=1550368#c24
The implementation is made of a single patch on the Python site module:
https://src.fedoraproject.org/rpms/python3/blob/master/f/002 51-change-user-install-location.patch
--
There are two issues related to the "sudo pip" change, but they already exist when pip is installed in $HOME/.local:
(3) Priority issue between PATH and PYTHONPATH directories.
When the user runs "pip", the pip binary may come from /usr, /usr/local or $HOME/.local/bin, but the Python pip module ("import pip") may come from a different path. Which binary and which module should be used?
Obvisouly, users can replace these two environment variables...
(4) Related to (3). Running "pip" may run pip binary of one pip version, but pick the "pip" Python module of another pip version.
For example, pip9 binary from /usr/bin/pip, but pip10 module from /usr/local.
Fedora works around issue (4) with a downstream patch on pip:
https://src.fedoraproject.org/rpms/python-pip/blob/master/f/
On Wednesday, May 23, 2018, Victor Stinner <vstinner@redhat.com> wrote: pip9-allow-pip10-import.patch
--
I don't well well how Linux distributions handle the issue with "sudo pip". So don't hesitate to correct me if I'm wrong :-) My goal is just to start a discussion about a common "upstream" solution.
Victor -- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/ar
chives/list/distutils-sig@python.org/message/OLGLHTSHLEPLHUT TVNU6L5QFTMNFIB6Z/
On Wednesday, May 23, 2018, Michael Sarahan <msarahan@anaconda.com> wrote:
On Wed, May 23, 2018 at 3:45 PM, Wes Turner <wes.turner@gmail.com> wrote:
On Wednesday, May 23, 2018, Michael Sarahan <msarahan@anaconda.com> wrote:
Thanks for starting this discussion, Victor! This topic is something we're very interested in at Anaconda.
I'd like to generalize the problem statement to the question of "how can we make pip behave well when it is sharing package management with something else? Similarly, how can we make the something else behave well with pip?"
We all share the pain of trying to have two package managers effectively manage the same space. The alternate-folder-for-pip approach is a good idea, but ultimately has issues that you pointed out.
After several great conversations with many people at PyCon last week, I came to the conclusion that conda and pip probably won't ever interoperate very well. In order for them to do so, conda must respect all of pip's constraints during its solving of dependencies, and likewise, pip must respect conda's constraints. We are investigating the first of those options and having some promising initial results, but the inverse is not something that seems feasible. It amounts to pip having a solver that is at least as good as the best supported package manager, and pip learning about *all* of any other package managers that it claims to be compatible with. That doesn't seem like a viable project.
What's the status on this? AFAIU, depsolving setup.py packages is blocked because it's necessary to execute setup.py to get the conditional requirements?
Yes, executing setup.py is a blocker. Wheels are improving this. The main complicating factor is conditional or optional dependencies, as you say, and how to express or execute the branch logic required. setup.py need not be executed all the time - only enough to gather the metadata to feed an index. It's much more of a security concern than a time concern. A first rough approach that didn't handle optional or conditional requirements seems reasonable as a proof of concept.
https://www.pypa.io/en/latest/roadmap/#pip-dependency-resolution
Ultimately, I believe a better approach is for the PyPA to define a minimal set of functionality and interfaces to PyPI that any package manager claiming to manage python packages must implement. Pip can be a reference implementation of that specification. Any distributor (Red Hat, Canonical, Homebrew, Anaconda) could then have their own implementations that use their solvers, but also can install software from PyPI at user's request, or as a fall-through when a native package is unavailable.
Metadata compatibility and adapter registration could help solve for this; though there's no money and not much demand. #PEP426JSONLD
With adapter registration, what would the adapters be? I'm just not sure what pip's role should be. If you define interfaces for solving and installing/uninstalling/upgrading the solved-for set of packages, maybe that's enough?
Plugins/adapters/interface implementations. http://zopeinterface.readthedocs.io/en/latest/adapter.html Perhaps ironically and conveniently, the well-regarded pluggy system for plugins does not depend on setuptools entry_points: https://pluggy.readthedocs.io/en/latest/ Some way to avoid adding plugin/adapter registration overhead in site.py would be necessary.
#PEP426JSONLD looks great, but do you see that as a glue between PyPI and pip, pip and other tools, or other tools and PyPI? I am impressed and encouraged by your research into the topic, and I'd like to know if there's a way that we can help with it if it would further our goals of having conda be able to either interop with pip happily or install directly from PyPI.
I piggybacked onto PEP426 because if we were going to substantially change metadata, we might as well make it linked data; ideally with a cross-language spec that would unfortunately take years to get compliance from EVERY vendor for/with. Metadata interoperability wouldn't be strictly necessary to achieve what I think you're describing; but otherwise there'd be such a disjoint dependency graph that determining what we've installed here and where we need to be would be frustratingly complex.
User interface could be unified by having "pip" on distributions be a wrapper around the native package manager, matching the exact minimal behavior of pip.
Would `sudo pip` then be the only way?
Heavens, no. I'm only concerned about the myriad blog posts out there that tell people to [sudo] pip install something. That is the user interface that exists and is most common. It is the universal command that might not be the right option, but by golly it's always an option. There is great value in having only one way to tell people to do things. I'm proposing that we make the underlying implementation of that user interface be vendor-specific. Vendors can and should keep their own interfaces to package management, too, because those are broader in scope than pip.
I'm not sure whether it'd be easier to debug interleaved calls to various package managers. Or to explain why `pip install` didn't work on my machine. Though I have often wondered whether I need to do `conda skeleton` PyPI (or fork the conda-forge template), and then manually merge version changes stably. There would need to be a map between pypi package name, conda package names, (os, ver) package names; which I think I addressed in that smattering of notes on the PEP426JSONLD issue. That catalog-to-catalog mapping data would need to be hosted somewhere. Warehouse can easily serve JSON if it's defined in the package. Maybe a blockchain with per-project TUF signing keys, package checksums/signatures, and VCS GPG keyrings could also host package metadata and package name mappings someday.
The same kind of approach may also be good for virtual environments, but it seems like there's less contention there. Conda is different enough from virtualenv that we get some friction, but I think and hope we can smooth that out over time.
`conda install pip` and `conda export -f environment.yml` seem to work okay (for Python, R, nodejs but not yet npm,)?
Solving dependencies in a container is still the correct way to avoid cruft, IMHO.
I agree, but sadly I can't force that on users. We (Anaconda and other distribution managers) still have to support people who are happily shooting themselves in the foot with "sudo pip" even when sudo is in no way necessary for a normal conda installation. Until we (the Python community) solve the package problems we have, we'll lose mindshare to less useful (IMHO) tools that are easier to manage packages with.
A warning would be good. Is checking for `uid=0` sufficient? IDK what sort of timeline would be needed to requiring a CLI flag to bypass an error message when running pip as root. Containers often do this without specifying `USER nonroot` first in their Dockerfiles. And then setting read-only or other-user permissions does require root and so is better handled by actual OS package managers like FPM, IMHO.
Best,
Michael
Hi,
pip is currently not well integrated on Linux: it conflicts with the system package manager like apt or rpm. When pip writes files into /usr, it can replace files written by the system package manager and so create different kind of issues. For example, if you check the system integry, you will likely see that some Python files have been modified.
I would like to open a discussion to see how each Linux vendor handles the issue, and see if a common solution can be designed.
Debian uses /usr for apt-get install and /usr/local for distutils and "sudo pip".
Fedora decided to change pip to install files into /usr/local by default, instead of /usr, so "sudo pip install" doesn't replace files installed by dnf (Fedora package manager): https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
It gives you 3 main places to install Python code: /usr (managed by dnf), /usr/local (managed by sudo pip), $HOME/.local (managed by pip --user).
Would it make sense to make the Fedora/Debian change upstream? At least, give an opt-in option for Linux vendors to use /usr/local?
I propose to make the change upstream because there are still issues, and I don't want to be alone to have to fix them :-) It should be easier if we agree on a filesystem layout and an implementation, so we can collaborate on issues!
Issues with the current Fedora implementation:
(1) When Python is embedded in an application, there is an issue with the current heuristic to decide if /usr/local should be added to sys.path:
https://bugzilla.redhat.com/show_bug.cgi?id=1532287
(2) On Fedora, "sudo pip install -U" currently removes old code from /usr and install the new one in /usr/local. We should leave /usr unchanged, since only dnf should touch /usr.
https://bugzilla.redhat.com/show_bug.cgi?id=1550368#c24
The implementation is made of a single patch on the Python site module:
https://src.fedoraproject.org/rpms/python3/blob/master/f/002 51-change-user-install-location.patch
--
There are two issues related to the "sudo pip" change, but they already exist when pip is installed in $HOME/.local:
(3) Priority issue between PATH and PYTHONPATH directories.
When the user runs "pip", the pip binary may come from /usr, /usr/local or $HOME/.local/bin, but the Python pip module ("import pip") may come from a different path. Which binary and which module should be used?
Obvisouly, users can replace these two environment variables...
(4) Related to (3). Running "pip" may run pip binary of one pip version, but pick the "pip" Python module of another pip version.
For example, pip9 binary from /usr/bin/pip, but pip10 module from /usr/local.
Fedora works around issue (4) with a downstream patch on pip:
https://src.fedoraproject.org/rpms/python-pip/blob/master/f/
On Wednesday, May 23, 2018, Victor Stinner <vstinner@redhat.com> wrote: pip9-allow-pip10-import.patch
--
I don't well well how Linux distributions handle the issue with "sudo pip". So don't hesitate to correct me if I'm wrong :-) My goal is just to start a discussion about a common "upstream" solution.
Victor -- Distutils-SIG mailing list distutils-sig@python.org https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/ Message archived at https://mail.python.org/mm3/ar
chives/list/distutils-sig@python.org/message/OLGLHTSHLEPLHUT TVNU6L5QFTMNFIB6Z/
participants (9)
-
Alex Walters -
Donald Stufft -
Matthias Klose -
Michael Sarahan -
Nick Coghlan -
Steve Dower -
Thomas Kluyver -
Victor Stinner -
Wes Turner