Thanks for starting this discussion, Victor!  This topic is something we're very interested in at Anaconda.

I'd like to generalize the problem statement to the question of "how can we make pip behave well when it is sharing package management with something else?  Similarly, how can we make the something else behave well with pip?"

We all share the pain of trying to have two package managers effectively manage the same space.  The alternate-folder-for-pip approach is a good idea, but ultimately has issues that you pointed out.

After several great conversations with many people at PyCon last week, I came to the conclusion that conda and pip probably won't ever interoperate very well.  In order for them to do so, conda must respect all of pip's constraints during its solving of dependencies, and likewise, pip must respect conda's constraints.  We are investigating the first of those options and having some promising initial results, but the inverse is not something that seems feasible.  It amounts to pip having a solver that is at least as good as the best supported package manager, and pip learning about *all* of any other package managers that it claims to be compatible with.  That doesn't seem like a viable project.

Ultimately, I believe a better approach is for the PyPA to define a minimal set of functionality and interfaces to PyPI that any package manager claiming to manage python packages must implement.  Pip can be a reference implementation of that specification.  Any distributor (Red Hat, Canonical, Homebrew, Anaconda) could then have their own implementations that use their solvers, but also can install software from PyPI at user's request, or as a fall-through when a native package is unavailable.

User interface could be unified by having "pip" on distributions be a wrapper around the native package manager, matching the exact minimal behavior of pip.

The same kind of approach may also be good for virtual environments, but it seems like there's less contention there.  Conda is different enough from virtualenv that we get some friction, but I think and hope we can smooth that out over time.

Best, 

Michael

On Wednesday, May 23, 2018, Victor Stinner <vstinner@redhat.com> wrote:
> Hi,
>
> pip is currently not well integrated on Linux: it conflicts  with the
> system package manager like apt or rpm. When pip writes files  into
> /usr, it can replace files written by the system package manager  and
> so create different kind of issues. For example, if you check the
> system integry, you will likely see that some Python files have been
> modified.
>
> I would like to open a discussion to see how each Linux vendor handles
> the issue, and see if a common solution can be designed.
>
> Debian uses /usr for apt-get install and /usr/local for distutils and
> "sudo pip".
>
> Fedora  decided to change pip to install files into /usr/local by
> default,  instead of /usr, so "sudo pip install" doesn't replace files
> installed  by dnf (Fedora package manager):
> https://fedoraproject.org/wiki/Changes/Making_sudo_pip_safe
>
> It  gives you 3 main places to install Python code: /usr (managed by
> dnf),  /usr/local (managed by sudo pip), $HOME/.local (managed by pip
> --user).
>
> Would it make sense to make the Fedora/Debian change upstream? At
> least, give an opt-in option for Linux vendors to use /usr/local?
>
> I  propose to make the change upstream because there are still issues,
> and  I don't want to be alone to have to fix them :-) It should be
> easier if  we agree on a filesystem layout and an implementation, so
> we can  collaborate on issues!
>
>
> Issues with the current Fedora implementation:
>
> (1)  When Python is embedded in an application, there is an issue with
> the  current heuristic to decide if /usr/local should be added to
> sys.path:
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1532287
>
> (2)  On Fedora, "sudo pip install -U" currently removes old code from
> /usr  and install the new one in /usr/local. We should leave /usr
> unchanged,  since only dnf should touch /usr.
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1550368#c24
>
> The implementation is made of a single patch on the Python site module:
>
> https://src.fedoraproject.org/rpms/python3/blob/master/f/00251-change-user-install-location.patch
>
> --
>
> There are two issues related to the "sudo pip" change, but they
> already exist when pip is installed in $HOME/.local:
>
> (3) Priority issue between PATH and PYTHONPATH directories.
>
> When  the user runs "pip", the pip binary may come from /usr,
> /usr/local or  $HOME/.local/bin, but the Python pip module ("import
> pip") may come from  a different path. Which binary and which module
> should be used?
>
> Obvisouly, users can replace these two environment variables...
>
> (4)  Related to (3). Running "pip" may run pip binary of one pip
> version,  but pick the "pip" Python module of another pip version.
>
> For example, pip9 binary from /usr/bin/pip, but pip10 module from /usr/local.
>
>
> Fedora works around issue (4) with a downstream patch on pip:
>
> https://src.fedoraproject.org/rpms/python-pip/blob/master/f/pip9-allow-pip10-import.patch
>
> --
>
> I  don't well well how Linux distributions handle the issue with "sudo
>  pip". So don't hesitate to correct me if I'm wrong :-) My goal is
> just  to start a discussion about a common "upstream" solution.
>
> Victor
> --
> Distutils-SIG mailing list
> distutils-sig@python.org
> https://mail.python.org/mm3/mailman3/lists/distutils-sig.python.org/
> Message archived at https://mail.python.org/mm3/archives/list/distutils-sig@python.org/message/OLGLHTSHLEPLHUTTVNU6L5QFTMNFIB6Z/
>