[Distutils] The future of invoking pip

[Donald Stufft]

There is currently a semi related set of problems that I'd really like to figure
out an answer too so we can begin to work on a migration path and close these
out. This is dealing with a fairly fundamental aspect of pip so I'm bringing it
up here to try and get wider discussion than the issue tracker or pypa-dev
list.

Currently pip installs a number of commands like ``pip``, ``pipX`` and
``pipX.Y`` where the X and X.Y corresponds to the version of Python that pip
is installed into. Pip installs into whatever Python is currently executing it
so this gives some ability to control which version of Python you're installing
into (``pip2.7`` for Python 2.7 etc).

However, this has a few problems:

* It really only works if you only have one version of Python installed for
  each X.Y series, as soon as you have multiple versions in one series
  (including alternative implementations besides CPython) you run into a
  problem of which Python 2.7 is pip2.7 for and how do you invoke it for the
  *other* Python 2.7's. Something like pip2.7.8 is really ugly, and still
  doesn't solve the problem when you have multiple 2.7.8 installs and
  pip-pypy2-4.0 is even uglier.

* The above gets *really* confusing when ``pipX`` or ``pip`` do not agree with
  what ``pythonX`` and ``python`` point to.

* Having pip need to be installed into each Python means you end up with a
  bunch of independent pip installations which all need to be independently
  updated. We've made this better by having recent pips warn you if you're not
  on the latest version, but it's not unusual for me personally to have 30+
  different installations of pip on my personal desktop. That isn't great.

* Having ``pip`` on Windows requires us to create a "script wrapper" which is
  just a shim .exe that just executes a python script. This is due to the fact
  that Windows special cases .exe in places. This causes problems because you
  can't actually do ``pip install --upgrade pip`` on Windows because you'll get
  an error trying to update the script wrapper, you need to do
  ``py -m pip install --upgrade pip`` so the .exe file isn't currently open.

One possible solution to the above problems is to try and move away from using
``pip``, ``pipX`` and ``pipX.Y`` and instead push people (and possibly
deprecate ``pip`` &etc) towards using ``python -m pip`` instead. This makes it
unambigious which python you're modifying and completely removes all of the
confusion around that. However this too has problems:

* There is a lot of documentation out there in many projects that tell people
  to use ``pip install ...``, the long tail on getting people moved to this
  will be very long.

* It's more to type, 10 more characters on *nix and 6 more characters on
  Windows which makes it more akward and annoying to use. This is particularly
  annoying inside of a virtual environment where there isn't really any
  ambiguity when one is activated.

* It still has the annoyance around having multiple pip installs all over the
  place and needing to manage those.

* We still support Python 2.6 which doesn't support executing a package only
  modules via ``-m``. So we'll break Python 2.6 unless people do
  ``python -m pip.__main__`` or we move pip/* to _pip/* and make a pip.py which
  will break things for people using pip as a library (which isn't currently
  supported).

Another possible option is to modify pip so that instead of installing into
site-packages we instead create an "executable zip file" which is a simple zip
file that has all of pip inside of it and a top level __main__.py. Python can
directly execute this as if it were installed. We would no longer have any
command except for ``pip`` (which is really this executable zip file). This
script would default to installing into ``python``, and you can direct it to
install into a different python using a (new) flag of ``-p`` or ``--python``
where you can specify either a full path or a command that we'll search $PATH
for.

* We'll break the usage of ``python -m pip``.

* We'll break the usage of ``pipX`` and ``pipX.Y``.

* We'll break the usage of anyone using pip as a library (but this is not
  actually supported).

* We still can't do ``pip install --upgrade pip`` by default, and since there's
  no ability to do ``py -m pip install --upgrade pip`` we'll need to figure
  something out (perhaps skip updating the .exe? I don't know).

The first option is the status quo and thus doesn't represent any *new*
breakage but has all of the problems we have currently. The second option
uses the ``-m`` operater but has problems with 2.6 and it's not as nice to
type but it removes all the ambiguity. The third option introduces the most
breakage, but it removes all the amiguity *and* it removes the need to manage
a number of pip installations.

A side conversation here is that pip currently bundles a number of dependencies
inside of itself because it can't really have any dependencies. This works
fine but it's a bit of an annoyance to maintain. The larger problem is that a
number of downstreams don't like this and have attempted to route around it to
varying degrees of success. One benefit of the third option is that we can
remove the need to directly copy the bundled libraries into the pip source code
and we can install just bundle it inside the built zip file.

Thoughts?

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA