[Distutils] The future of invoking pip

[Donald Stufft]

On November 5, 2015 at 8:04:41 PM, Robert Collins (robertc at robertcollins.net) wrote:
> On 6 November 2015 at 10:08, Donald Stufft wrote:
> ...
> > One possible solution to the above problems is to try and move away from using
> > ``pip``, ``pipX`` and ``pipX.Y`` and instead push people (and possibly
> > deprecate ``pip`` &etc) towards using ``python -m pip`` instead. This makes it
> > unambigious which python you're modifying and completely removes all of the
> > confusion around that. However this too has problems:
> >
> > * There is a lot of documentation out there in many projects that tell people
> > to use ``pip install ...``, the long tail on getting people moved to this
> > will be very long.
>  
> I don't see that as a specific problem. It drags out the deprecation
> period is all.
>  
> > * It's more to type, 10 more characters on *nix and 6 more characters on
> > Windows which makes it more akward and annoying to use. This is particularly
> > annoying inside of a virtual environment where there isn't really any
> > ambiguity when one is activated.
>  
> cat > /usr/bin/pip << EOF
> python -m pip $@
> EOF
>  
> Seriously - isn't the above entirely sufficient?

It’s not particularly hard to do this (we could even make a pip-cli package on PyPI that did it). It’s just one more step and one more “weird” thing for newcomers. 

>  
> > * It still has the annoyance around having multiple pip installs all over the
> > place and needing to manage those.
>  
> This is a mixed thing. You *need* those installs when pip drops support for 2.6.

Well, as I said to Paul, nothing in any of these proposals mandates that you must only have one installation. The only difference is the third thing makes it *possible* to only have one install (though since ``pip`` is the only name, it means it’s harder to install pip into 2.6 when it is altinstall’d into /usr/bin along with other Pythons since both the 2.6 supporting pip and the non-2.6 supporting pip will want to be called “pip”. It’s less of a big deal for virtual environments because they already have a separate bin directory. This would almost be akin to RHEL only supporting VM installations of their really old OSs.

>  
> > * We still support Python 2.6 which doesn't support executing a package only
> > modules via ``-m``. So we'll break Python 2.6 unless people do
> > ``python -m pip.__main__`` or we move pip/* to _pip/* and make a pip.py which
> > will break things for people using pip as a library (which isn't currently
> > supported).
>  
> Or the same wrapper approach can deal with this - as long as there is
> a pip.__main__ on all Pythons.

python -m pip is implemented via pip.__main__ so python -m pip.__main__ will function for as long as python -m pip does (unless Python drastically changes).

>  
> > Another possible option is to modify pip so that instead of installing into
> > site-packages we instead create an "executable zip file" which is a simple zip
> > file that has all of pip inside of it and a top level __main__.py. Python can
> > directly execute this as if it were installed. We would no longer have any
> > command except for ``pip`` (which is really this executable zip file). This
> > script would default to installing into ``python``, and you can direct it to
> > install into a different python using a (new) flag of ``-p`` or ``--python``
> > where you can specify either a full path or a command that we'll search $PATH
> > for.
>  
> I don't like this because:
> - pip won't be able to be interrogated in the same way as all other
> python packages can be

I’m not sure what you mean be “interrogated” but it doesn’t actually *prevent* it, it just makes it a bit harder. You’d be able to restore the current ``import`` ability (assuming that’s what you meant by interrogated) by doing something like:

    $ PYTHONPATH=`which pip` python -c “import pip”

> - the breakage is huge :(.

This is my biggest problem with it :(

> - it doesnt' actually solve the windows problem

Yea, there’s no clean (and generic) solution to that besides python -m pip.

> - it makes it hard to use pip except via a subprocess - and while its
> not supported, it is at least in principle something we could be
> working on (and there was interest in that on python-dev the other
> month IIRC).

I’m almost entirely certain we’ll never really support using pip as anything but a subprocess. Having two APIs (a subprocess one, and a Python one) isn’t great and we’re always going to need a subprocess one for things like Chef/Puppet that isn’t written in Python. Even in that thread, I believe they decided to go with subprocess because of the problems of trying to run pip in process (things won’t show up as installed and such).

This is a bit of a mixed thing too though, because right now it’s trivially easy to use pip via import and so people do it when it’s not appropriate to do so (like inside of ``setup.py`` files), making it more difficult would make it so people who are doing it think twice about it before they do it.

>  
> ...
> > A side conversation here is that pip currently bundles a number of dependencies
> > inside of itself because it can't really have any dependencies. This works
> > fine but it's a bit of an annoyance to maintain. The larger problem is that a
> > number of downstreams don't like this and have attempted to route around it to
> > varying degrees of success. One benefit of the third option is that we can
> > remove the need to directly copy the bundled libraries into the pip source code
> > and we can install just bundle it inside the built zip file.
>  
> Well, we could do that today - only bundle in get-pip.py, have
> installed copies have dependencies.

Well we can’t really do it in get-pip.py in a way that would satisfy our constraints. 

There are two main ways to do bundling/vendoring in Python. Either you add it to your application like we do now with pip._vendor.* which typically requires adjusting imports of the items you’re bundling/vendoring (at the very least, for any of their own dependencies). We currently deal with this by just manually adjusting those imports whenever we pull in a new version. In this version you don’t do ``import six`` but you instead do ``from pip._vendor import six``.

The other way of doing it is by including the unmodified sources somewhere and then munging sys.path to add them to it before doing any imports. This is easier to deal with in an automated system since it doesn’t involve needing to modify anything, but it pollutes the sys.path of the Python process which makes it unfriendly to people trying to run things in the same process.

Currently pip does the first thing. We picked that because we decided that even though we don’t support ``import pip``, that enough people did it that we wanted to not be as invasive to the current running process. If ``import pip`` became harder then that argument doesn’t really make sense any more and we can easily switch to the second way. Even ``import pip`` doesn’t become harder, it might make sense to switch though.

The “stick pip + dependencies in a zip file” actually handles all of the munging of sys.path for us because everything would just be at the top level of the zip file (which Python will add to the front of sys.path for us). So it will all just work automatically. It would remove the immediately easy ``import pip`` so there’s no longer any real concern about polluting the current running process. We could replicate that without the zip thing just by manually munging sys.path and deciding we don’t care about polluting the current running process when people do ``import pip``.

Being able to do the bundling automatically and without manual patching makes it easier for systems like Debuntu to automatically do it in their build process, but in no scenario would we start supporting people to install it without doing that.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA