[Distutils] Request for comment: Proposal to change behaviour of pip install

[Robert Collins]

On 26 June 2016 at 05:16, Donald Stufft <donald at stufft.io> wrote:
>
>> On Jun 25, 2016, at 12:31 PM, Ian Cordasco <graffatcolmingov at gmail.com> wrote:
>>
>> On Sat, Jun 25, 2016 at 5:25 AM, Pradyun Gedam <pradyunsg at gmail.com> wrote:
>>> Hello List!
>>>
>>> There is currently a proposal to change the behaviour to pip install to
>>> upgrade a package that is passed even if it is already installed.
>>>
>>> This behaviour change is accompanied with a change in the upgrade strategy -
>>> pip would stop “eagerly” upgrading dependencies and would become more
>>> conservative, upgrading a dependency only when it doesn’t meet lower
>>> constraints of the newer version of a parent. Moreover, the behaviour of pip
>>> install --target would also be changed so that --upgrade no longer affects
>>> it.
>>>
>>> A PEP-style write-up
>>> (https://gist.github.com/pradyunsg/4c9db6a212239fee69b429c96cdc3d73)
>>> documents the reasoning behind this proposed change, what the change is and
>>> some alternate proposals that were rejected in the discussions.
>>>
>>> This is a request for comments on the pull-request implementing this
>>> proposal (https://github.com/pypa/pip/pull/3806) for your views, thoughts
>>> and possible concerns related to it.
>>
>> Having `pip install` implicitly upgrade a package will break the way
>> tooling like ansible, puppet, salt, and chef all work. Most expect
>> that if you do `pip install requests` twice you won't get two
>> different versions. At the very least, there should be an option added
>> that implies that if the version of the specified package is already
>> installed and satisfactory, that it is not upgraded.
>
> So this may be true, but it’s also not particularly hard to work around
> similarly to what they do already for system package managers, they can
> do ``pip list`` or ``pip freeze`` to see if something is already installed.
> Not only will this work across versions, but it’ll also work *better*
> because it won’t involve hitting the network to determine this information.

Would we wait to make this change for the existing config systems to
implement this change, and get it into all their stable versions? If
not, then it doesn't matter whether its easy to work around - we're
causing countless installs of those systems to break if we make the
change.

>> If you want to change the upgrade behaviour of pip, I suggest not
>> modifying the existing `pip install` and instead deprecating that in
>> favor of a new command. Silently pulling the rug out from under people
>> seems like an incredibly bad idea.

+1 ^

> This is going to break things, no matter what we do— and the current
> behavior is also broken in ways that are causing other projects to need
> to do broken things.

How so? I haven't actually seen a bug or issue describing what folk are doing.

> The current solution I think breaks *less* things and once the initial
> pain of breakage is over will end up with a much nicer situation. Importantly
> the vast bulk of the breakage will be centralized in things like Chef where
> one group can make a change and fix it for thousands.

I think the primary outcome of the proposed plan is that we won't hear
about breakage until it shows up on 'haveibeenpawned.com'.

There are I think two discussions to have:
 - what should we be giving users as a UX
 - how to get there

For the how-to-get-there aspect, I see absolutely no reason to make
incompatible changes. We can add new options -O or whatever and leave
the existing ones alone. Changing the behaviour for 'install NAME' I
could potentially see, but I'd hesitate on that too. Our users don't
react well to instability.

For the what-should-we-aim-for question, thats more tricky.

There are I think several different aspects that all interact, and
make comparisons to dnf/apt etc quite meaningless.

Firstly, those systems have automated cron-like systems to keep them
secure; they can afford to not worry about security issues in their
dependency chains because everything will be pushed out automatically.
venvs have no such mechanism today - pip doesn't even have a suitable
command today (though you can of course write one around it). Without
one of these systems, I expect that venvs will accrete *years* old
versions of software, and we'll see the exact inverse breakage that
this proposed change to -U is about: lower minimums in PyPI are often
useless.

Secondly, apt/dnf etc have a presumption that all versions of all
software either work together or explicitly do not (via a combination
of conflicts and dependencies) - because they're dealing with an order
of magnitude less software that we are, and the system is an
integrated system, not a federated system like pip's data model. pip
doesn't have that presumption - hence this whole proposal - but the
consequence is that rather than giving uses the most *likely* to work
combination of software, we're going to give them the *least* likely
to work combination [because there are billions of combinations of
latest-X + older-Y, but only one latest-X + latest-Y].

Lastly, by defaulting to non-recursive upgrades we're putting the
burden on our users to identify sensitive components and manage them
much more carefully.

Wearing my user hat, while I hate it when pip breaks something, I
really want pip to do as much thinking for me as possible, and this
seems like a really weird combination of do-more-thinking(upstall
named things), and do-less-thinking-for-me (leave everything else at
random versions).

-Rob