[Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev)

[Paul Moore]

So there's an ongoing debate over pip's behaviour around disallowing
external hosting by default (see thread "pip: cdecimal an externally
hosted file and may be unreliable" over on python-dev for the latest
round).

It appears that the reason for disallowing external hosting (as
opposed to unverifiable downloads) is purely about reliability - we
can't be sure that an external host provides the same level of uptime
as PyPI[1]. Given that, it seems to me that the situation is, for an
externally hosted package foo:

    `pip install foo` - fails immediately, 100% of the time
    `pip install --allow-external foo foo` - works in all but a few
cases where foo's host is down[2]

I cannot understand how guaranteed failure is ever better than
"occasional but rare" failure.

For situations where it is critical to minimise the risk of an
external host outage causing a deployment to fail, the only answer is
to not use foo, or to host foo on your own private index. In both
cases, all you need is to know that foo is externally hosted to do
that - you certainly don't need pip to fail.

As a concrete proposal:

1. Remove --allow-external/--allow-all-external and make it the
default behaviour.
2. Add a new command to pip, maybe something like `pip check-external`
which checks a set of reuirements, and reports the requirements that
are externally hosted and which hosts they rely on. That gives users
who need 100% reliability the information they need to implement the
appropriate solution. Without causing pain for users who don't.

Note that the above is based on the fact[3] that *there are no
security risks to --allow-external*. I am not arguing for a reduction
in security, or a change to any sort of threat model.

Comments?

Paul

[1] Donald explicitly stated that this is the case in the earlier
thread (https://mail.python.org/pipermail/python-dev/2014-May/134454.html).
I think Nick confirmed this (although I don't have a reference to
hand). If it's not true then we need to be a lot clearer and more
explicit about *why* ignoring external hosting by default is needed,
because it seems nobody knows :-(
[2] BTW, the syntax of `--allow-external` is hideous, but that's a
side issue I don't want to debate right now.
[3] See the first note.