[Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev)

[Donald Stufft]

On May 9, 2014, at 9:38 AM, Paul Moore <p.f.moore at gmail.com> wrote:

> On 9 May 2014 14:12, Donald Stufft <donald at stufft.io> wrote:
>> I think that you’re conflating any bug report about these two flags with bug
>> reports about externally hosted things at all.
> 
> That may well be true. I find this whole thing confusing (which is
> sort of my point, I guess). Don't we get a lot of reports where the
> user is advised to add one of the flags, that doesn't work, so they
> are then advised to add the other? Or am I misremembering and people
> are advised to add one, but actually add the other and then say the
> advice didn't work? Either way, only having one set of flags would
> remove that confusion.
> 
> But yes, I'm going from my notoriously bad memory here.

So originally in order to get pip to consider external hosted files at all
you had to have —allow[-all]-external. On top of that if you wanted
to install something that was unverifiable you would have to add
—allow-unverifiable.

This meant to install something that was “unsafe” you had to do:

$ pip install —allow-external foo —allow-unverifiable foo foo

This has since been changed so that —allow-unverifiable implies
—allow-external so that all that would be required is:

$ pip install —allow-unverifiable foo foo

So that’s the state of the flags, as to the actual messages that people
get:

pip does not attempt to discover (most) unsafe links if it knows it isn’t
going to use them. It does this as a (substantial) performance
improvement. However the side effect of this is that we don’t actually
know ahead of time if there are any unverifiable files so at best what
we can do is saying that maybe there are some files that can be
discovered if you add a flag.

Typically what people do is they add the —allow-external flag, find
it doesn’t work then add the —allow-unverifiable flag. The —allow-external
flag rarely actually helps them because very few projects are
safely hosted externally.

> 
>> The developer then tries to do his ``pip install -r requirements.txt``
>> and suddenly it fails. This developer knows that pip installs from PyPI so
>> he goes to pypi.python.org and sees it works fine in his browser, he goes to
>> status.python.org and sees there are no downtimes.
> 
> Personally, I'd try manually downloading the file from PyPI, which
> would fail. I certainly wouldn't just check the pypi homepage and
> assume that meant a download I could easily check directly "must" be
> OK... Or I'd do "pip install -v" and see "downloading http://... -
> FAILED". Problem solved.
> 
> I think you're assuming the developer is a lot less capable than I
> would. But maybe our experiences differ.
> 

That story is influenced directly from my experiences as being somewhat
of a prominent person and having lots of people contact me personally
when things fail.

>> ``pip install foo`` where foo is hosted externally fails. The developer
>> is told that this file is hosted externally and might be unavailable in the
>> future. They decide they don't care and they just want a thing to work so they
>> add ``--allow-external foo`` to their requirements.txt and go on their merry
>> way.
> 
> I see your point but honestly I'd expect such a developer to simply
> put allow-all-external=true in his pip.ini once, probably so long ago
> that he's forgotten, so there's no gain. If he actually cares enough
> to track externally hosted files, adding a comment to requirements.txt
> would be just as good, and having something like ```pip install
> --list-external``` to show him what was externally hosted helps him
> get the information he needs.

I’m not sure anyone has ever mentioned to me that they’ve done this,
I do have people who have mentioned to me that they have
added the —allow-external foo to their requirements.txt. Obviously
not scientific or complete but just what I’ve been told.

> 
> I'm not saying people mightn't want to consider externally hosted but
> safe files specially, just that we could offer better tools for doing
> so than we currently do.
> 

The tooling could likely be better yes, I think the impact of doing so is going
to be extremely low because barely anyone is even using that particular
feature.

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20140509/42e92a8e/attachment.sig>