[Distutils] PEP 438, pip and --allow-external (was: "pip: cdecimal an externally hosted file and may be unreliable" from python-dev)

[Paul Moore]

On 11 May 2014 13:47, Donald Stufft <donald at stufft.io> wrote:
>> https://pypi.python.org/simple/egenix-mx-base/ has verifiable external
>> links. I'm pretty surprised that Donald hasn't heard of mx-base.
>
> egenix-mx-base does not have verifiable external links.Verifiable external
> links must be both directly linked to from the /simple/ index page and
> must include a hash. egenix-mx-base does not do this.

OK, that clarifies that, and also makes it clear that what constitutes
"safe" is not immediately obvious (something you've been saying a lot,
but which never eally hit home to me before).

So, some questions:

1. Is MAL aware that egenix-mx-base is not verifiably externally
hosted[1], and if so, what is he asking for? Automatic download with
no need for opt-in of unverifiable external downloads? That seems
pretty much in conflict with the whole intent of PEP 438.
2. Does Nick feel that opt-in for unverifiable external downloads is
something that could be removed as part of this discussion? From what
I've seen the consensus view of the PyPA team[2] is that removing
opt-in for unverifiable external links is not on the table (indeed,
there's a move to simply not allow them at all at some point).
3. Has anyone considered other options, such as MAL adding an extra
link page (https://downloads.egenix.com/python/download_url/egenix-mx-base/3.2.7/
has a link to https://downloads.egenix.com/python/download_url/egenix-mx-base/index.html
but it's dead, so maybe he intended to have one but it got lost) and
users use --extra-index-url instead of --allow-unverifiable? Do we
need to take a step back and look at the wider picture here?
4. Do we need to have this level of detailed discussion with every one
of the people with external links? If not, then how do we choose who
is worth asking? (No disrespect to MAL or Stefan, IMO both of them are
clearly in the "worth talking to" group).

While relationship management is important, it's also the case that
the PyPA team were told during the original discussion about bundling
pip that doing so would not affect their autonomy. If people feel that
the PyPA team are not doing a good job of managing user feedback,
that's one thing, but I'm not sure how there's a "delegation of
authority from python-dev to PyPA" involved - python-dev never had any
authority over pip to delegate. Rather the other way around, PyPA
accepted giving python-dev a certain level of influence as a result of
being bundled, and promoted as the "official" packaging solution.

I'm happy to discuss how the PyPA ensures that the views and desires
of pip users are best catered for. I don't think it's something we've
always done a good job of, and regardless of anything else I think we
need to learn some lessons from how this issue has been handled. But
it's our job to get that right, and while advice is always
appreciated, we also need some time to discuss things among ourselves,
which can be hard to get. And when we do make a decision, it needs to
be respected.

Finally, and most importantly, can I remind people that the behaviour
under question is actually covered by PEP 438, and pip is only
following that PEP's requirements:

"""
The second update should change the default mode to allow only
installation of rel="internal" package files, and allow installation
of externally-hosted packages only when the user supplies an option.

The installer should distinguish between verifiable and non-verifiable
external links. A verifiable external link is a direct link to an
installable file from the PyPI simple/ index that includes a hash in
the URL fragment ("#hashtype=hashvalue") which can be used to verify
the integrity of the downloaded file. A non-verifiable external link
is any link (other than those explicitly supplied by the user of an
installer tool) without a hash, scraped from external HTML, or
injected into the search via some other non-PyPI source (e.g.
setuptools' dependency_links feature).

Installers should provide a blanket option to allow installing any
verifiable external link. Non-verifiable external links should only be
installed if the user-provided option specifies exactly which external
domains can be used or for which specific package names external links
can be used.
"""

This discussion should actually be about changing PEP 438. If the PEP
changes, pip will have to change to follow. Contrariwise, unless the
PEP changes, pip should not change. I'm sure the level of unhappiness
over the proposed solutions won't change, but what it *will* do is to
take any question of whether PyPA or python-dev have the "authority"
out of the equation. That, to my mind, would be a significant benefit
in relationship management terms - having PyPA be cast as the "bad
guys" in the current scenario is very uncomfortable for me.

Paul

[1] That's a horrible phrase. Sorry I can't think of a better one.
[2] There hasn't been a vote or anything, it's just what I've seen in
various comments.