[Distutils] PEP470 installation security problems

[Paul Moore]

On 8 October 2014 19:09, M.-A. Lemburg <mal at egenix.com> wrote:
> Thanks for your clarification, Paul.

In the interest of making sure everyone is understanding each other,
I'm going to follow up on this. I think there are some perceptions
that differ slightly, and some concerns that people have, that make
this a sensitive subject. I hope that by being open, I don't misword
something and cause offence or concern. Like you, I'm aware that the
process with pip has worked out well so far, and I don't want to
disrupt that.

> I just want to remind everyone that PEPs can be augments and mistakes
> can be fixed by superseding one PEP with another. It's a well
> working process, one that is accepted in Python land and in line
> with the core development process.

I think the intention with PEP 470 is precisely that, to supersede PEP
438. It's unfortunate that PEP 438 didn't work out so well in
practice, but we've learned some lessons, and hopefully we're getting
better at how we handle this. Specifically, I think that on the client
side, PEP 438 defined too many implementation details and didn't work
in terms of functional requirements. There's a tension here in that
PEPs have to speak in terms of "installers" and not target pip
specifically, as it's important to us that pip competes on an equal
footing with other installers, and we don't act as if we have a
privileged position.

It's also important that the relevant PEPs are actually PEPs about
*PyPI* changes. The installer details are simply advice to installers
on how to adapt to those changes. But the pip team takes that a stage
further and tries to ensure that we "eat our own dogfood" and
implement the advice that is included in the PEPs. It's relevant in
that context that the most popular "other installer" (easy_install)
does not always follow the recommendations in the PEPs, so all of the
negative user feedback to UI changes gets directed at pip rather than
at "the PEP" or some other general target.

> Since pip now is part of the Python stdlib (even though not bound
> by its release process), and the pip user base is identical with
> the CPython user base, the PEP process also applies to pip.

My perception is somewhat different (although the practical results
are similar, so this is more for context than anything else). It's
important to me that pip is *not* part of the stdlib, as the release
cycles of the stdlib are too slow for pip. What is part of the stdlib
is *ensurepip*, which is a mechanism to install pip into a Python
installation. When the subject of pip being included with Python came
up, the key concern from the pip side of things was that we are still
in a process of dynamic change (wheel support is still under
development and improvement, for example) and the constraints of core
Python / stdlib would stifle that development. Specifically, the
"Policies and Governance" section of PEP 453 explicitly notes that
"the bootstrapped software" (i.e. pip) will not come under CPython
policies, while still noting the need for co-operation.

Once again, it's worth noting that PEPs 438 and 470 are focused on
*PyPI*, rather than on pip. Being a good example of an installer by
following the recommendations in the PEPs for "installation programs"
is part of the pip team's responsibility to work together with the
CPython team, not a requirement of PEP 453.

I'm completely aware, by the way, that it's pretty naive to speak of
pip as merely one of many "installation programs" when in fact there's
very few real alternatives. We do so specifically to keep ourselves
honest...

> That's the consequence of playing the role of an officially
> sanctioned part of the ecosystem and comes as part of the
> responsibility resulting from PEP 435.

I hope the above explains why I see pip's responsibilities slightly
differently. In practical terms, I think it's unlikely that our
differing perspectives will result in any real differences.

> So far this has worked out well, which is why I'm surprised
> by some statements in this discussion.

There has been a fair bit of frustration in this discussion, and that
has resulted in some statements that have maybe been a little more
black and white than they should have been. But frankly I think
everyone has been working really hard to try to understand each
others' perspectives, and I hope that will continue. That's basically
why I wrote this note - I'm hoping that it's a bit clearer now why the
pip team are sensitive to strong statements like "the PEP process
applies to pip" which are oversimplifications of a rather complex and
still evolving relationship between the core CPython and PyPA teams.

Paul