[Distutils] PEP470 installation security problems

[holger krekel]

On Tue, Oct 07, 2014 at 08:00 -0400, Donald Stufft wrote:
> > On Oct 7, 2014, at 6:09 AM, holger krekel <holger at merlinux.eu> wrote:
> >> I had thought of similar things, and my reasons for not using an <a
> >> href> and instead using a meta tag and for removing the old URLs
> >> instead of just making this in addition to is:
> >> 
> >> 1. I don’t *want* users of older versions of pip/easy_install to
> >> implicitly be fetching these things, they should be able to opt in as
> >> well and indeed all the mechanisms exist in pip/easy_install for them
> >> to already do so. The only thing that doesn’t exist is the discovery
> >> mechanism.
> > 
> > I think it's better to generally avoid deliberately breaking things.
> > Things break enough even when we don't intend them to.
> > 
> > IOW, Pypi should IMO aim to preserve working with as many client side
> > scenarios as possible -- while adding things and improving for newer
> > versions of clients.
> 
> And here I think is where the crux of our disagreement lies I think.
> 
> I think that PyPI should preserve working with as many client side
> scenarios as possible, except where there is good reason to do so. 
>
> I believe the fact that the vast bulk of the cases we’d be breaking are
> people who are silently, and often unknowingly, being directed to
> download some code over unauthenticated channels is a very good reason
> to break those cases. Especially given the fact that there is a fairly
> trivial work around for people who want to restore that behavior.
> 
> In a way this is similar to switching Python to enforcing TLS verification
> by default, which afaik Guido has blessed even for 2.7 assuming that there
> is a sane way to restore the default behavior and configure it.

Are you saying that PEP470's breaking of backard compatibility is 
deliberate and helps to defend against MITM attacks during installation?
That might be true although i note that hacked servers (see also: bash,
ssl) are much more common than MITM attacks and a hacked server can do
SSL just fine.

In any case, I see two security related downsides of PEP470, one of
them severe.

For one, current multi-index operations are riskier than PEP438's
validated external release file urls.  Because currently you only need
to trust pypi.python.org has not been hacked but with PEP470 you
need to trust the integrity of the external site as well.
IIUC you and Nick think this is acceptable because people deliberately
make that choice by supplying an explicit option to use the external
index, right?  If so i think the PEP should also be clear on the fact that
Pip/pypi's external repo support is far inferior to typical linux repos
because release files are not signed etc.

Worse security problems loom with current multi-index ops like
the --extra-index-url option which is advertised prominently in PEP470.
You recommend to use it for private package indexes, but it can
trivially compromise user machines: you register a private package name
publically to pypi and add some malware release files, and can then
infect all machines which execute an innocent "pip install
--extra-index-url ...".  I think we conversed about this issue earlier but i
don't see the PEP discussing it but rather it recommends using it
without a direct call for caution (*).  I maintain this attack is more
serious than MITM attacks for which you are even ready to break backward
compat.

Donald, Nick, i am not against the goals of PEP470 per se but in its
current form i see it rather causing damage.  When i explained to companies
the dangers of pip multi-index operations they were rather alarmed and urged
me to do something about it within the devpi context.  But PEP470 pretends
all is fine and everybody should move to multi-index immediately -- that's
premature at least if not outright endagering users even today because
they take the advise in the draft PEP470 for granted because it comes
from Nick and Donald who usually know what they are talking about.

At the very least we need to have clear discussion in the PEP about it
and safer options for pip and PEP470 needs to MANDATE it for pip and
maybe even for easy_install -- you could follow the devpi
"pypi_whitelist" design to prevent mixed private/public package links
and introduce a "--private-index-url" which means that pip would look
first there and when it finds links for a name it would not consider
other/public indexes unless the name is explicitely whitelisted.  I
admit i am not happy about the usability of that but it gives a good
secure default against public packages infecting private package installs.

best,
holger

(*) I saw that PEP470 in a different section says "Installers SHOULD
implement some mechanism for removing or otherwise disabling use of the
default repository." but that's just a "SHOULD" and even if implemented 
it will not fix fix things retro-actively for older pip/easy_install
users -- but you claim fixing things for them is within the PEP470 scope
above.