[Python-Dev] pip: cdecimal an externally hosted file and may be unreliable [sic]

[Donald Stufft]

On May 8, 2014, at 8:12 AM, Stefan Krah <stefan at bytereef.org> wrote:

> Victor Stinner <victor.stinner at gmail.com> wrote:
>> I don't understand your email. Can you please elaborate?
> 
> There is nothing wrong with the package.  The remark is a joke provoked by
> a long history of a campaign [1] against external packages on distutils-sig.
> 
> Many tools (like crate.io, when it was still up) have made derogatory
> remarks about external packages.  Now the latest version of the officially
> sanctioned download tool (pip) spits out copious warnings, one of which
> is the subject of this thread.

pip doesn't install externally hosted packaged by default because people should
be aware of what servers they depend on. Installs that depend on externally
hosted packages are brittle and more prone to failure. Every single external
server adds *another* SPOF into any particular install set. Even if every
external server has a 99.9% uptime, when you combine multiple of them the total
uptime of any particular set of requirements drops quickly. This has been a
major complaint that people have had over time.  However they are not a
security issue so they are easy to turn back on globally if a person wishes to
make their own deployment more brittle (--allow-all-external). 

However in addition to externally hosted packages, there are also unverifiable
packages. These are packages where there is no path to verify the download.
Generally an unverifiable package is one that either is linked to directly but
has no hash information encoded in the URL in a way that pip understands or,
the more common case, it is a package that is linked from a page that is linked
too on PyPI. Since we cannot build a path of trust to verify the package it is
trivial for an attacker to MITM it and execute arbitrary Python code on the
end users machine. This case *is* a security issue and pip goes out of it's way
to *greatly* discourage depending on packages that require this kind of
install.

> 
> 
> External packages are being singled out unfairly:
> 
>  1) Anyone can upload any package to PyPI (i.e. the index is not curated
>     at all).

Sure, the threat model of PyPI isn't that you can install anything willy nilly
and be safe. It's that when you ask for package X version Y, you should get
what the *author* published and not what attacker who happens to be in a
position to MITM you sends you.

> 
>  2) Last time I looked, access credentials (via "lost login") were sent
>     out in plaintext.

The existence of other security issues is not an excuse to not fix a security
issue. There are other problems and we're slowly working on trying to clear
them out.

> 
>  3) AFAIK people can upload a different (malicious) version of a
>     package with *the exact same name*.

Yes, a malicious author is needfully outside of the threat model for PyPI/pip.

> 
>  4) pip generally downloads the latest version, so a malicious person
>     can provide a good package for several years until people trust
>     him, then change to a trojaned version.

Yes, a malicious author is needfully outside of the threat model for PyPI/pip.

> 
>  5) Looking at the list of certificates that is in my default cert
>     store, I don't find SSL trustworthy at all.

TLS may not be the best method of authenticating package downloads, however it
is an easy one and it is significantly better than having no authentication at
all. You move your attack surface from "anyone who has a privileged network
position" to "anyone who has a privileged network position and also has the
ability to generate erroneously signed CA certificates". That is a
significantly smaller attack surface.

> 
>  6) D.J. Bernstein, who is somewhat security minded, has been shipping
>     his software *for years* with just plain HTTP and published checksums.

Argument from authority doesn't really hold up very well when DJB doesn't
distribute is software in a way that is intended to be consumed mechanically.
Also while he may be a crypto expert he is far from an expert on successfully
distributing software, unless you also think that the signature checking in
most OS provided package managers is pointless.

> 
> 
> To sum it up:
> 
>  1) Don't use pip to install packages directly from PyPI if security
>     really matters.

“security” isn’t a useful term on it’s own. You have to define a threat
model. A better answer here is don’t use pip to install packages directly
from PyPI if a malicious author is within your threat model (modulo other
issue we’re still working on, but as I said above, the existence of a second
security issue isn’t a reason not to fix the first issue).

> 
>  2) The best security we currently get is either
> 
>      a) with package signatures (*if* you can get the author's key via
>         a trustworthy channel, which is rarely the case).
> 
>      b) with decent checksums that are recorded on public mailing
>         lists at the time the package is announced (it would be
>         hard for an attacker to modify all mailing list archives.)
> 
>     Whether a package is internal or external is orthogonal to both points.

Again as I said above, the internal vs external isn’t a security related decision
it is a “uptime” related decision. Verifiable vs unverifiable *is* a security
related decision.

> 
> 
> With all these points, I find it questionable for an "official" install
> tool to make security related remarks about just one category of weaknesses.
> 
> After all, people might be led to believe that pip is some sort of apt-get
> and all uploaded packages are safe.
> 
> 
> Stefan Krah
> 
> 
> 
> [1] Note that the joke is quite innocent in comparison to what I've read on
>    distutils-sig about the subject.
> 
> 
> 
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/python-dev/attachments/20140508/fa88483c/attachment.sig>