[Distutils] What to do about the PyPI mirrors

[Donald Stufft]

On Aug 6, 2013, at 8:22 AM, Nick Coghlan <ncoghlan at gmail.com> wrote:

> On 6 August 2013 16:59, Christian Theune <ct at gocept.com> wrote:
>> Hi,
>> 
>> Thanks for all the feedback, I'll calm down a bit and ponder some more structured reply.
>> 
>> However, you're responding to the technicalities. I didn't see any consideration to the user pain. It seems irrelevant. Almost like arguing with the TSA about taking off your shoes.
> 
> User pain is the only reason for not making the change tomorrow.
> People need time to adjust, or to propose alternative solutions.
> 
>> f.pypi.python.org is going to go away. And *everyone* using it needs to change it. Manually - or else.
> 
> Delegating subdomains of python.org without a contractual relationship
> in place was a fundamental mistake. It should never have happened. We
> can either admit "We screwed up and set up a seriously flawed
> mirroring system" and take steps to fix it, or we can leave the
> HTTP-only mirrors open as a security hole forever.
> 
> One means by which I could see an f.pypi.python.org DNS record being
> left in place indefinitely is if the TUF folks are able to come up
> with a scheme for offering end-to-end security for the *existing* PyPI
> metadata, *and* the TUF metadata is mirrored by bandersnatch *and* the
> TUF client side integrity checks are invoked by pip. In that case, the
> security argument regarding the lack of TLS on the subdomains would be
> rendered moot, and the backwards compatibility argument for keeping it
> active would win.

It would be rendered moot as far as any tooling that was updated to work
with it (such as pip etc) however the browser level attacks would still be
in play. If those can be solved essentially require giving mirror operators
a SSL certificate for N.pypi.python.org (which still does not preclude a
malicious mirror operator or a compromised mirror from being used to
steal logins on PyPI).

> 
> Another potential alternative might be for Gocept to approach the PSF
> about getting an SSL certificate for that domain, ensuring pip and
> setuptools both support HSTS, and then switching that mirror over to
> using HSTS (so even configurations hardcoded to use
> http://f.pypi.python.org will still get a validated secure
> connection).

FWIW this doesn't make it secure unless they change their configuration to
point to https://f.pypi.python.org/simple instead of http://f.pypi.python.org/simple.*

Programmatic libraries typically don't support HSTS (as HSTS it primarily used
to prevent attacks that don't typically apply to command line clients). And certainly
none of the existing tools support it. So given that we'd be relying on the redirect
that upgrades the connection to HTTPS an attacker could simply return HTML
instead of the redirect to HTTPS.

Hence why it makes sense to remove, because either way users will need to edit
their existing configurations if they intend to be secure and moving to a different
domain will prevent the in browser attacks as well.

* This is also true of anyone who has hard coded an url to http://pypi.python.org/simple/
   however there's no reasonable way to fix that.

> 
> Both of those approaches would close the security hole, while leaving
> the domain in place. If upgrading pip and easy_install clients is a
> more acceptable solution than updating affected configurations to use
> a different domain name, then these are certainly options we should
> discuss.
> 
> The only option which I consider completely out of the question is
> leaving f.pypi.python.org (or any other *.pypi.python.org subdomain)
> in place indefinitely as an insecure HTTP-only endpoint.
> 
>> Other communities, like the Linux distributions, are doing simple, file-based stuff for ages. They did not learn from us, and AFAICT we didn't learn from them?
> 
> This case *is* a matter of us learning from other mirroring systems:
> none of them are based on delegating subdomains to third parties,
> they're all based on lists of mirror URLs, and some mechanism for
> retrieving that list. However, as long as the flawed way remains
> blessed as the official mirroring network, it's difficult for an
> alternative model to gain any traction.
> 
> Regards,
> Nick.
> 
> -- 
> Nick Coghlan   |   ncoghlan at gmail.com   |   Brisbane, Australia


-----------------
Donald Stufft
PGP: 0x6E3CBCE93372DCFA // 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 801 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <http://mail.python.org/pipermail/distutils-sig/attachments/20130806/6ebd04c0/attachment.pgp>