Disabling non HTTPS access to APIs on PyPI
Historically PyPI was only available over either HTTP or unvalidated HTTPS, and over time we’ve been pushing more and more traffic onto HTTPS. In Warehouse the decision was made to *not* redirect “API” URLs from HTTP to HTTPS, but to rather return an error accessing them from HTTP. This is because while logged in views have HSTS to ensure HTTPS in the browser (and with humans manually entering them into the URL bar regularly they are more error prone) APIs which are typically accessed by automated clients with an URL configured or hardcoded typically do not respect HSTS, so if you had a script that did ``curl http://pypi.python.org/simple/%60%60 http://pypi.python.org/simple/%60%60, it would silently get redirects to https and appear to “work”, but you wouldn’t get any of the security properties of TLS because an attacker would just intercept the request prior to the redirect happening.
Today I’ve backported this changed to the current production deployment of PyPI, which means that you can no longer access /simple/ and /packages/ over HTTP and you will have to directly go to HTTPS. For most people this should have no effect, because most tooling should be defaulting to HTTPS anyways, however if you’re using a significantly old version of tooling, it may still be defaulting to the HTTP url and will now stop functioning.
The recommended remediation is to upgrade your tooling to versions that support verified TLS connections and which default to the proper HTTPS URLs.
Heads up to anyone running pypiserver, this breaks the default redirection from pypiserver to PyPI. There's an open issue[1] to fix it. In the meantime, you can use the pypiserver option `--fallback-url https://pypi.python.org/simple/%60 to work around it.
--nate
[1] https://github.com/pypiserver/pypiserver/issues/179
On Thu, Oct 26, 2017 at 11:11 AM, Donald Stufft donald@stufft.io wrote:
Historically PyPI was only available over either HTTP or unvalidated HTTPS, and over time we’ve been pushing more and more traffic onto HTTPS. In Warehouse the decision was made to *not* redirect “API” URLs from HTTP to HTTPS, but to rather return an error accessing them from HTTP. This is because while logged in views have HSTS to ensure HTTPS in the browser (and with humans manually entering them into the URL bar regularly they are more error prone) APIs which are typically accessed by automated clients with an URL configured or hardcoded typically do not respect HSTS, so if you had a script that did ``curl http://pypi.python.org/simple/%60%60, it would silently get redirects to https and appear to “work”, but you wouldn’t get any of the security properties of TLS because an attacker would just intercept the request prior to the redirect happening.
Today I’ve backported this changed to the current production deployment of PyPI, which means that you can no longer access /simple/ and /packages/ over HTTP and you will have to directly go to HTTPS. For most people this should have no effect, because most tooling should be defaulting to HTTPS anyways, however if you’re using a significantly old version of tooling, it may still be defaulting to the HTTP url and will now stop functioning.
The recommended remediation is to upgrade your tooling to versions that support verified TLS connections and which default to the proper HTTPS URLs.
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
On Thu, Oct 26, 2017 at 5:11 PM, Donald Stufft donald@stufft.io wrote:
Historically PyPI was only available over either HTTP or unvalidated HTTPS, and over time we’ve been pushing more and more traffic onto HTTPS. In Warehouse the decision was made to *not* redirect “API” URLs from HTTP to HTTPS, but to rather return an error accessing them from HTTP. This is because while logged in views have HSTS to ensure HTTPS in the browser (and with humans manually entering them into the URL bar regularly they are more error prone) APIs which are typically accessed by automated clients with an URL configured or hardcoded typically do not respect HSTS, so if you had a script that did ``curl http://pypi.python.org/simple/%60%60, it would silently get redirects to https and appear to “work”, but you wouldn’t get any of the security properties of TLS because an attacker would just intercept the request prior to the redirect happening.
Today I’ve backported this changed to the current production deployment of PyPI, which means that you can no longer access /simple/ and /packages/ over HTTP and you will have to directly go to HTTPS. For most people this should have no effect, because most tooling should be defaulting to HTTPS anyways, however if you’re using a significantly old version of tooling, it may still be defaulting to the HTTP url and will now stop functioning.
The recommended remediation is to upgrade your tooling to versions that support verified TLS connections and which default to the proper HTTPS URLs.
+1
This will probably (unfortunately) break some things for some people, which is worrying. But it is the right thing to do and good advice in general.
-----Original Message----- From: Distutils-SIG [mailto:distutils-sig-bounces+tritium- list=sdamon.com@python.org] On Behalf Of Erik Bray Sent: Friday, October 27, 2017 11:16 AM To: Distutils distutils-sig@python.org Subject: Re: [Distutils] Disabling non HTTPS access to APIs on PyPI
On Thu, Oct 26, 2017 at 5:11 PM, Donald Stufft donald@stufft.io wrote:
Historically PyPI was only available over either HTTP or unvalidated HTTPS, and over time we’ve been pushing more and more traffic onto HTTPS. In Warehouse the decision was made to *not* redirect “API” URLs from HTTP
to
HTTPS, but to rather return an error accessing them from HTTP. This is because while logged in views have HSTS to ensure HTTPS in the browser
(and
with humans manually entering them into the URL bar regularly they are
more
error prone) APIs which are typically accessed by automated clients with an URL configured or hardcoded typically do not respect HSTS, so if you had a script that did ``curl http://pypi.python.org/simple/%60%60, it would silently get redirects to https and appear to “work”, but you wouldn’t get any of
the
security properties of TLS because an attacker would just intercept the request prior to the redirect happening.
Today I’ve backported this changed to the current production deployment
of
PyPI, which means that you can no longer access /simple/ and /packages/
over
HTTP and you will have to directly go to HTTPS. For most people this should have no effect, because most tooling should be defaulting to HTTPS
anyways,
however if you’re using a significantly old version of tooling, it may still be defaulting to the HTTP url and will now stop functioning.
The recommended remediation is to upgrade your tooling to versions that support verified TLS connections and which default to the proper HTTPS
URLs.
+1
This will probably (unfortunately) break some things for some people, which is worrying. But it is the right thing to do and good advice in general.
Might want to post a message on the front page of pypi.python.org when this becomes effective (if it's not there already.)
Distutils-SIG maillist - Distutils-SIG@python.org https://mail.python.org/mailman/listinfo/distutils-sig
- Are there issue tickets which contain the search-indexed ERROR_STRINGS users may encounter due to this change?
- Does it make sense to add an update regarding this necessary security upgrade to https://status.python.org (which can be subscribed to and followed on http://www.twitter.com/PythonStatus )?
On Thursday, October 26, 2017, Donald Stufft donald@stufft.io wrote:
Historically PyPI was only available over either HTTP or unvalidated HTTPS, and over time we’ve been pushing more and more traffic onto HTTPS. In Warehouse the decision was made to *not* redirect “API” URLs from HTTP to HTTPS, but to rather return an error accessing them from HTTP. This is because while logged in views have HSTS to ensure HTTPS in the browser (and with humans manually entering them into the URL bar regularly they are more error prone) APIs which are typically accessed by automated clients with an URL configured or hardcoded typically do not respect HSTS, so if you had a script that did ``curl http://pypi.python.org/simple/%60%60, it would silently get redirects to https and appear to “work”, but you wouldn’t get any of the security properties of TLS because an attacker would just intercept the request prior to the redirect happening.
Today I’ve backported this changed to the current production deployment of PyPI, which means that you can no longer access /simple/ and /packages/ over HTTP and you will have to directly go to HTTPS. For most people this should have no effect, because most tooling should be defaulting to HTTPS anyways, however if you’re using a significantly old version of tooling, it may still be defaulting to the HTTP url and will now stop functioning.
The recommended remediation is to upgrade your tooling to versions that support verified TLS connections and which default to the proper HTTPS URLs.
participants (5)
-
Alex Walters -
Donald Stufft -
Erik Bray -
Nate Coraor -
Wes Turner