On 30 Aug 2014 06:08, "Ethan Furman" <ethan@stoneleaf.us> wrote:

On 08/29/2014 01:00 PM, M.-A. Lemburg wrote:

...

On 29.08.2014 21:47, Alex Gaynor wrote:

...

I've just submitted PEP 476, on enabling certificate validation by

...

...

HTTPS clients in Python. Please have a look and let me know what you

default for think.

...

Thanks for the PEP. I think this is generally a good idea, but some important parts are missing from the PEP:

* transition plan:

I think starting with warnings in Python 3.5 and going for exceptions in 3.6 would make a good transition

Going straight for exceptions in 3.5 is not in line with our normal procedures for backwards incompatible changes.

* configuration:

It would be good to be able to switch this on or off without having to change the code, e.g. via a command line switch and environment variable; perhaps even controlling whether or not to raise an exception or warning.

* choice of trusted certificate:

Instead of hard wiring using the system CA roots into Python it would be good to just make this default and permit the user to point Python to a different set of CA roots.

This would enable using self signed certs more easily. Since these are often used for tests, demos and education, I think it's important to allow having more control of the trusted certs.

+1 for PEP with above changes.

Ditto from me. In relation to changing the Python CLI API to offer some of the wget/curl style command line options, I like the idea of providing recipes in the docs for implementing them at the application layer, but postponing making the *default* behaviour configurable that way. Longer term, I'd like to actually have a per-runtime configuration file for some of these things that also integrated with the pyvenv support, but that requires untangling the current startup code first (and there are only so many hours in the day). Regards, Nick.

-- ~Ethan~

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On Sun, 31 Aug 2014 03:25:25 +0200, Antoine Pitrou <solipsis@pitrou.net> wrote:

On Sun, 31 Aug 2014 09:26:30 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:

...

...

...

* configuration:

It would be good to be able to switch this on or off without having to change the code, e.g. via a command line switch and environment variable; perhaps even controlling whether or not to raise an exception or warning.

* choice of trusted certificate:

Instead of hard wiring using the system CA roots into Python it would be good to just make this default and permit the user to point Python to a different set of CA roots.

This would enable using self signed certs more easily. Since these are often used for tests, demos and education, I think it's important to allow having more control of the trusted certs.

+1 for PEP with above changes.

Ditto from me.

In relation to changing the Python CLI API to offer some of the wget/curl style command line options, I like the idea of providing recipes in the docs for implementing them at the application layer, but postponing making the *default* behaviour configurable that way.

I'm against any additional environment variables and command-line options. It will only complicate and obscure the security parameters of certificate validation.

The existing knobs have already been mentioned in this thread, I won't mention them here again.

Do those knobs allow one to instruct urllib to accept an invalid certificate without changing the program code? --David

On Aug 31, 2014, at 2:09 AM, Nick Coghlan <ncoghlan@gmail.com> wrote:

At the same time, we need to account for the fact that most existing organisations still trust in perimeter defence for their internal network security, and hence tolerate (or even actively encourage) the use of unsecured connections, or skipping certificate validation, internally. This is actually a really terrible idea, but it's still incredibly common due to the general failure of the technology industry to take usability issues seriously when we design security systems (at least until recently) - doing the wrong "unsafe" thing is genuinely easier than doing things right.

Just a quick clarification in order to be a little clearer, this change will (obviously) only effect those who trust perimeter security *and* decided to install an invalid certificate instead of just using HTTP. I'm not saying that this doesn't happen, just being specific (I'm not actually sure why they would install a TLS certificate at all if they are trusting perimeter security, but I'm sure folks do). --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 31 August 2014 07:45, Nick Coghlan <ncoghlan@gmail.com> wrote:

There's also the fact that most corporate Python users are unlikely to know that PyPI exists, let alone that it contains a module called "requests" that does SSL certificate validation by default. Those of us in the corporate world that interact directly with upstream are still the exception rather than the rule)

I think this point deserves just a little bit more emphasis. This is why any solution that begins with 'use PyPI' is insufficient. I've worked on requests for 3 years now and most of my colleagues have never heard of it, and it's not because I don't talk about it (I talk about it all the time!). When building internal tools, corporate environments frequently restrict themselves to the standard library. This is because it's hard enough to get adoption of a tool when it requires a new language runtime, let alone if you have to get people ramped up on package distribution as well! I have enough trouble getting people to upgrade Python versions at work: trying to get them up to speed on pip and PyPI is worse. It is no longer tenable in the long term for Python to trust the network: you're right in this regard Nick. In the past, on this very list, I've been bullish about fixing up Python's network security position. I was an aggressive supporter of PEP 466 (and there are some corners of PEP 466 that I think didn't go far enough). However, I'm with you here: we should do this once and do it right. Corporate users *will* bump into it, and they will look to the docs to fix it. That fix needs to be easy and painless. A user-level cert store is a good start, and if cryptography.io aren't interested in it I might take a look at implementing it under the certifi.io umbrella instead. Cory

On 31.08.2014 16:16, R. David Murray wrote:

Self -signed certificates are not crazy in an internal corporate environment even when properly playing the defense in depth game. Once you've acked the cert the first time, you will be warned if it changes (like an ssh host key). Sure, as Nick says the corp could set up an internal signing authority and make sure everyone has their CA...and they *should*...but realistically, that is probably relatively rare at the moment, because it is not particularly easy to accomplish (distributing the CA everywhere it needs to go is still a Hard Problem, though it has gotten a lot better).

It's very simple to trust a self-signed certificate: just download it and stuff it into the trust store. That's all. A self-signed certificate acts as its own root CA (so to speak). But there is a downside, too. The certificate is trusted for any and all connections. Python's SSL module has no way to trust a specific certificate for a host. Christian

Le 31/08/2014 19:03, Paul Moore a écrit :

On 31 August 2014 17:27, Christian Heimes <christian@python.org> wrote:

...

It's very simple to trust a self-signed certificate: just download it and stuff it into the trust store.

"Stuff it into the trust store" is the hard bit, though. I have honestly no idea how to do that.

You certainly shouldn't do so. If an application has special needs that require trusting a self-signed certificate, then it should expose a configuration setting to let users specify the cert's location. Stuffing self-signed certs into the system trust store is really a measure of last resort. There's another case which isn't solved by this, though, which is when a cert is invalid. The common situation being that it has expired (renewing certs is a PITA and therefore expired certs are more common than it sounds they should be). In this case, there is no way to whitelist it: you have to disable certificate checking altogether. This can be exposed by the application as configuration option if necessary, as well. Regards Antoine.

Le 31/08/2014 20:28, Paul Moore a écrit :

I can't see how that would be something the application would know. For example, pip allows me to specify an "alternate cert bundle" but not a single additional cert. So IIUC, I can't use my local index that serves https using a self-signed cert. I'd find it hard to argue that it's pip's responsibility to think of that use case - pretty much any program that interacts with a web service *might* need to interact with a self-signed dummy version, if only under test conditions.

Well, it's certainly pip's responsibility more than Python's. What would Python do? Provide a setting that would blindly add a cert for all uses of httplib? pip knows about the use cases here, Python doesn't. (perhaps you want to serve your local index using http, though) Regards Antoine.

Le 31/08/2014 21:12, Paul Moore a écrit :

On 31 August 2014 19:37, Antoine Pitrou <antoine@python.org> wrote:

...

Well, it's certainly pip's responsibility more than Python's. What would Python do? Provide a setting that would blindly add a cert for all uses of httplib?

That's more or less my point, pip doesn't have that much better idea than Python. I was talking about putting the cert in my local cert store, so that *I* can decide, and applications don't need to take special care to allow me to handle this case.

What do you call your local cert store? If you mean the system cert store, then that will affect all users. Regards Antoine.

On 1 Sep 2014 06:32, "Paul Moore" <p.f.moore@gmail.com> wrote:

On 31 August 2014 21:15, Antoine Pitrou <antoine@python.org> wrote:

...

What do you call your local cert store?

I was referring to Christian's comment

...

It's very simple to trust a self-signed certificate: just download it

and stuff it into the trust store.

From his recent response, I guess he meant the system store, and he agrees that this is a bad option.

OK, that's fair, but:

a) Is there really no OS-level personal trust store? I'm thinking of Windows here for my own personal use, but the same question applies elsewhere. b) I doubt my confusion over Christian's response is atypical. Based on what he said, if we hadn't had the subsequent discussion, I would probably have found a way to add a cert to "the store" without understanding the implications. While it's not Python's job to educate users, it would be a shame if its default behaviour led people to make ill-informed decisions.

Right, this is why I came to the conclusion we need to follow the browser vendors lead here and support a per-user Python specific supplementary certificate cache before we can start validating certs by default at the *Python* level. There are still too many failure modes for cert management on private networks for us to safely ignore the use case of needing to force connections to services with invalid certs. We don't need to *solve* that problem here today - we can push it back to Alex (and anyone else interested) as a building block to investigate providing as part of cryptography.io or certi.fi, with a view to making a standard library version of that (along with any SSL module updates) part of PEP 476. In the meantime, we can update the security considerations for the ssl module to make it clearer that the defaults are set up for trusted networks and that using it safely on the public internet may mean you're better off with a third party library like requests or Twisted. (I'll start another thread shortly that is highly relevant to that topic) Regards, Nick.

Maybe an SSL HOWTO would be a useful addition to the docs, if anyone feels motivated to write one.

Regardless, thanks for the education!

Paul _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On 31.08.2014 22:30, Paul Moore wrote:

On 31 August 2014 21:15, Antoine Pitrou <antoine@python.org> wrote:

...

What do you call your local cert store?

I was referring to Christian's comment

...

It's very simple to trust a self-signed certificate: just download it and stuff it into the trust store.

I was referring to the the trust store of the SSLContext object and not to any kind of cert store of the operating system. Sorry for the confusion.

a) Is there really no OS-level personal trust store? I'm thinking of Windows here for my own personal use, but the same question applies elsewhere.

Windows and OSX have superior cert stores compared to Linux and BSD. They have means for user and system wide cert stores and trust settings Linux just have one central directory or file with all trusted certs. My KDE has some options to disable certs but I don't know how to make use of the configuration. Even worse: Linux distros doesn't make a different between purposes. On Windows a user can trust a certificate for S/MIME but not for server auth or client auth. Ubuntu just puts all certification in one directory but it's wrong. :( https://bugs.launchpad.net/ubuntu/+source/ca-certificates/+bug/1207004 Christian

On 31.08.2014 08:09, Nick Coghlan wrote:

As Antoine says here, I'm also opposed to adding more Python specific configuration options. However, I think there may be something worthwhile we can do that's closer to the way browsers work, and has the significant benefit of being implementable as a PyPI module first (more on that in a separate reply).

I'm on your and Antoine's side and strictly against any additional environment variables or command line arguments. That would make the whole validation process even more complex and harder to understand. There might be a better option to give people and companies the option to tune the SSL module to their needs. Python already have a customization hook for the site module called sitecustomize. How about another module named sslcustomize? Such a module could be used to tune the ssl module to the needs of users, e.g. configure a different default context, add certificates to a default context etc. Companies could install them in a system global directory on their servers. Users could put them in their own user site directory and even each virtual env can have one sslcustomize of its own. It's fully backward compatible, doesn't add any flags and developers have the full power of Python for configuration and customization. Christian

On Mon, 01 Sep 2014 08:10:58 +1000, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 1 Sep 2014 07:43, "Christian Heimes" <christian@python.org> wrote:

...

On 31.08.2014 08:09, Nick Coghlan wrote:

...

As Antoine says here, I'm also opposed to adding more Python specific configuration options. However, I think there may be something worthwhile we can do that's closer to the way browsers work, and has the significant benefit of being implementable as a PyPI module first (more on that in a separate reply).

I'm on your and Antoine's side and strictly against any additional environment variables or command line arguments. That would make the whole validation process even more complex and harder to understand.

There might be a better option to give people and companies the option to tune the SSL module to their needs. Python already have a customization hook for the site module called sitecustomize. How about another module named sslcustomize? Such a module could be used to tune the ssl module to the needs of users, e.g. configure a different default context, add certificates to a default context etc.

Companies could install them in a system global directory on their servers. Users could put them in their own user site directory and even each virtual env can have one sslcustomize of its own. It's fully backward compatible, doesn't add any flags and developers have the full power of Python for configuration and customization.

And means a user specific store (if one became available) could be configured there.

Yes, I think this would address my concerns, especially if combined with a clear recipe in the documentation on how to optionally disable cert validation at the application layer.

Assuming sslcustomize was in site-packages rather than the standard library directories, you would also be able to use virtual environments with an appropriate sslcustomize module to disable cert checking even if the application you were running didn't support direct configuration.

It sounds like this would address my concerns as well (I don't really care *how* it is implemented as long as I don't have to touch the code of a third party application when I upgrade my python version to 3.5...remember, the context here is backward compatibility concerns). Does it address the issue of accepting an invalid cert, though? --David

On 1 Sep 2014 16:05, "Nick Coghlan" <ncoghlan@gmail.com> wrote:

The final change would be to seed the context factory map appropriately for the standard library modules where we wanted to keep the *old* default:

for modname in ("nntplib", "poplib", "imaplib", "ftplib", "smtplib", "asyncio.selector_events", "urllib.request", "http.client"): named_contexts[modname] = create_legacy_context

The list I have above is for *all* current uses of "sss._create_stdlib_context". The backwards incompatible part of PEP 476 would then just be about removing names from that list (currently just "http.client", but I'd suggest "asyncio.selector_events" as another candidate, taking advantage of asyncio's provisional API status).

Update on this: Christian & I both like the SSL named context and customisation idea independently of the proposal to change the default behaviour, so we're going to write it up as a separate PEP. Changing the default behaviour would then be reduced to a proposal to take a couple of module names off the "uses legacy SSL settings" list. Cheers, Nick.

On 1 September 2014 16:07, Paul Moore <p.f.moore@gmail.com> wrote:

On 31 August 2014 23:10, Nick Coghlan <ncoghlan@gmail.com> wrote:

...

Assuming sslcustomize was in site-packages rather than the standard library directories, you would also be able to use virtual environments with an appropriate sslcustomize module to disable cert checking even if the application you were running didn't support direct configuration.

Would this mean that a malicious package could install a custom sslcustomize.py and so add unwanted certs to the system? I guess we have to assume that installed packages are trusted, but I just wanted to be explicit.

Yes, it would have exactly the same security failure modes as sitecustomize, except it would only fire if the application imported the ssl module. The "-S" and "-I" switches would need to disable the implied "sslcustomize", just as they disable "import site". Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On 1 September 2014 17:13, Christian Heimes <christian@python.org> wrote:

On 01.09.2014 08:44, Nick Coghlan wrote:

...

Yes, it would have exactly the same security failure modes as sitecustomize, except it would only fire if the application imported the ssl module.

The "-S" and "-I" switches would need to disable the implied "sslcustomize", just as they disable "import site".

A malicious package can already play havoc with your installation with a custom ssl module. If somebody is able to sneak in a ssl.py then you are screwed anyway. sslcustomize is not going to make the situation worse.

That's not quite true - we're fairly careful about putting the standard library before userspace directories, so aside from the "current directory" problem, shadowing "ssl" itself can be tricky to arrange. "sslcustomize" would be more like "sitecustomize" - since it wouldn't normally be in the standard library, it can appear anywhere on sys.path, rather than having to be injected ahead of the standard library. I think that's OK though - compared to the security nightmare that is downloading modules from PyPI and running "./setup.py install" (or, even worse, "sudo ./setup.py install"), this would be a rather esoteric attack vector, and the existing -S and -I mechanisms could be used to defend against it :) Cheers, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Mon, Sep 1, 2014 at 10:41 PM, Antoine Pitrou <antoine@python.org> wrote:

Not sure why. Just put another module named "ssl" in sys.modules directly. You can also monkeypatch the genuine ssl module.

That has to be done inside the same process. But imagine this scenario: You have a program that gets invoked as root (or some other user than yourself), and you're trying to fiddle with what it sees. You don't have root access, but you can manipulate the file system, to the extent that your userid has access. What can you do to affect this other program? ChrisA

On Mon, Sep 1, 2014 at 11:34 PM, Antoine Pitrou <solipsis@pitrou.net> wrote:

On Mon, 1 Sep 2014 23:24:39 +1000 Chris Angelico <rosuav@gmail.com> wrote:

...

On Mon, Sep 1, 2014 at 10:41 PM, Antoine Pitrou <antoine@python.org> wrote:

...

Not sure why. Just put another module named "ssl" in sys.modules directly. You can also monkeypatch the genuine ssl module.

That has to be done inside the same process. But imagine this scenario: You have a program that gets invoked as root (or some other user than yourself), and you're trying to fiddle with what it sees. You don't have root access, but you can manipulate the file system, to the extent that your userid has access. What can you do to affect this other program?

If you're root you shouldn't run untrusted code. See https://docs.python.org/3/using/cmdline.html#cmdoption-I

Right, which is why sslcustomize has to be controlled by that, but the possibility of patching (or monkeypatching) ssl.py isn't as big a deal. ChrisA

On 2 Sep 2014 00:08, "Antoine Pitrou" <solipsis@pitrou.net> wrote:

On Mon, 1 Sep 2014 23:42:10 +1000 Chris Angelico <rosuav@gmail.com> wrote:

...

...

...

That has to be done inside the same process. But imagine this scenario: You have a program that gets invoked as root (or some other user than yourself), and you're trying to fiddle with what it sees. You don't have root access, but you can manipulate the file system,

...

...

...

the extent that your userid has access. What can you do to affect

to this

...

...

...

other program?

If you're root you shouldn't run untrusted code. See https://docs.python.org/3/using/cmdline.html#cmdoption-I

Right, which is why sslcustomize has to be controlled by that, but the possibility of patching (or monkeypatching) ssl.py isn't as big a deal.

To be frank I don't understand what you're arguing about.

When I said "shadowing ssl can be tricky to arrange", Chris correctly interpreted it as referring to the filesystem based privilege escalation scenario that isolated mode handles, not to normal in-process monkeypatching or module injection. I don't consider the latter cases to be interesting attack scenarios, as they imply the attacker is *already* running arbitrary Python code inside your CPython process, so you've already lost. Cheers, Nick.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On 2 Sep 2014 00:59, "Antoine Pitrou" <solipsis@pitrou.net> wrote:

On Tue, 2 Sep 2014 00:53:11 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:

...

...

To be frank I don't understand what you're arguing about.

When I said "shadowing ssl can be tricky to arrange", Chris correctly interpreted it as referring to the filesystem based privilege escalation scenario that isolated mode handles, not to normal in-process monkeypatching or module injection.

There's no actual difference. You can have a sitecustomize.py that does the monkeypatching or the shadowing. There doesn't seem to be anything "tricky" about that.

Oh, now I get what you mean - yes, sitecustomize already poses the same kind of problem as the proposed sslcustomize (hence the existence of the related command line options). I missed that you had switched to talking about using that attack vector, rather than trying to shadow stdlib modules directly through the filesystem (which is the only tricky thing I was referring to). Cheers, Nick.

On 01.09.2014 17:35, Nick Coghlan wrote:

Oh, now I get what you mean - yes, sitecustomize already poses the same kind of problem as the proposed sslcustomize (hence the existence of the related command line options).

If an attacker is able to place a module like sitecustomize.py in an import directory or any .pth file in a site-packages directory than this Python installation is compromised. .pth files are insidious because they are always loaded and their code is always executed. I don't see how sslcustomize is going to make a difference here.

On 2 Sep 2014 03:08, "Donald Stufft" <donald@stufft.io> wrote:

...

On Sep 1, 2014, at 1:01 PM, Christian Heimes <christian@python.org>

wrote:

...

On 01.09.2014 17:35, Nick Coghlan wrote:

...

Oh, now I get what you mean - yes, sitecustomize already poses the same kind of problem as the proposed sslcustomize (hence the existence of the related command line options).

If an attacker is able to place a module like sitecustomize.py in an import directory or any .pth file in a site-packages directory than this Python installation is compromised. .pth files are insidious because they are always loaded and their code is always executed. I don't see how sslcustomize is going to make a difference here.

Right, this is the point I was trying to make. If you’ve installed a malicious package it’s game over. There’s nothing Python can do to help you.

Yes, that's what I said originally when pointing out that isolated mode and the switch to disable site module processing would need to disable sslcustomize processing as well. Antoine was replying to a side comment about it being tricky to shadow stdlib modules. I left out the qualifier "directly" in my original comment, and he left out "indirectly through sitecustomize" in his initial reply, so we were talking past each for a while. Cheers, Nick.

--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Aug 31, 2014, at 5:43 PM, Christian Heimes <christian@python.org> wrote:

On 31.08.2014 08:09, Nick Coghlan wrote:

...

As Antoine says here, I'm also opposed to adding more Python specific configuration options. However, I think there may be something worthwhile we can do that's closer to the way browsers work, and has the significant benefit of being implementable as a PyPI module first (more on that in a separate reply).

I'm on your and Antoine's side and strictly against any additional environment variables or command line arguments. That would make the whole validation process even more complex and harder to understand.

There might be a better option to give people and companies the option to tune the SSL module to their needs. Python already have a customization hook for the site module called sitecustomize. How about another module named sslcustomize? Such a module could be used to tune the ssl module to the needs of users, e.g. configure a different default context, add certificates to a default context etc.

Companies could install them in a system global directory on their servers. Users could put them in their own user site directory and even each virtual env can have one sslcustomize of its own. It's fully backward compatible, doesn't add any flags and developers have the full power of Python for configuration and customization.

This may be a dumb question, but why can’t sitecustomize do this already? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 31 August 2014 12:21, R. David Murray <rdmurray@bitdance.com> wrote:

Do those knobs allow one to instruct urllib to accept an invalid certificate without changing the program code?

My first reply ended up being a context dump of the challenges created by legacy corporate intranets that may not be immediately obvious to folks that spend most of their time working on or with the public internet. I decided to split these more technical details out to a new reply for the benefit of folks that already know all that history :) To answer David's specific question, the existing knobs at the OpenSSL level (SSL_CERT_DIR and SSL_CERT_FILE ) let people add an internal CA, opt out of the default CA system, and trust *specific* self-signed certs. What they don't allow is a global "trust any cert" setting - exceptions need to be added at the individual cert level or at the CA level, or the application needs to offer an option to not do cert validation at all. That "trust anything" option at the platform level is the setting that is a really bad idea - if an organisation thinks it needs that (because they have a lot of self-signed certs, but aren't verifying their HTTPS connections to those servers), then what they really need is an internal CA, where their systems just need to be set up to trust the internal CA in addition to the platform CA certs. With Alex's proposal, organisations that are already running an internal CA should be just fine - Python 3.5 will see the CA cert in the platform cert store and accept certs signed by it as valid. (Note: the Python 3.4 warning should take this into account, which could be a problem since we don't currently do validity checks against the platform store by default. The PEP needs to cover the mechanics of that in more detail, as I think it means we'll need to make *some* changes to the default configuration even in Python 3.4 to get accurate validity data back from OpenSSL) However, we also need to accept that there's a reason browser vendors still offer "click through insecurity" for sites with self-signed certificates, and tools like wget/curl offer the option to say "don't check the certificate": these are necessary compromises to make SSL based network connections actually work on many current corporate intranets. It is corporate environments that also make it desirable to be able to address this potential problem at a *user* level, since many Python users in a large organisations are actually running Python entirely out of their home directories, rather than as a system installation (they may not even have admin access to their own systems). My suggestion at this point is that we take a leaf from both browser vendors and the design of SSH: make it easy to *add* a specific self-signed cert to the set a *particular user* trusts by default (preferably *only* for a particular host, to limit the power of such certs). "python -m ssl" doesn't currently do anything interesting, so it could be used to provide an API for managing that user level certificate store. A Python-specific user level cert store is something that could be developed as a PyPI library for Python 2.7.9+ and 3.4+ (Is cert management considered in scope for cryptography.io? If so, that could be a good home). So while I agree with the intent of PEP 476, and like the suggested end state, I'm back to thinking that the transition plan for existing corporate users needs more work before it can be accepted. This is especially true since it becomes another barrier to migrating from Python 2.7 to Python 3.5+ (a warning in Python 3.4 doesn't help with that aspect, although a new -3 warning might). A third party module that offers a user level certificate store, and a gevent.monkey style way of opting in to this behaviour for existing Python versions would be one way to provide a more compelling transition plan. Regards, Nick. -- Nick Coghlan | ncoghlan@gmail.com | Brisbane, Australia

On Aug 29, 2014, at 4:00 PM, "M.-A. Lemburg" <mal@egenix.com> wrote:

* choice of trusted certificate:

Instead of hard wiring using the system CA roots into Python it would be good to just make this default and permit the user to point Python to a different set of CA roots.

This would enable using self signed certs more easily. Since these are often used for tests, demos and education, I think it's important to allow having more control of the trusted certs.

If I recall OpenSSL already allows this to be configured via envvar and the python API already allows it to be configured via API.

On Fri, 29 Aug 2014 17:11:35 -0400, Donald Stufft <donald@stufft.io> wrote:

Sorry I was on my phone and didn’t get to fully reply to this.

...

On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg <mal@egenix.com> wrote:

* configuration:

It would be good to be able to switch this on or off without having to change the code, e.g. via a command line switch and environment variable; perhaps even controlling whether or not to raise an exception or warning.

I’m on the fence about this, if someone provides a certificate that we can validate against (which can be done without touching the code) then the only thing that really can’t be “fixed” without touching the code is if someone has a certificate that is otherwise invalid (expired, not yet valid, wrong hostname, etc). I’d say if I was voting on this particular thing I’d be -0, I’d rather it didn’t exist but I wouldn’t cry too much if it did.

Especially if you want an accelerated change, there must be a way to *easily* get back to the previous behavior, or we are going to catch a lot of flack. There may be only 7% of public certs that are problematic, but I'd be willing to bet you that there are more not-really-public ones that are critical to day to day operations *somewhere* :) wget and curl have 'ignore validation' as a command line flag for a reason. --David

On Aug 29, 2014, at 5:42 PM, R. David Murray <rdmurray@bitdance.com> wrote:

On Fri, 29 Aug 2014 17:11:35 -0400, Donald Stufft <donald@stufft.io> wrote:

...

Sorry I was on my phone and didn’t get to fully reply to this.

...

On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg <mal@egenix.com> wrote:

* configuration:

It would be good to be able to switch this on or off without having to change the code, e.g. via a command line switch and environment variable; perhaps even controlling whether or not to raise an exception or warning.

I’m on the fence about this, if someone provides a certificate that we can validate against (which can be done without touching the code) then the only thing that really can’t be “fixed” without touching the code is if someone has a certificate that is otherwise invalid (expired, not yet valid, wrong hostname, etc). I’d say if I was voting on this particular thing I’d be -0, I’d rather it didn’t exist but I wouldn’t cry too much if it did.

Especially if you want an accelerated change, there must be a way to *easily* get back to the previous behavior, or we are going to catch a lot of flack. There may be only 7% of public certs that are problematic, but I'd be willing to bet you that there are more not-really-public ones that are critical to day to day operations *somewhere* :)

wget and curl have 'ignore validation' as a command line flag for a reason.

Right, that’s why I’m on the fence :) On one hand, it’s going to break things for some people, (arguably they are already broken, just silently so, but we’ll leave that argument aside) and a way to get back the old behavior is good. There are already ways within the Python code itself, so that’s covered. From outside of the Python code there are ways if the certificate is untrusted but otherwise valid which are pretty easy to do. The major “gap” is when you have an actual invalid certificate due to expiration or hostname or some other such thing. On the other hand Python is not wget/curl and the people who are most likely to be the target for a “I can’t change the code but I need to get the old behavior back” are people who are likely to not be invoking Python itself but using something written in Python which happens to be using Python. IOW they might be executing “foobar” not “python -m foobar”. Like I said though, I’m personally fine either way so don’t take this as being against that particular change! --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On Fri, 29 Aug 2014 17:11:35 -0400 Donald Stufft <donald@stufft.io> wrote:

Another problem with this is that I don’t think it’s actually possible to do. Python itself isn’t validating the TLS certificates, OpenSSL is doing that. To my knowledge OpenSSL doesn’t have a way to say “please validate these certificates and if they don’t validate go ahead and keep going and just let me get a warning from it”.

Actually, there may be a solution. In client mode, OpenSSL always verifies the server cert chain and stores the verification result in the SSL structure. It will then only report an error if the verify mode is not SSL_VERIFY_NONE. (see ssl3_get_server_certificate() in s3_clnt.c) The verification result should then be readable using SSL_get_verify_result(), even with SSL_VERIFY_NONE. (note this is only from reading the source code and needs verifying) Then we could have the following transition phase: - define a new CERT_WARN value for SSLContext.verify_mode - use that value as the default in the HTTP stack (people who want the old silent default will have to set verify_mode explicitly to VERIFY_NONE) - with CERT_WARN, SSL_VERIFY_NONE is passed to OpenSSL and Python manually calls SSL_get_verify_result() after a handshake; if there was a verification error, a warning is printed out And in the following version we switch the HTTP default to CERT_REQUIRED. Regards Antoine.

On Aug 29, 2014, at 5:58 PM, M.-A. Lemburg <mal@egenix.com> wrote:

On 29.08.2014 23:11, Donald Stufft wrote:

...

Sorry I was on my phone and didn’t get to fully reply to this.

...

On Aug 29, 2014, at 4:00 PM, M.-A. Lemburg <mal@egenix.com> wrote:

On 29.08.2014 21:47, Alex Gaynor wrote:

...

Hi all,

I've just submitted PEP 476, on enabling certificate validation by default for HTTPS clients in Python. Please have a look and let me know what you think.

PEP text follows.

Thanks for the PEP. I think this is generally a good idea, but some important parts are missing from the PEP:

* transition plan:

I think starting with warnings in Python 3.5 and going for exceptions in 3.6 would make a good transition

Going straight for exceptions in 3.5 is not in line with our normal procedures for backwards incompatible changes.

As far as a transition plan, I think that this is an important enough thing to have an accelerated process. If we need to provide a warning than let’s add it to the next 3.4 otherwise it’s going to be 2.5+ years until we stop being unsafe by default.

Fine with me; we're still early in the Python 3.4 patch level releases.

...

Another problem with this is that I don’t think it’s actually possible to do. Python itself isn’t validating the TLS certificates, OpenSSL is doing that. To my knowledge OpenSSL doesn’t have a way to say “please validate these certificates and if they don’t validate go ahead and keep going and just let me get a warning from it”. It’s a 3 way switch, no validation, validation if a certificate is provided, and validation always.

Now that’s strictly for the “verify the certificate chain” portion, the hostname verification is done entirely on our end and we could do something there… but I’m not sure it makes sense to do so if we can’t do it for invalid certificates too.

OpenSSL provides a callback for certificate validation, so it is possible to issue a warning and continue with accepting the certificate.

Ah right, I forgot about that. I was thinking in terms of CERT_NONE, CERT_OPTIONAL, CERT_REQUIRED. I think it’s fine to add a warning if possible to Python 3.4, I just couldn’t think off the top of my head a way of doing it.

...

...

* configuration:

It would be good to be able to switch this on or off without having to change the code, e.g. via a command line switch and environment variable; perhaps even controlling whether or not to raise an exception or warning.

I’m on the fence about this, if someone provides a certificate that we can validate against (which can be done without touching the code) then the only thing that really can’t be “fixed” without touching the code is if someone has a certificate that is otherwise invalid (expired, not yet valid, wrong hostname, etc). I’d say if I was voting on this particular thing I’d be -0, I’d rather it didn’t exist but I wouldn’t cry too much if it did.

If you're testing code or trying out some new stuff, you don't want to get a valid cert first, but instead go ahead with a self signed one. That's the use case.

...

...

* choice of trusted certificate:

Instead of hard wiring using the system CA roots into Python it would be good to just make this default and permit the user to point Python to a different set of CA roots.

This would enable using self signed certs more easily. Since these are often used for tests, demos and education, I think it's important to allow having more control of the trusted certs.

Like my other email said, the Python API has everything needed to easily specify your own CA roots and/or disable the validations. The OpenSSL library also allows you to specify either a directory or a file to change the root certificates without code changes. The only real problems with the APIs are that the default is bad and an unrelated thing where you can’t pass in an in memory certificate.

Are you sure that's possible ? Python doesn't load the openssl.cnf file and the SSL_CERT_FILE, SSL_CERT_DIR env vars only work for the openssl command line binary, AFAIK.

I’m not 100% sure on that. I know they are not limited to the command line binary as ruby uses those environment variables in the way I described above. I do not believe that Ruby has done anything special to enable the use of those variables. It’s possible we’re doing something differently that bypasses those variables though. If that is the case then yes let’s add it, ideally doing whatever it needs to be to make OpenSSL respect those variables, or else respecting them ourselves.

In any case, Python will have to tap into the OS CA root provider using special code and this code could easily be made to check other dirs/files as well.

The point is that it should be possible to change this default at the Python level, without needing application code changes.

Ok, I’m not opposed to it FWIW. Just sayiing I’m pretty sure those things already exist in the form of environment variables and at the python level APIs. Not sure what else there is, global state for the “default”? A CLI flag? --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

On 30.08.2014 00:22, Antoine Pitrou wrote:

SSL_CERT_DIR and SSL_CERT_FILE are used, if set, when SSLContext.load_verify_locations() is called.

Actually, come to think of it, this allows us to write a better test for that method. Patch welcome!

The environment vars are used only when SSLContext.set_default_verify_paths() is called. load_verify_locations() loads certificates from a given file, directory or memory but it doesn't look at the env vars. create_default_context() calls SSLContext.load_default_certs() when neither cafile, capath nor cadata is given as an argument. SSLContext.load_default_certs() then calls SSLContext.set_default_verify_paths(). However there is a catch: SSLContext.set_default_verify_paths() is not called on Windows. In retrospective it was a bad decision by me to omit the call. http://hg.python.org/cpython/file/164a17eca081/Lib/ssl.py#l376 Christian PS: SSL_CERT_DIR and SSL_CERT_FILE are the default names. It's possible to change the names in OpenSSL. ssl.get_default_verify_paths() returns the names and paths to the default verify locations.

On 30.08.2014 04:44, Alex Gaynor wrote:

Thanks for the rapid feedback everyone!

I want to summarize the action items and discussion points that have come up so far:

To add to the PEP:

* Emit a warning in 3.4.next for cases that would raise a Exception in 3.5 * Clearly state that the existing OpenSSL environment variables will be respected for setting the trust root

I'd also suggest to compile Python with OPENSSL_LOAD_CONF, since that causes OpenSSL to read the global openssl.cnf file for additional configuration.

Discussion points:

* Disabling verification entirely externally to the program, through a CLI flag or environment variable. I'm pretty down on this idea, the problem you hit is that it's a pretty blunt instrument to swing, and it's almost impossible to imagine it not hitting things it shouldn't; it's far too likely to be used in applications that make two sets of outbound connections: 1) to some internal service which you want to disable verification on, and 2) some external service which needs strong validation. A global flag causes the latter to fail silently when subjected to a MITM attack, and that's exactly what we're trying to avoid. It also makes things much harder for library authors: I write an API client for some API, and make TLS connections to it. I want those to be verified by default. I can't even rely on the httplib defaults, because someone might disable them from the outside.

The reasoning here is the same as for hash randomization. There are cases where you want to test your application using self-signed certificates which don't validate against the system CA root list. In those cases, you do know what you're doing. The test would fail otherwise and the reason is not a bug in your code, it's just the fact that the environment you're running it in is a test environment. Ideally, all applications should give you this choice, but this is unlikely to happen, so it's good to be able to change the Python default, since with the proposed change, most applications will probably continue to use the Python defaults as they do now. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 30 2014)

...

...

Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2014-08-27: Released eGenix PyRun 2.0.1 ... http://egenix.com/go62 2014-09-19: PyCon UK 2014, Coventry, UK ... 20 days to go 2014-09-27: PyDDF Sprint 2014 ... 28 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On 30.08.2014 12:40, Antoine Pitrou wrote:

On Sat, 30 Aug 2014 12:19:11 +0200 "M.-A. Lemburg" <mal@egenix.com> wrote:

...

...

To add to the PEP:

* Emit a warning in 3.4.next for cases that would raise a Exception in 3.5 * Clearly state that the existing OpenSSL environment variables will be respected for setting the trust root

I'd also suggest to compile Python with OPENSSL_LOAD_CONF, since that causes OpenSSL to read the global openssl.cnf file for additional configuration.

Python links against OpenSSL as a shared library, not statically. It's unlikely that setting a compile constant inside Python would affect OpenSSL at all.

The change is to the OpenSSL API, not the OpenSSL lib. By setting the variable you enable a few special calls to the config loader functions in OpenSSL when calling the initializer it: https://www.openssl.org/docs/crypto/OPENSSL_config.html

...

...

Discussion points:

* Disabling verification entirely externally to the program, through a CLI flag or environment variable. I'm pretty down on this idea, the problem you hit is that it's a pretty blunt instrument to swing, and it's almost impossible to imagine it not hitting things it shouldn't; it's far too likely to be used in applications that make two sets of outbound connections: 1) to some internal service which you want to disable verification on, and 2) some external service which needs strong validation. A global flag causes the latter to fail silently when subjected to a MITM attack, and that's exactly what we're trying to avoid. It also makes things much harder for library authors: I write an API client for some API, and make TLS connections to it. I want those to be verified by default. I can't even rely on the httplib defaults, because someone might disable them from the outside.

The reasoning here is the same as for hash randomization. There are cases where you want to test your application using self-signed certificates which don't validate against the system CA root list.

That use case should be served with the SSL_CERT_DIR and SSL_CERT_FILE env vars (or, better, by specific settings *inside* the application).

I'm against multiplying environment variables, as it makes it more difficult to assess the actual security of a setting. The danger of an ill-secure setting is much more severe than with hash randomization.

You have a point there. So how about just a python run-time switch and no env var ? -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 30 2014)

...

...

Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2014-08-27: Released eGenix PyRun 2.0.1 ... http://egenix.com/go62 2014-09-19: PyCon UK 2014, Coventry, UK ... 20 days to go 2014-09-27: PyDDF Sprint 2014 ... 28 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On 30.08.2014 12:55, Antoine Pitrou wrote:

On Sat, 30 Aug 2014 12:46:47 +0200 "M.-A. Lemburg" <mal@egenix.com> wrote:

...

The change is to the OpenSSL API, not the OpenSSL lib. By setting the variable you enable a few special calls to the config loader functions in OpenSSL when calling the initializer it:

https://www.openssl.org/docs/crypto/OPENSSL_config.html

Ah, ok. Do you have experience with openssl.cnf? Apparently, it is meant for offline tools such as certificate generation, I am not sure how it could impact certification validation.

I'm still exploring this: the OpenSSL documentation is, well, less than complete on these things, so searching mailing lists and reading source code appears to be the only reasonable way to figure out what is possible and what not. The openssl.cnf config file is indeed mostly used by the various openssl subcommands (e.g. req and ca), but it can also be used to configuring engines and my hope is that configuration of e.g. default certificate stores also becomes possible. One of the engines can tap into the Windows certificate store, for example.

...

...

That use case should be served with the SSL_CERT_DIR and SSL_CERT_FILE env vars (or, better, by specific settings *inside* the application).

I'm against multiplying environment variables, as it makes it more difficult to assess the actual security of a setting. The danger of an ill-secure setting is much more severe than with hash randomization.

You have a point there. So how about just a python run-time switch and no env var ?

Well, why not, but does it have a value over letting the code properly configure their SSLContext?

Yes, because when Python changes the default to be validating and more secure, application developers will do the same as they do now: simply use the defaults ;-) -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 30 2014)

...

...

Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2014-08-27: Released eGenix PyRun 2.0.1 ... http://egenix.com/go62 2014-09-19: PyCon UK 2014, Coventry, UK ... 20 days to go 2014-09-27: PyDDF Sprint 2014 ... 28 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On 30.08.2014 15:32, R. David Murray wrote:

On Sat, 30 Aug 2014 14:03:57 +0200, "M.-A. Lemburg" <mal@egenix.com> wrote:

...

On 30.08.2014 12:55, Antoine Pitrou wrote:

...

On Sat, 30 Aug 2014 12:46:47 +0200 "M.-A. Lemburg" <mal@egenix.com> wrote:

...

...

That use case should be served with the SSL_CERT_DIR and SSL_CERT_FILE env vars (or, better, by specific settings *inside* the application).

I'm against multiplying environment variables, as it makes it more difficult to assess the actual security of a setting. The danger of an ill-secure setting is much more severe than with hash randomization.

You have a point there. So how about just a python run-time switch and no env var ?

Well, why not, but does it have a value over letting the code properly configure their SSLContext?

Yes, because when Python changes the default to be validating and more secure, application developers will do the same as they do now: simply use the defaults ;-)

But neither of those addresses the articulated use case: someone *using* a program implemented in python that does not itself provide a way to disable the new default security (because it is *new*). Only an environment variable will do that.

Since the environment variable is opt-in, I think the "consenting adults" argument applies to Alex's demure about "multiple connections". It could still emit the warnings.

That would be a possibility as well, yes. I'd just like to see a way to say: I know what I'm doing and I'm not in the mood to configure my own CA list, so please go ahead and accept whatever certs you find -- much like what --no-check-certificate does for wget. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Source (#1, Aug 30 2014)

...

...

Python Projects, Consulting and Support ... http://www.egenix.com/ mxODBC.Zope/Plone.Database.Adapter ... http://zope.egenix.com/ mxODBC, mxDateTime, mxTextTools ... http://python.egenix.com/

2014-08-27: Released eGenix PyRun 2.0.1 ... http://egenix.com/go62 2014-09-19: PyCon UK 2014, Coventry, UK ... 20 days to go 2014-09-27: PyDDF Sprint 2014 ... 28 days to go eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/

On Aug 30, 2014, at 12:19 PM, M.-A. Lemburg wrote:

The reasoning here is the same as for hash randomization. There are cases where you want to test your application using self-signed certificates which don't validate against the system CA root list.

In those cases, you do know what you're doing. The test would fail otherwise and the reason is not a bug in your code, it's just the fact that the environment you're running it in is a test environment.

Exactly. I have test cases where I have to load up a self-signed cert via .load_cert_chain() and in the good-path tests, I expect to make successful https connections. I also have test cases that expect to fail when: * I load bogus self-signed certs * I have an http server masquerading as an https server * I load an expired self-signed cert It certainly makes sense for the default to be the most secure, but other use cases must be preserved. Cheers, -Barry

The Windows certificate store is used by ``load_default_certs``: * https://github.com/python/cpython/blob/master/Lib/ssl.py#L379-L381 * https://docs.python.org/3.4/library/ssl.html#ssl.enum_certificates Cheers, Alex

Paul Moore <p.f.moore@gmail.com>:

Cool, in which case this sounds like a good plan. I have no particular opinion on whether there should be a global Python-level "don't check certificates" option, but I would suggest that the docs include a section explaining how a user can implement a "--no-check-certificates" flag in their program if they want to (with appropriate warnings as to the risks, of course!). Better to explain how to do it properly than to say "you shouldn't do that" and have developers implement awkward or incorrect hacks in spite of the advice.

Will there be a way to specify a particular CA certificate (as in "wget --ca-certificate")? Will there be a way to specify a particular CA certificate directory (as in "wget --ca-directory")? Marko

Zitat von Christian Heimes <christian@python.org>:

On 30.08.2014 17:22, Alex Gaynor wrote:

...

The Windows certificate store is used by ``load_default_certs``:

* https://github.com/python/cpython/blob/master/Lib/ssl.py#L379-L381 * https://docs.python.org/3.4/library/ssl.html#ssl.enum_certificates

The Windows part of load_default_certs() has one major flaw: it can only load certificates that are already in Windows's cert store. However Windows comes only with a small set of default certs and downloads more certs on demand. In order to trigger a download Python or OpenSSL would have to use the Windows API to verify root certificates.

It's better than you think. Vista+ has a weekly prefetching procedure that should assure that virtually all root certificates are available: http://support.microsoft.com/kb/931125/en-us BTW, it's patented: http://www.google.de/patents/US6816900 Regards, Martin

On Aug 29, 2014, at 7:44 PM, Alex Gaynor <alex.gaynor@gmail.com> wrote:

Disabling verification entirely externally to the program, through a CLI flag or environment variable. I'm pretty down on this idea, the problem you hit is that it's a pretty blunt instrument to swing, and it's almost impossible to imagine it not hitting things it shouldn't; it's far too likely to be used in applications that make two sets of outbound connections: 1) to some internal service which you want to disable verification on, and 2) some external service which needs strong validation. A global flag causes the latter to fail silently when subjected to a MITM attack, and that's exactly what we're trying to avoid. It also makes things much harder for library authors: I write an API client for some API, and make TLS connections to it. I want those to be verified by default. I can't even rely on the httplib defaults, because someone might disable them from the outside.

I would strongly recommend against such a mechanism. For what it's worth, Twisted simply unconditionally started verifying certificates in 14.0 with no "disable" switch, and (to my knowledge) literally no users have complained. Twisted has a very, very strict backwards compatibility policy. For example, I once refused to accept the deletion of a class that raised an exception upon construction, on the grounds that someone might have been inadvertently importing that class, and they shouldn't see an exception until they've seen a deprecation for one release. Despite that, we classified failing to verify certificates as a security bug, and fixed it with no deprecation period. When users type the 's' after the 'p' and before the ':' in a URL, they implicitly expect browser-like certificate verification. The lack of complaints is despite the fact that 14.0 has been out for several months now, and, thanks to the aforementioned strict policy, users tend to upgrade fairly often (since they know they can almost always do so without fear of application-breaking consequences). According to PyPI metadata, 14.0.0 has had 273283 downloads so far. Furthermore, "disable verification" is a nonsensical thing to do with TLS. "select a trust root" is a valid configuration option, and OpenSSL already provides it via the SSL_CERT_DIR environment variable, so there's no need for Python to provide anything beyond that. -glyph

Antoine Pitrou <solipsis <at> pitrou.net> writes:

And how many people are using Twisted as an HTTPS client? (compared to e.g. Python's httplib, and all the third-party libraries building on it?)

I don't think anyone could give an honest estimate of these counts, however there's two factors to bare in mind: a) It's extremely strongly recommended to use requests to make any HTTP requests precisely because httplib is negligent in certificate and hostname checking by default, b) We're talking about Python3, which has fewer users than Python2.

...

Furthermore, "disable verification" is a nonsensical thing to do with TLS.

It's not. For example, if you have an expired cert, all you can do AFAIK is to disable verification.

It really is a nonsensical operation, accepting any random TLS certificate without pinning or use of a certificate authorities makes a valid connection completely indistinguishable from a MITM attack. If I were the emperor of the universe (or even just Python ;-)) I wouldn't allow this operation at all, however, I'm not and you can still disable any and all verification. It just requires you to pass a different argument, which doesn't seem overly burdensome. This whole scenario seems to be predicated on a siutation where: You have a peer whose certificate you can't change, and you have a piece of code you can't change, and you're going to upgrade your Python installation, and you want to talk to this peer, and you need to use an encrypted channel, but don't really care if it's being MITM'd. It doesn't seem to me that this is reasonably Python's responsibility to deal with the fact that you have no ability to upgrade any of your infrastructure, except your Python version. Alex

On 3 Sep 2014 08:18, "Alex Gaynor" <alex.gaynor@gmail.com> wrote:

Antoine Pitrou <solipsis <at> pitrou.net> writes:

...

And how many people are using Twisted as an HTTPS client? (compared to e.g. Python's httplib, and all the third-party libraries building on it?)

I don't think anyone could give an honest estimate of these counts,

however

there's two factors to bare in mind: a) It's extremely strongly recommended to use requests to make any HTTP requests precisely because httplib is negligent in certificate and hostname checking by default, b) We're talking about Python3, which has fewer users than Python2.

Creating *new* incompatibilities between Python 2 & Python 3 is a major point of concern. One key focus of 3.5 is *reducing* barriers to migration, and this PEP would be raising a new one. It's a change worth making, but we have time to ensure there are easy ways to do things like skipping cert validation, or tolerate expired certificates. Regards, Nick.

On 3 Sep 2014 09:08, "David Reid" <dreid@dreid.org> wrote:

Nick Coghlan <ncoghlan <at> gmail.com> writes:

...

Creating *new* incompatibilities between Python 2 & Python 3 is a major

point

...

of concern.

Clearly this change should be backported to Python2.

Proposing to break backwards compatibility in a maintenance release would require an *even better* migration story for handling legacy infrastructure. There's a reason the Py3 compatibility break was accompanied by a long term support commitment for Python 2 (and why that commitment was extended another 5 years when it became clear the implications of the binary and text handling were more significant than we originally realised). It's important to remember that the tech industry in general, and open source focused companies in particular, tend to be amongst the best of the best when it comes to infrastructure management (and we're still pretty terrible at it). The vast majority of infrastructure in the world *isn't* in the hands of those companies though - it's in the hands of companies where having any kind of computing infrastructure at all is merely a cost of doing business. Alex's "ridiculous" scenario doesn't sound ridiculous to me - it sounds like a typical corporate computing environment. Regards, Nick.

-David

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On Sep 2, 2014, at 7:47 PM, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:

On Sep 2, 2014, at 4:28 PM, Nick Coghlan <ncoghlan@gmail.com <mailto:ncoghlan@gmail.com>> wrote:

...

On 3 Sep 2014 09:08, "David Reid" <dreid@dreid.org <mailto:dreid@dreid.org>> wrote:

...

Nick Coghlan <ncoghlan <at> gmail.com <http://gmail.com/>> writes:

...

Creating *new* incompatibilities between Python 2 & Python 3 is a major point of concern.

Clearly this change should be backported to Python2.

Proposing to break backwards compatibility in a maintenance release (...)

As we keep saying, this is not a break in backwards compatibility, it's a bug fix. Yes, systems might break, but that breakage represents an increase in security which may well be operationally important. Not everyone with a "working" application has the relevant understanding an expertise to know that Python's HTTP client is exposing them to surveillance. These applications should break. That is the very nature of the fix. It is not a "compatibility break" that the system starts correctly rejecting invalid connections.

By way of analogy, here's another kind of breach in security: an arbitrary remote code execution vulnerability in XML-RPC. I think we all agree that any 0day RCE vulnerabilities in Python really ought to be fixed and could be legitimately included without worrying about backwards compatibility breaks. (At least... gosh, I hope so.)

Perhaps this arbitrary remote execution looks harmless; the use of an eval() instead of an int() someplace. Perhaps someone discovered that they can do "3 + 4" in their XML-RPC and the server does the computation for them. Great! They start relying on this in their applications to use symbolic values in their requests instead of having explicit enumerations. This can save you quite a bit of code!

When the RCE is fixed, this application will break, and that's fine. In fact that's the whole point of issuing the fix, that people will no longer be able to make arbitrary computation requests of your server any more. If that server's maintainer has the relevant context and actually wants the XML-RPC endpoint to enable arbitrary RCE, they can easily modify their application to start doing eval() on the data that they received, just as someone can easily modify their application to intentionally disable all connection security. (Let's stop calling it "certificate verification" because that sounds like some kind of clerical detail: if you disable certificate verification, TLS connections are unauthenticated and unidentified and therefore insecure.)

For what it's worth, on the equivalent Twisted change, I originally had just these concerns, but my mind was changed when I considered what exactly the user-interface ramifications were for people typing that 's' for 'secure' in URLs. I was convinced, and we made the change, and there have been no ill effects that I'm aware of as a result. In fact, there has been a renewed interest in Twisted for HTTP client work, because we finally made security work more or less like it's supposed to, and the standard library is so broken.

I care about the health of the broader Python community, so I will passionately argue that this change should be made, but for me personally it's a lot easier to justify that everyone should use Twisted (at least since 14+) because transport security in the stdlib is such a wreck and even if it gets fixed it's going to have easy options to turn it off unilaterally so your application can never really be sure if it's getting transport security when it's requesting transport security.

I completely agree with everything Glyph has said in this post. (To the shock of everyone involved I’m sure!). --- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

Antoine Pitrou writes:

On Tue, 2 Sep 2014 16:47:35 -0700 Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:

...

As we keep saying, this is not a break in backwards compatibility, it's a bug fix.

Keeping saying it doesn't make it magically true.

It's not "magically" true, it is "just" true. What the hardliners fail to acknowledge is that this is *not a bug in Python, it's a bug in the whole system*, and *mostly* in the environment. Changing Python will not change the environment, and applications will fail, with unknown consequences. Saying they "should" fail *right* now is bogus when you don't even know what those applications are, or what other security measures may be in place: Now is better than never. Although never is often better than *right* now. On the other hand, I commend the Twisted developers for putting their values into their code with their reputation on the line. I hope they win big with this move! Shouldn't we all hope for that? Steve

On 3 Sep 2014 18:28, "Cory Benfield" <cory@lukasa.co.uk> wrote:

This is definitely true, and this change is both. The only question that matters is whether we believe we're doing users a service by breaking their code. I'd argue, along with Glyph, Alex and Donald, that we are. I've been on the losing side of this debate a number of times though, and I expect I will be again.

The default stdlib behaviour will change in 3.5, I don't think anyone is disputing that. While I earlier said that should depend on the sslcustomize PEP, I now think they should be made orthogonal so the SSL customisation PEP can focus on its potential for *increasing* security in properly configured environments rather than deliberately decreasing it after upgrading to Python 3.5 in improperly configured ones. The backwards compatibility argument only applies to Python 2 maintenance releases (where dreid indicated an intention to request backporting the change), and there I'm quite happy to take the position of "use requests, Twisted or Python 3.5+ to get HTTPS done right". There are a variety of reasons not to use the Python 2 stdlib for modern networking, and making better tools more readily accessible to Python 2 users by backporting ensurepip is my preferred answer. Regards, Nick.

Summary: There needs to be a simple way to opt out at install time. It would be far better to offer more fine-grained control, but leaving that better solution to downstream is acceptable. On 3 September 2014 01:19, Antoine Pitrou <solipsis at pitrou.net> wrote:

RFC 2818 (HTTP over TLS) has the following language in section 3.1:

...

If the hostname is available, the client MUST check it against the server's identity as presented in the server's Certificate message, in order to prevent man-in-the-middle attacks.

...

If the client has external information as to the expected identity of the server, the hostname check MAY be omitted.

This second case is pretty common, in my experience. I still see it on the public internet, but mismatches are almost the expected case on the intranet, and many installation guides begin by saying to ignore the security warnings. I think it best not to name my employer in this context, but I work for an IT firm large enough that you've heard of it. As bad as our internal situation is, it is still better than a typical client's infrastructure, except that clients often have fewer surfaces to expose in the first place. Internal websites and applications tend to have information that "needs protection" only because saying otherwise requires a long bureaucratic process with little payoff. (Also true at many clients.) Nick has already posted a subset of the reasons why a site may be "signed" with a certificate that is self-signed, expired, and/or limited to the wrong hostname/subdomain. In the long run, I agree that it is better to default to secure. But in the short and medium term, there has to be a workaround, and I would prefer that the simplest workaround not be "retire the application, and don't use python again." I believe that the minimal acceptable workaround is that the Release Notes have an URL pointing to an install-time recipe telling the admin how to change the default back globally. Examples of good enough: "Add this file to site-packages" "Add this environment variable with this setting" "Add this command line switch to your launch script" Examples of not good enough: "Edit your application to change ..." "Edit your system store ..." (affecting more than python) Obviously, it would be great to offer finer control, so that the stricter default can be used when it is OK. (Per installation? Per application? Per run? Per domain? Per protocol? Per certificate? Per rejection reason? Treat anything in subdomain1.example.com as valid for hostname.example.com? Self-signing is OK for this IP range?) I would be pleasantly surprised if this level of API can even be standardized in time, and I agree that it is reasonable to leave it to 3rd party modules and downstream distributions. But I think Python itself should provide at least the single big hammer -- and that hammer should be something that can be used once at installation time (perhaps by changing the launch script), instead of requiring user interaction. -jJ -- If there are still threading problems with my replies, please email me with details, so that I can try to resolve them. -jJ

On Mon, Sep 8, 2014 at 3:44 PM, Cory Benfield <cory@lukasa.co.uk> wrote:

On 8 September 2014 18:23, Jim J. Jewett <jimjjewett@gmail.com> wrote:

...

Summary: There needs to be a simple way to opt out at install time. It would be far better to offer more fine-grained control, but leaving that better solution to downstream is acceptable.

Does this argument apply to a hypothetical 2.7 backport of this change, or does it apply to making the change in 3.5? (Or of course both.)

I believe the argument applies even to 3.5, given that there was no deprecation period. The concern is obviously stronger for maintenance releases. I am not saying that secure-by-default should wait until until 3.6; I am saying that the "rush" requires even more attention than usual to backwards compatibility. This actually argues *for* backporting the fix as at least opt-in, so that 2.7/3.4 can serve as the "make your changes now, test them without all the other new features" releases. Nick's suggestion of a monkey-patching .pth file would be sufficient backwards compatibility support, if the recipe were referenced from the release notes (not just the python lib documentation). Support for partial opt-in -- whether per-process, per call, per address, etc -- would be nice, but it isn't required for backwards compatibility. I think that means an -X option for "noverifyhttps" should NOT be added. It doesn't get users closer to the final solution; it just adds the noise of a different workaround. I assume that adding _unverified_urlopen or urlopen(context=...) do provide incremental improvements compatible with the eventual full opt-in. If so, adding them is probably reasonable, but I think the PEP should explicitly list all such approved half-measures as a guard against API feature creep. -jJ

The multiple threads going on are confusing (or maybe GMail makes them more confusing), but the architecture you are sketching here sounds good. I can't find get_default_context() in the repo, but perhaps I need to refresh, or perhaps you're talking about a design in a PEP. On Mon, Sep 8, 2014 at 8:03 PM, Nick Coghlan <ncoghlan@gmail.com> wrote:

On 9 Sep 2014 10:48, "Jim J. Jewett" <jimjjewett@gmail.com> wrote:

...

I assume that adding _unverified_urlopen or urlopen(context=...) do provide incremental improvements compatible with the eventual full opt-in. If so, adding them is probably reasonable, but I think the PEP should explicitly list all such approved half-measures as a guard against API feature creep.

From Guido's and your feedback, I think we may need two things to approve this for 3.4.2 (putting 2.7 aside for now):

1. "context" parameter support in urllib.request (to opt out on a per-call basis) 2. a documented way to restore the old behaviour via sitecustomize (which may involve monkeypatching)

The former change seems non-controversial.

I think the more fine-grained solution for the latter can wait until 3.5 (and will be an independent PEP), we just need an interim workaround for 3.4 that could conceivably be backported to 2.7.

On that front, ssl currently has two context factories: get_default_context() and _get_stdlib_context. One possible option would be to:

1. Rename "_get_stdlib_context" to "_get_unverified_context" 2. Add "_get_https_context" as an alias for "get_default_context"

Opting out on a per-call basis means passing an unverified context.

Opting out globally would mean monkeypatching _get_https_context to refer to _get_unverified_context instead.

These would both be documented as part of transition, but with clear security warnings. The use of the leading underscores in the names is intended to emphasise "you probably don't want to be using this".

Regards, Nick.

...

-jJ _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (python.org/~guido)

On 09.09.2014 05:03, Nick Coghlan wrote:

On 9 Sep 2014 10:48, "Jim J. Jewett" <jimjjewett@gmail.com <mailto:jimjjewett@gmail.com>> wrote:

...

I assume that adding _unverified_urlopen or urlopen(context=...) do provide incremental improvements compatible with the eventual full opt-in. If so, adding them is probably reasonable, but I think the PEP should explicitly list all such approved half-measures as a guard against API feature creep.

From Guido's and your feedback, I think we may need two things to approve this for 3.4.2 (putting 2.7 aside for now):

1. "context" parameter support in urllib.request (to opt out on a per-call basis) 2. a documented way to restore the old behaviour via sitecustomize (which may involve monkeypatching)

What's with our plan to introduce sslcustomize? Is the idea for a configuration module and named contexts off the table? For reference: I came up with the idea to introduce the module "sslcustomize" similar to sitecustomize. Contrary to sitecustomize the module is imported at the end of the ssl module. Based on my idea Nick proposed another addition to the SSL module. He proposed a ssl.named_contexts mapping from module names to factory functions that create SSLContext objects. http://permalink.gmane.org/gmane.comp.python.devel/149292 I still prefer the general idea over the monkey patching idea because it provides a clean but simple interface for structured configuration. Monkey patching of stdlib modules is ugly and error-prone. Christian

On Tue, Sep 9, 2014 at 12:11 PM, Christian Heimes <christian@python.org> wrote:

On 09.09.2014 05:03, Nick Coghlan wrote:

...

On 9 Sep 2014 10:48, "Jim J. Jewett" <jimjjewett@gmail.com <mailto:jimjjewett@gmail.com>> wrote: From Guido's and your feedback, I think we may need two things to approve this for 3.4.2 (putting 2.7 aside for now):

...

1. "context" parameter support in urllib.request (to opt out on a per-call basis) 2. a documented way to restore the old behaviour via sitecustomize (which may involve monkeypatching)

What's with our plan to introduce sslcustomize? Is the idea for a configuration module and named contexts off the table?

In a perfect world, half-measures would not be needed, and so neither would sslcustomize. In the real world, half-measures are needed, but offering too many of them adds so much confusion that things can actually get worse in practice. In other words, sslcustomize could be great, but getting it wrong would be a step backwards -- so start it as a 3rd party module. Since the biggest users are likely supported customers of downstream distributions, it makes sense to let them take the lead, though I'm sure they would appreciate a proof of concept.

I still prefer the general idea over the monkey patching idea because it provides a clean but simple interface for structured configuration. Monkey patching of stdlib modules is ugly and error-prone.

The primary use case for monkey patching is to support Separation of Roles. (Exact titles will of course differ by business.) If you need structured configuration, then you are already treating some calls differently from others, which means that you are already doing partial remediation. I agree that monkey patching is the wrong choice if you are doing partial remediation. But this partial remediation also implies that a "Developer" and "Business Owner" are involved to decide which calls need to be changed, and whether to change the call vs dropping the functionality vs convincing the owner of the other end of the connection to do things right in the first place. A Developer in charge of her own environment doesn't need to monkey patch -- but she could just do the right thing today, or switch to a framework that does. sslcustomize may be a really good way for her to document "these are the strange exceptions in our existing environment", if it is done right. A "Deployment Engineer" may not even know python, and is certainly not authorized to make changes beyond "configuration". Convincing someone that a .py file is a configuration knob probably requires an exception that is painful to get. (And saying "oh, this is just where we list security stuff that we're ignoring" won't make it easier.) Changing the the urlopen calls would therefore be unacceptable even if source code were available -- and sometimes it isn't. The Deployment Engineer is often responsible for upgrading the infrastructure components (possibly including python) for security patches, so he has to be able to deploy 3.4.x or 2.7.y (though *probably* not 3.5.0) without any changes to the application itself -- and usually without access to whatever regression tests the application itself uses. (Ideally, someone else who does have that access is involved, but ... not always.) What the Deployment Engineer *can* do is modify the environment around the application. He can write a shell script that sets environment variables and or command line options. He can probably even add a required component -- which might in practice just be a pre-written module like sslcustomize, or a .pth file that does the monkey patch on each launch. But *adding* such a component is orders of magnitude simpler (from a bureaucratic perspective) than *modifying* one that already exists. The end user often can't do anything outside the application's own UI, which is why the change has to be handled once at deployment, instead of as break-fix per call site or per bad certificate. -jJ

On 9 Sep 2014 03:25, "Jim J. Jewett" <jimjjewett@gmail.com> wrote:

Examples of good enough: "Add this file to site-packages" "Add this environment variable with this setting" "Add this command line switch to your launch script"

The monkeypatching hack I proposed would work correctly in sitecustomize (or a *.pth file). Cost would be loading httplib in all cases. A -X option for "noverifyhttps" might also be feasible. Cheers, Nick.

On Tue, 02 Sep 2014 20:59:54 -0400, Terry Reedy <tjreedy@udel.edu> wrote:

On 9/2/2014 7:47 PM, Glyph Lefkowitz wrote:

...

On Sep 2, 2014, at 4:28 PM, Nick Coghlan <ncoghlan@gmail.com <mailto:ncoghlan@gmail.com>> wrote:

...

On 3 Sep 2014 09:08, "David Reid" <dreid@dreid.org <mailto:dreid@dreid.org>> wrote:

...

...

...

Clearly this change should be backported to Python2.

Proposing to break backwards compatibility in a maintenance release (...)

For code that depends on a bug, fixing the bug breaks compatibility between releases without and with the fix. Nothing new. We occasionally don't backpart disruptive bugfixes because of that. We also backport some security fixes to otherwise unmaintained versions.

...

As we keep saying, this is not a break in backwards compatibility, it's a bug fix.

Disagreement on whether changing the default for certificate validation is a 'bugfix' or 'enhancement' (and in the colloquial or python tracker sense) the seems like the crux of the disagreement here. The case for 'bug', at least in the colloquial meaning, is at least plausible. On the other hand, the tracker meaning of 'behavior' issue (contrary to intention and doc) is more strict in including implementation bugs but not necessarily design bugs.

Up to this point the suggestion has been to change the default only in 3.5, and what we are arguing about is what kind of and how much aid to give to those for whom 3.5 breaks backward compatibility. So arguing about feature versus bug is irrelevant. The top proposal so far is an sslcustomize.py file that could be used to either decrease or increase the default security. This is a much less handy solution than application options (eg, curl, wget) that allow disabling security for "this cert" or "this CLI session". It also is more prone to unthinking abuse since it is persistent. So perhaps it is indeed not worth it. (That's why I suggested an environment variable...something you could specify on the command line for a one-off.) The more I think about it the less I like sslcustomize. I think I'd rather do without the ability to turn off checking without modifying code rather than introduce an intentional vector for a system-wide decrease in security. --David

On Wed, 03 Sep 2014 16:31:13 +0200, Antoine Pitrou <solipsis@pitrou.net> wrote:

On Tue, 02 Sep 2014 21:29:16 -0400 "R. David Murray" <rdmurray@bitdance.com> wrote:

...

The top proposal so far is an sslcustomize.py file that could be used to either decrease or increase the default security. This is a much less handy solution than application options (eg, curl, wget) that allow disabling security for "this cert" or "this CLI session". It also is more prone to unthinking abuse since it is persistent. So perhaps it is indeed not worth it. (That's why I suggested an environment variable...something you could specify on the command line for a one-off.)

I'll be fine with not adding any hooks at all, and letting people configure their application code correctly :-)

Again, the problem arises when it is not *their* application code, but a third party tool that hasn't been ported to 3.5. I'm OK with letting go of this invalid-cert issue myself, given the lack of negative feedback Twisted got. I'll just keep my fingers crossed. --David

Ethan Furman <ethan <at> stoneleaf.us> writes:

I apologize if I missed this point, but if we have the source code then it is possible to go in and directly modify the application/utility to be able to talk over https to a router with an invalid certificate? This is an option when creating the ssl_context?

-- ~Ethan~

Yes, it's totally possible to create (and pass to ``http.client``) an ``SSLContext`` which doesn't verify various things. My proposal is only about changing what happens when you don't explicitly pass a context. Cheers, Alex

On 03.09.2014 19:29, Ethan Furman wrote:

Excellent. Last question (I hope): it is possible to (easily) create an SSLContext that will verify against a self-signed certificate?

Yes: context = ssl.create_default_context(cafile="/path/to/selfsigned.pem") That works iff the certificate is valid, not expired and its CN or SAN matches the hostname of the service. When the hostname doesn't match then you have to set context.check_hostname = False Christian

On 09/03/2014 12:10 PM, R. David Murray wrote:

On Wed, 03 Sep 2014 10:09:36 -0700, Ethan Furman <ethan@stoneleaf.us> wrote:

...

On 09/03/2014 08:58 AM, R. David Murray wrote:

...

I'm OK with letting go of this invalid-cert issue myself, given the lack of negative feedback Twisted got. I'll just keep my fingers crossed.

I apologize if I missed this point, but if we have the source code then it is possible to go in and directly modify the application/utility to be able to talk over https to a router with an invalid certificate? This is an option when creating the ssl_context?

The immediately preceding paragraph that you didn't quote said that the context was 3rd party applications, not source code under your control. Yes, you can (usually) still hack the source, but there are good reasons to prefer to not do that, unfamiliarity with the codebase being just one of them.

I appreciate that there is a distinction, yet in most cases we have the source code available (it is the nature of Python) and if push comes to shove (and a bunch of other colloquialisms) then modifying that source code can get you up and running again. -- ~Ethan~

On Wed, 3 Sep 2014 10:54:55 -0700 Guido van Rossum <guido@python.org> wrote:

Let's take the plunge on this issue for the next 2.7 release (3.5 being a done deal).

I'm entirely against this.

Yes, some people will find that they have an old script accessing an old service which breaks. Surely some of the other changes in the same 2.7 bugfix release will also break some other scripts. People deal with it. Probably 90% of the time it's an annoyance (but no worse than any other minor-release upgrade -- you should test upgrades before committing to them, and if all else fails, roll it back).

Python is routinely updated to bugfix releases by Linux distributions and other distribution channels, you usually have no say over what's shipped in those updates. This is not like changing the major version used for executing the script, which is normally a manual change.

Today (working at Dropbox, a much smaller company!) I don't even remember the last time I had to deal with such a browser complaint -- internal services here all redirect to SSL, and not a browser that can find fault with their certs.

Good for you. I still sometimes get warnings about expired certificates - and sometimes ones that don't exactly match the domain being fetched (for example, the certificate wouldn't be valid for that specific subdomain - note that CAs often charge a premium for multiple subdomains, which why small or non-profit Web sites sometimes skimp on them). You shouldn't assume that the experience of well-connected people in the Silicon Valley is representative of what people over the world encounter. Yes, where there's a lot of money and a lot of accumulated domain competence, security procedures are updated and followed more scrupulously...

But at least some of the time it will be a wake-up call and an expired certificate will be replaced, resulting in more security for all.

Only if you are actually the one managing that certificate and the machine it's installed one... Regards Antoine.

Antoine, I think we are well past the point where arguments can sway positions. There clearly is no agreement on this issue. So please treat my post as a BDFL tie-breaker. I will just give you one thing to ponder -- those small/non-profit websites that can't afford proper certs are exactly the ones that will be hosting malware soon. Sorry for them, and the certificate vendors certainly aren't in it for charity, but they must fix their certificate issues (and probably improve many other sysadmin practices). --Guido On Wed, Sep 3, 2014 at 11:37 AM, Antoine Pitrou <solipsis@pitrou.net> wrote:

On Wed, 3 Sep 2014 10:54:55 -0700 Guido van Rossum <guido@python.org> wrote:

...

Let's take the plunge on this issue for the next 2.7 release (3.5 being a done deal).

I'm entirely against this.

...

Yes, some people will find that they have an old script accessing an old service which breaks. Surely some of the other changes in the same 2.7 bugfix release will also break some other scripts. People deal with it. Probably 90% of the time it's an annoyance (but no worse than any other minor-release upgrade -- you should test upgrades before committing to them, and if all else fails, roll it back).

Python is routinely updated to bugfix releases by Linux distributions and other distribution channels, you usually have no say over what's shipped in those updates. This is not like changing the major version used for executing the script, which is normally a manual change.

...

Today (working at Dropbox, a much smaller company!) I don't even remember the last time I had to deal with such a browser complaint -- internal services here all redirect to SSL, and not a browser that can find fault with their certs.

Good for you. I still sometimes get warnings about expired certificates - and sometimes ones that don't exactly match the domain being fetched (for example, the certificate wouldn't be valid for that specific subdomain - note that CAs often charge a premium for multiple subdomains, which why small or non-profit Web sites sometimes skimp on them).

You shouldn't assume that the experience of well-connected people in the Silicon Valley is representative of what people over the world encounter. Yes, where there's a lot of money and a lot of accumulated domain competence, security procedures are updated and followed more scrupulously...

...

But at least some of the time it will be a wake-up call and an expired certificate will be replaced, resulting in more security for all.

Only if you are actually the one managing that certificate and the machine it's installed one...

Regards

Antoine.

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org

-- --Guido van Rossum (python.org/~guido)

On Thu, 4 Sep 2014 09:19:56 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:

...

Python is routinely updated to bugfix releases by Linux distributions and other distribution channels, you usually have no say over what's shipped in those updates. This is not like changing the major version used for executing the script, which is normally a manual change.

We can potentially deal with the more conservative part of the user base on the redistributor side - so long as the PEP says it's OK for us to not apply this particular change if we deem it appropriate to do so.

So people would believe python.org that they would get HTTPS cert validation by default, but their upstream distributor would have disabled it for them? That's even worse... Of course, people could read distribution-specific package changelogs, but nobody does that.

2.7.9 is going to be a somewhat "interesting" release that requires careful attention anyway (due to the completion of the PEP 466 backports), so if Guido's OK with it, sure, let's kill the "HTTPS isn't" problem for Python 2 as well.

Possible unvoluntary breakage due to a large backport is one thing. Deliberate breakage is another. Regards Antoine.

On 09/03/2014 05:00 PM, Ethan Furman wrote:

On 09/03/2014 04:36 PM, Antoine Pitrou wrote:

...

On Thu, 4 Sep 2014 09:19:56 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:

...

...

Python is routinely updated to bugfix releases by Linux distributions and other distribution channels, you usually have no say over what's shipped in those updates. This is not like changing the major version used for executing the script, which is normally a manual change.

We can potentially deal with the more conservative part of the user base on the redistributor side - so long as the PEP says it's OK for us to not apply this particular change if we deem it appropriate to do so.

So people would believe python.org that they would get HTTPS cert validation by default, but their upstream distributor would have disabled it for them? That's even worse...

I agree. If the vendors don't want to have validation by default, they should stick with 2.7.8.

If good argument can be made for why we should make validation by default optional, then that point should be well made in Python's release notes, and some easy programmatic way to tell if validation is on or off (which may just be more docs saying call SSLContext and examine the results: xxx means you're validating, yyy means you are not). -- ~Ethan~

On Thu, 4 Sep 2014 13:11:38 +1000 Nick Coghlan <ncoghlan@gmail.com> wrote:

That leaves Python 2.7, and I have to say I'm now persuaded that a backport (including any required httplib and urllib features) is the right way to go. One of the tasks I'd been dreading as a follow-on from PEP 466 was organising the code audit to make sure our existing Python 2 applications are properly configuring SSL. If we instead change Python 2.7.9 to validate certificates by default, then the need to do that audit *goes away*, replaced by the far more mundane tasking of doing integration testing on 2.7.9, which we'd have to do *anyway*.

What are "our existing Python 2 applications"? Is it a Red Hat-specific statement? What is the "code audit" you are talking about? Regards Antoine.

On Sep 3, 2014, at 1:54 PM, Guido van Rossum <guido@python.org> wrote:

On Wed, Sep 3, 2014 at 8:58 AM, R. David Murray <rdmurray@bitdance.com <mailto:rdmurray@bitdance.com>> wrote: I'm OK with letting go of this invalid-cert issue myself, given the lack of negative feedback Twisted got. I'll just keep my fingers crossed.

I'm with this sentiment (cautiously +1) -- and not just because of Twisted's experience or Glyph's passion.

Network security is much more important now than it was five years ago -- and yet Python 2.7 is at least that old. My own experience has changed a lot: five years ago (when I worked at Google!) it was common to find internal services that required SSL but had a misconfigured certificate, and the only way to access those services was to override the browser complaints. Today (working at Dropbox, a much smaller company!) I don't even remember the last time I had to deal with such a browser complaint -- internal services here all redirect to SSL, and not a browser that can find fault with their certs. If I did get a complaint about a certificate I would fire off an email to a sysadmin alerting them to the issue.

Let's take the plunge on this issue for the next 2.7 release (3.5 being a done deal). Yes, some people will find that they have an old script accessing an old service which breaks. Surely some of the other changes in the same 2.7 bugfix release will also break some other scripts. People deal with it. Probably 90% of the time it's an annoyance (but no worse than any other minor-release upgrade -- you should test upgrades before committing to them, and if all else fails, roll it back). But at least some of the time it will be a wake-up call and an expired certificate will be replaced, resulting in more security for all.

+1, this makes me unreasonably happy.

I don't want to start preaching security doom and gloom (the experts are doing enough of that :-), but the scale and sophistication of attacks (whether publicized or not) is constantly increasing, and routine maintenance checks on old software are just one of the small ways that we can help the internet become more secure. (And please let the PSF sysadmin team beef up *.python.org <http://python.org/> -- sooner or later some forgotten part of our infrastructure *will* come under attack.)

This is an ongoing effort amongst the Infra team, part of the process is moving infrastructure away from hand crafted servers towards servers managed by config management as well as making sure all our services are behind TLS as well.

-- --Guido van Rossum (python.org/~guido <http://python.org/~guido>) _______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/donald%40stufft.io

--- Donald Stufft PGP: 7C6B 7C5D 5E2B 6356 A926 F04F 6E3C BCE9 3372 DCFA

OK, that changes my position for 2.7 (but not for 3.5). I had assumed there was a way to disable the cert check by changing one parameter to the urlopen() call. (And I had wanted to add that there should be a clear FAQ about the subject.) If this isn't possible that changes the situation. (But I still think that once we do have that simple change option we should do it, in a later 2.7 upgrade.) I apologize for speaking before I had read all facts, and I'll await what you and Nick come up with. --Guido On Wed, Sep 3, 2014 at 12:26 PM, Christian Heimes <christian@python.org> wrote:

On 03.09.2014 19:54, Guido van Rossum wrote:

...

Let's take the plunge on this issue for the next 2.7 release (3.5 being a done deal). Yes, some people will find that they have an old script accessing an old service which breaks. Surely some of the other changes in the same 2.7 bugfix release will also break some other scripts. People deal with it. Probably 90% of the time it's an annoyance (but no worse than any other minor-release upgrade -- you should test upgrades before committing to them, and if all else fails, roll it back). But at least some of the time it will be a wake-up call and an expired certificate will be replaced, resulting in more security for all.

I'm +1 for Python 3.5 but -1 for Python 2.7.

The SSLContext backport will landed in Python 2.7.9 (to be released). No Python 2 user is familiar with the feature yet. But more importantly: None of the stdlib modules support the new feature, too. httplib, imaplib ... they all don't take a SSLContext object as an argument. PEP-466 does not include the backport for the network modules. Without the context argument there is simply no clean way to configure the SSL handshake properly.

The default settings must stay until we decide to backport the context argument and have a way to configure the default behavior. Nick and me are planing a PEP.

Christian

-- --Guido van Rossum (python.org/~guido)

Guido van Rossum <guido <at> python.org> writes:

OK, that changes my position for 2.7 (but not for 3.5). I had assumed there was a way to disable the cert check by changing one parameter to the urlopen() call. (And I had wanted to add that there should be a clear FAQ about the subject.) If this isn't possible that changes the situation. (But I still think that once we do have that simple change option we should do it, in a later 2.7 upgrade.) I apologize for speaking before I had read all facts, and I'll await what you and Nick come up with. --Guido

This probably doesn't surprise anyone, but I'm more than happy to do the back- porting work for httplib, and any other modules which need SSLContext support; does this require an additional PEP, or does it fit under PEP466 or PEP476? Alex

On 4 Sep 2014 06:39, "Alex Gaynor" <alex.gaynor@gmail.com> wrote:

Guido van Rossum <guido <at> python.org> writes:

...

OK, that changes my position for 2.7 (but not for 3.5). I had assumed

there

...

was a way to disable the cert check by changing one parameter to the urlopen() call. (And I had wanted to add that there should be a clear FAQ about the subject.) If this isn't possible that changes the situation. (But I still think that once we do have that simple change option we should do it, in a later 2.7 upgrade.) I apologize for speaking before I had read all facts, and I'll await what you and Nick come up with. --Guido

This probably doesn't surprise anyone, but I'm more than happy to do the back- porting work for httplib, and any other modules which need SSLContext support; does this require an additional PEP, or does it fit under PEP466 or PEP476?

I suggest writing up a separate PEP for 2.7 that covers exactly what would need to be backported in order to make the same HTTPS handling change in Python 2. For 476, I suggest taking my list of modules that call "_create_stdlib_cert" and being completely explicit as to which ones are *not* changing (as that will help clarify the scope of the backport proposal). I learned that lesson with PEP 453 - it's well worth making the Python 3 PEP easier to accept by making it independent of the inevitably more controversial Python 2 backport PEP :) Cheers, Nick.

Alex

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On 03.09.2014 21:37, Victor Stinner wrote:

Thanks, you replied before I asked the question :-) (If certificates are validated by default, how do you disable the checks?)

Sorry, I didn't follow the whole discussion and Python 2.7 changes related to security. Does Python 2.7 support loading (automatically) system certificate authorities? Like the Windows certificate store: https://docs.python.org/dev/whatsnew/3.4.html#whatsnew34-win-cert-store

Yes! Alex and others have ported all SSL features back to Python 2.7. The SSL module in Python 2.7.9 will have the same feature set as Python 3.4. Christian

On Wed, Sep 3, 2014 at 3:48 PM, Stephen J. Turnbull <stephen@xemacs.org> wrote:

Guido van Rossum writes:

...

lot: five years ago (when I worked at Google!) it was common to find internal services that required SSL but had a misconfigured certificate, and the only way to access those services was to override the browser complaints. Today (working at Dropbox, a much smaller company!) I don't even remember the last time I had to deal with such a browser complaint --

I would tend to discount your recent experience, then. Smaller (and possibly even more important in this fast-developing area, younger) organizations are a lot more nimble about things like this.

As a defensive data point: I don't remember a single instance of this happening for Google internal services, at least since I arrived in 2007. I'm not doubting that Guido remembers some thing(s) but in general people here at Google would not stand for that, then or now. I would not call it common, especially five years ago. Common things I _have_ encountered over the years everywhere I've been both internal and external: services that listen on the https port 443 but don't have a valid cert as they are intended only for http port 80 access. Those are becoming somewhat less common, the only thing I regularly see that on anymore is random home router web config UIs as issuing a signed server certificate for security hole ridden commodity embedded box is... a challenge. (I'm not commenting on the PEP plans as it seems the right things are happening for now) -gps @ Google

On Tue, 02 Sep 2014 22:16:18 -0000, Alex Gaynor <alex.gaynor@gmail.com> wrote:

This whole scenario seems to be predicated on a siutation where: You have a peer whose certificate you can't change, and you have a piece of code you can't change, and you're going to upgrade your Python installation, and you want to talk to this peer, and you need to use an encrypted channel, but don't really care if it's being MITM'd. It doesn't seem to me that this is reasonably Python's responsibility to deal with the fact that you have no ability to upgrade any of your infrastructure, except your Python version.

I would say that is an excellent summary, except that you are sometimes *forced* to use an encrypted channel (the device only opens the https port). That is, in the real messy world of network devices, accepting an invalid cert is not a nonsensical operation. (Of course, what I really want is to be able to say "I know this cert has invalid fields, accept it anyway, but warn me if it changes when I connect again".) It's a good point that the actual number of people who will be hit by this and can't hack the code in question will be (possibly vanishingly) small, especially if you only consider python3. But we've already had one call for a backport :) --David