SSL certificates recommendations for downstream python packagers
Hi, I am managing the team responsible for providing python packaging at Enthought, and I would like to make sure we are providing a good (and secure) out of the box experience for SSL. My understanding is that PEP 476 is the latest PEP that concerns this issue, and that PEP recommends using the system store: https://www.python.org/dev/peps/pep-0476/#trust-database. But looking at binary python distributions from python.org, that does not seem to a.ways be the case. I looked at the following: * 3.5.3 from python.org for OS X (64 bits): this uses the old, system openssl * 3.6.0 from python.org for OS X: this embeds a recent openssl, but ssl seems to be configured to use non existing paths (ssl..get_default_verify_paths()), and indeed, cert validation seems to fail by default with those installers * 3.6.0 from python.org for windows: I have not found how the ssl module finds the certificate, but certification seems to work Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ? David
On 2017-01-30 14:53, David Cournapeau wrote:
Hi,
I am managing the team responsible for providing python packaging at Enthought, and I would like to make sure we are providing a good (and secure) out of the box experience for SSL.
My understanding is that PEP 476 is the latest PEP that concerns this issue, and that PEP recommends using the system store: https://www.python.org/dev/peps/pep-0476/#trust-database. But looking at binary python distributions from python.org <http://python.org>, that does not seem to a.ways be the case. I looked at the following:
* 3.5.3 from python.org <http://python.org> for OS X (64 bits): this uses the old, system openssl
On macOS, system OpenSSL uses TEA to look up trust anchors from the keyring, https://hynek.me/articles/apple-openssl-verification-surprises/
* 3.6.0 from python.org <http://python.org> for OS X: this embeds a recent openssl, but ssl seems to be configured to use non existing paths (ssl..get_default_verify_paths()), and indeed, cert validation seems to fail by default with those installers
I guess you haven't read the warning on the download page, https://www.python.org/downloads/release/python-360/ :) tl;dr Python on macOS can no longer use the keyring for certificates. You have to install certifi. Personally I (as a maintainer of the ssl module) do neither like certifi nor the approach taken. It's wrong and dangerous because certifi by-passes system policies and does not include local CAs. But it's the best we got.
* 3.6.0 from python.org <http://python.org> for windows: I have not found how the ssl module finds the certificate, but certification seems to work
The magic happens in load_default_certs(), https://hg.python.org/cpython/file/tip/Lib/ssl.py#l445 . The approach does work most of the time. Sometimes it fails because Windows download root CA's on demand. The code does not trigger a download. Steve and I have some ideas to improve the situation, https://bugs.python.org/issue28747 .
Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?
My semi-official recommendations are: * On Linux and BSD, use OpenSSL from your vendor (Red Hat, SuSE, Ubuntu, Debian, Arch, Gentoo, ...). Vendors compile OpenSSL with special flags to make get_default_verify_paths() work. * If your vendor prefers LibreSSL, you might run into minor issue. LibreSSL is not fully compatible to OpenSSL. Your vendor also compiles LibresSL with special flags. * Windows kinda works. We are going to improve the situation for 3.7. * On macOS ... well, please contact Apple service and demand a solution. Cory Benfield's TLS ABC PEP and Paul Kehrer's work on SecureTransport bindings for Python will let you use Apple's native TLS lib (with some restrictions regarding fork()). In the mean time you are screwed and have to rely on certifi. As last resort, use certifi. You HAVE to make sure to keep it up to date and you have to deal with local policies and CAs yourself. Christian
On 30 Jan 2017, at 13:53, David Cournapeau <cournape@gmail.com> wrote:
Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?
There *are* no default cert locations on Windows or macOS that can be accessed by OpenSSL. I cannot stress this strongly enough: you cannot provide a platform-native certificate validation logic for Python *and* use OpenSSL for certificate validation on Windows or macOS. (macOS can technically do this when you link against the system OpenSSL, at the cost of using a catastrophically insecure version of OpenSSL.) The only program I am aware of that does platform-native certificate validation on all three major desktop OS platforms is Chrome. It does this using a fork of OpenSSL to do the actual TLS, but the platform-native crypto library to do the certificate validation. This is the only acceptable way to do this, and Python does not expose the appropriate hooks to do it from within Python code. This would require that you carry substantial patches to the standard library to achieve this, all of which would be custom code. I strongly recommend you don't undertake to do this unless you are very confident of your ability to write this code correctly. The best long term solution to this is to stop using OpenSSL on platforms that don't consider it the 'blessed' approach. If you're interested in following that work, we're currently discussing it on the security-SIG, and you'd be welcome to join. Cory
On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield <cory@lukasa.co.uk> wrote:
On 30 Jan 2017, at 13:53, David Cournapeau <cournape@gmail.com> wrote:
Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?
There *are* no default cert locations on Windows or macOS that can be accessed by OpenSSL.
I cannot stress this strongly enough: you cannot provide a platform-native certificate validation logic for Python *and* use OpenSSL for certificate validation on Windows or macOS. (macOS can technically do this when you link against the system OpenSSL, at the cost of using a catastrophically insecure version of OpenSSL.)
Ah, thanks, that's already useful information. Just making sure I understand: this means there is no way to use python's SSL library to use the system store on windows, in particular private certifications that are often deployed by internal ITs in large orgs ?
The only program I am aware of that does platform-native certificate validation on all three major desktop OS platforms is Chrome. It does this using a fork of OpenSSL to do the actual TLS, but the platform-native crypto library to do the certificate validation. This is the only acceptable way to do this, and Python does not expose the appropriate hooks to do it from within Python code. This would require that you carry substantial patches to the standard library to achieve this, all of which would be custom code. I strongly recommend you don't undertake to do this unless you are very confident of your ability to write this code correctly.
That's exactly what I was afraid of and why I asked before attempting anything.
The best long term solution to this is to stop using OpenSSL on platforms that don't consider it the 'blessed' approach. If you're interested in following that work, we're currently discussing it on the security-SIG, and you'd be welcome to join.
Thanks, I will see if it looks like I have anything to contribute. David
On 2017-01-30 22:00, David Cournapeau wrote:
On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield <cory@lukasa.co.uk <mailto:cory@lukasa.co.uk>> wrote:
> On 30 Jan 2017, at 13:53, David Cournapeau <cournape@gmail.com <mailto:cournape@gmail.com>> wrote: > > Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?
There *are* no default cert locations on Windows or macOS that can be accessed by OpenSSL.
I cannot stress this strongly enough: you cannot provide a platform-native certificate validation logic for Python *and* use OpenSSL for certificate validation on Windows or macOS. (macOS can technically do this when you link against the system OpenSSL, at the cost of using a catastrophically insecure version of OpenSSL.)
Ah, thanks, that's already useful information.
Just making sure I understand: this means there is no way to use python's SSL library to use the system store on windows, in particular private certifications that are often deployed by internal ITs in large orgs ?
That works with CPython because we get all trust anchors from the cert store. However Python is not able to retrieve *additional* certificates. A new installation of Windows starts off with a minimal set of trust anchors. Chrome, IE and Edge use the proper APIs. There is no way to get internal CA on macOS with 3.6, with certifi, or with self-build OpenSSL. Christian
On Mon, Jan 30, 2017 at 9:14 PM, Christian Heimes <christian@python.org> wrote:
On 2017-01-30 22:00, David Cournapeau wrote:
On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield <cory@lukasa.co.uk <mailto:cory@lukasa.co.uk>> wrote:
> On 30 Jan 2017, at 13:53, David Cournapeau <cournape@gmail.com
<mailto:cournape@gmail.com>> wrote:
> > Are there any official recommendations for downstream packagers
beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?
There *are* no default cert locations on Windows or macOS that can be accessed by OpenSSL.
I cannot stress this strongly enough: you cannot provide a platform-native certificate validation logic for Python *and* use OpenSSL for certificate validation on Windows or macOS. (macOS can technically do this when you link against the system OpenSSL, at the cost of using a catastrophically insecure version of OpenSSL.)
Ah, thanks, that's already useful information.
Just making sure I understand: this means there is no way to use python's SSL library to use the system store on windows, in particular private certifications that are often deployed by internal ITs in large orgs ?
That works with CPython because we get all trust anchors from the cert store. However Python is not able to retrieve *additional* certificates. A new installation of Windows starts off with a minimal set of trust anchors. Chrome, IE and Edge use the proper APIs.
Hm. Is this documented anywhere ? We have customers needing "private/custom" certificates, and I am unsure where to look for. David
On 2017-01-30 22:19, David Cournapeau wrote:
Hm. Is this documented anywhere ? We have customers needing "private/custom" certificates, and I am unsure where to look for.
For full control it is advised to use a custom SSLContext that only loads the internal CA. https://docs.python.org/3/library/ssl.html#ssl.create_default_context With OpenSSL you can also set SSL_CERT_FILE and SSL_CERT_DIR env vars. It doesn't work with LibreSSL, though. import os, ssl os.environ['SSL_CERT_FILE'] = '/path/to/internalca.pem' os.environ['SSL_CERT_DIR'] = os.devnull ctx = ssl.create_default_context() Christian
On 30 Jan 2017, at 21:00, David Cournapeau <cournape@gmail.com> wrote:
On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield <cory@lukasa.co.uk <mailto:cory@lukasa.co.uk>> wrote:
On 30 Jan 2017, at 13:53, David Cournapeau <cournape@gmail.com <mailto:cournape@gmail.com>> wrote:
Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?
There *are* no default cert locations on Windows or macOS that can be accessed by OpenSSL.
I cannot stress this strongly enough: you cannot provide a platform-native certificate validation logic for Python *and* use OpenSSL for certificate validation on Windows or macOS. (macOS can technically do this when you link against the system OpenSSL, at the cost of using a catastrophically insecure version of OpenSSL.)
Ah, thanks, that's already useful information.
Just making sure I understand: this means there is no way to use python's SSL library to use the system store on windows, in particular private certifications that are often deployed by internal ITs in large orgs ?
If only it were that simple! No, you absolutely *can* do that. You can extract the trust roots from the system trust store, convert them into PEM/DER-encoded files, and load them into OpenSSL. That will work. The problem is that both SecureTransport and SChannel have got a number of differences from OpenSSL. In no particular order: 1. Their chain building logic is different. This means that, given a collection of certificates presented by a server and a bundle of already-trusted certs, each implementation may build a different trust chain. This may cause one implementation to refuse to validate where the others do, or vice versa. This is very common with older OpenSSLs. 2. SecureTransport and SChannel both use the system trust DB, which on both Windows and mac allows the setting of custom policies. OpenSSL won’t respect these policies, which means you can fail-open (that is, export and use a root certificate that the OS believes should not be trusted for a given use case). There is no way to export these trust policies into OpenSSL. 3. SecureTransport, SChannel, and OpenSSL all support different X.509 extensions and understand them differently. This means that some certs may be untrusted for certain uses by Windows but trusted for those uses by OpenSSL, for example. In general, it is unwise to mix trust stores. If you want to use your OS’s trust store, the best approach is to use the OS’s TLS stack as well. At least that way when a user says “It works in my browser”, you know it should work for you too. Cory
On 2017-01-31 10:19, Cory Benfield wrote:
On 30 Jan 2017, at 21:00, David Cournapeau <cournape@gmail.com <mailto:cournape@gmail.com>> wrote:
On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield <cory@lukasa.co.uk <mailto:cory@lukasa.co.uk>> wrote:
> On 30 Jan 2017, at 13:53, David Cournapeau <cournape@gmail.com <mailto:cournape@gmail.com>> wrote: > > Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?
There *are* no default cert locations on Windows or macOS that can be accessed by OpenSSL.
I cannot stress this strongly enough: you cannot provide a platform-native certificate validation logic for Python *and* use OpenSSL for certificate validation on Windows or macOS. (macOS can technically do this when you link against the system OpenSSL, at the cost of using a catastrophically insecure version of OpenSSL.)
Ah, thanks, that's already useful information.
Just making sure I understand: this means there is no way to use python's SSL library to use the system store on windows, in particular private certifications that are often deployed by internal ITs in large orgs ?
If only it were that simple!
No, you absolutely *can* do that. You can extract the trust roots from the system trust store, convert them into PEM/DER-encoded files, and load them into OpenSSL. That will work.
The problem is that both SecureTransport and SChannel have got a number of differences from OpenSSL. In no particular order:
1. Their chain building logic is different. This means that, given a collection of certificates presented by a server and a bundle of already-trusted certs, each implementation may build a different trust chain. This may cause one implementation to refuse to validate where the others do, or vice versa. This is very common with older OpenSSLs. 2. SecureTransport and SChannel both use the system trust DB, which on both Windows and mac allows the setting of custom policies. OpenSSL won’t respect these policies, which means you can fail-open (that is, export and use a root certificate that the OS believes should not be trusted for a given use case). There is no way to export these trust policies into OpenSSL.
One small correction, it is possible to export some of the trust settings to a TRUSTED CERTIFICATE and import them into OpenSSL. It works correctly in 1.0.1 and since 1.0.2e or f. Trust settings are stored in X509_AUX extension after the actual certificate and signature. OpenSSL's default loaders for cert dir and cert file do load auxiliary trust information. (Of course the feature is experimental and was broken in 1.0.2 for a long time until I discovered the issue and fixed it...) https://linux.die.net/man/1/x509 TRUST SETTINGS Please note these options are currently experimental and may well change. A trusted certificate is an ordinary certificate which has several additional pieces of information attached to it such as the permitted and prohibited uses of the certificate and an "alias". Normally when a certificate is being verified at least one certificate must be "trusted". By default a trusted certificate must be stored locally and must be a root CA: any certificate chain ending in this CA is then usable for any purpose. Trust settings currently are only used with a root CA . They allow a finer control over the purposes the root CA can be used for. For example a CA may be trusted for SSL client but not SSL server use.
On 31 Jan 2017, at 09:33, Christian Heimes <christian@python.org> wrote:
One small correction, it is possible to export some of the trust settings to a TRUSTED CERTIFICATE and import them into OpenSSL. It works correctly in 1.0.1 and since 1.0.2e or f. Trust settings are stored in X509_AUX extension after the actual certificate and signature. OpenSSL's default loaders for cert dir and cert file do load auxiliary trust information.
Ah, good spot. I suspect the code you’d need to write to safely extract that functionality is pretty subtle. I definitely don’t trust myself to get it right. Cory
On 31 January 2017 at 09:19, Cory Benfield <cory@lukasa.co.uk> wrote:
In general, it is unwise to mix trust stores. If you want to use your OS’s trust store, the best approach is to use the OS’s TLS stack as well. At least that way when a user says “It works in my browser”, you know it should work for you too.
As a bystander (and an "end user" of this stuff) the message I'm getting here is a bit worrying. To take a step back from the sysadmin issues here, is the statement It's safe to use Python (either via the stdlib, or various 3rd party libraries like requests) to access https URLs correct? I understand that "safe" is a complex concept here, but in terms of promoting Python, I'd be using the term in the sense of "at least as acceptable as using something like C# or Java" - in other words I'm not introducing any new vulnerabilities if I argue for Python over one of those languages? Paul
On 31 Jan 2017, at 09:56, Paul Moore <p.f.moore@gmail.com> wrote:
On 31 January 2017 at 09:19, Cory Benfield <cory@lukasa.co.uk> wrote:
In general, it is unwise to mix trust stores. If you want to use your OS’s trust store, the best approach is to use the OS’s TLS stack as well. At least that way when a user says “It works in my browser”, you know it should work for you too.
As a bystander (and an "end user" of this stuff) the message I'm getting here is a bit worrying. To take a step back from the sysadmin issues here, is the statement
It's safe to use Python (either via the stdlib, or various 3rd party libraries like requests) to access https URLs
correct? I understand that "safe" is a complex concept here, but in terms of promoting Python, I'd be using the term in the sense of "at least as acceptable as using something like C# or Java" - in other words I'm not introducing any new vulnerabilities if I argue for Python over one of those languages?
My answer would be “Yes*”. The asterisk is that it works best when you understand how you are rooting your trust stores. On Linux, if you use your distro provided OpenSSL and its trust store it is as safe as the rest of your system. On macOS, if you use the system Python it is unsafe. If you use a Python compiled against Homebrew’s OpenSSL, it is safe, though it is only safe based on a snapshot in time (this is for boring Homebrew-related reasons that are nothing to do with Python). On Windows, it is safe, though potentially less functional. Requests is basically safe, though it is definitely possible. If you are trying to reach websites on the open web using Requests, that’s safe, subject to the criteria below. The answer gets murkier if you do substantial configuration of a trust database. If you have added or removed root certificates from a system trust store, or you have configured the trust database on a Windows or Apple machine, then it all gets murky fast. If you’ve only added or expanded trust then Python is still safe on all those platforms, it is just less functional. If you’ve removed trust then it is possible that a Python process will trust something that other processes on your machine will not. However, you should be aware that that’s basically the default state of being for TLS. Java is basically in the same place as Python today, but with the added fun of having a hand-rolled TLS implementation (JSSE). C# is a different story: the stdlib uses SSPI for certificate validation, which basically means SChannel *on Windows*. In the wider .NET ecosystem I have no idea how this is being done. So C# applications are Windows-native safe on Windows, and are a crapshoot elsewhere. For Java vs Python, I’d say we’re slightly ahead right now. Again, the long-term solution to this fix is to allow us to use SChannel and SecureTransport to provide TLS on the relevant platforms. This will also let people use GnuTLS or NSS or whatever other TLS implementations float their boat on other Unices. I’ll be bringing a draft PEP in front of python-dev sometime in the next month to start this work: if you’re interested, I recommend helping out with that process! Cory
On 31 January 2017 at 14:54, Cory Benfield <cory@lukasa.co.uk> wrote:
So C# applications are Windows-native safe on Windows, and are a crapshoot elsewhere. For Java vs Python, I’d say we’re slightly ahead right now.
That's precisely the sort of answer I was after. Many thanks. The additional detail is interesting, but starts being scary again. I think the "advantage" languages like Java has is that no-one really discusses the details - so it seems like things are fine - but it devolves into a "how do we get this to work?" mess if you try to do anything hard. That's not a real advantage, but unfortunately politics often trumps technical accuracy in my area of work :-( (My job is often to make technically correct politically acceptable - who knew that's what "coding" really was?)
Again, the long-term solution to this fix is to allow us to use SChannel and SecureTransport to provide TLS on the relevant platforms. This will also let people use GnuTLS or NSS or whatever other TLS implementations float their boat on other Unices. I’ll be bringing a draft PEP in front of python-dev sometime in the next month to start this work: if you’re interested, I recommend helping out with that process!
That sounds fantastic. I'm not sure how much I'd be able to help, beyond whining "but I don't care about all this, just make it work and make it eeeeasyyyyyy" :-) but I'll certainly watch the discussions and do what I can. Thanks, Paul
On Tue, Jan 31, 2017 at 9:19 AM, Cory Benfield <cory@lukasa.co.uk> wrote:
On 30 Jan 2017, at 21:00, David Cournapeau <cournape@gmail.com> wrote:
On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield <cory@lukasa.co.uk> wrote:
On 30 Jan 2017, at 13:53, David Cournapeau <cournape@gmail.com> wrote:
Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?
There *are* no default cert locations on Windows or macOS that can be accessed by OpenSSL.
I cannot stress this strongly enough: you cannot provide a platform-native certificate validation logic for Python *and* use OpenSSL for certificate validation on Windows or macOS. (macOS can technically do this when you link against the system OpenSSL, at the cost of using a catastrophically insecure version of OpenSSL.)
Ah, thanks, that's already useful information.
Just making sure I understand: this means there is no way to use python's SSL library to use the system store on windows, in particular private certifications that are often deployed by internal ITs in large orgs ?
If only it were that simple!
No, you absolutely *can* do that. You can extract the trust roots from the system trust store, convert them into PEM/DER-encoded files, and load them into OpenSSL. That will work.
From the sound of it, it looks like this is simply not possible ATM with
Right, I guess it depends on what one means by "can". In my context, it was to be taken as "can it work without the end user having to do anything". We provide them a python-based tool, and it has to work with the system store out of the box. If the system store is updated through e.g. group policy, our python tool automatically get that update. python, at least not without 3rd party libraries. David
The problem is that both SecureTransport and SChannel have got a number of differences from OpenSSL. In no particular order:
1. Their chain building logic is different. This means that, given a collection of certificates presented by a server and a bundle of already-trusted certs, each implementation may build a different trust chain. This may cause one implementation to refuse to validate where the others do, or vice versa. This is very common with older OpenSSLs. 2. SecureTransport and SChannel both use the system trust DB, which on both Windows and mac allows the setting of custom policies. OpenSSL won’t respect these policies, which means you can fail-open (that is, export and use a root certificate that the OS believes should not be trusted for a given use case). There is no way to export these trust policies into OpenSSL. 3. SecureTransport, SChannel, and OpenSSL all support different X.509 extensions and understand them differently. This means that some certs may be untrusted for certain uses by Windows but trusted for those uses by OpenSSL, for example.
In general, it is unwise to mix trust stores. If you want to use your OS’s trust store, the best approach is to use the OS’s TLS stack as well. At least that way when a user says “It works in my browser”, you know it should work for you too.
Cory
On Mon, Jan 30, 2017 at 8:50 PM, Cory Benfield <cory@lukasa.co.uk> wrote:
On 30 Jan 2017, at 13:53, David Cournapeau <cournape@gmail.com> wrote:
Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?
There *are* no default cert locations on Windows or macOS that can be accessed by OpenSSL.
Also, doesn't that contradict the wording of PEP 476, specifically " Python would use the system provided certificate database on all platforms. Failure to locate such a database would be an error, and users would need to explicitly specify a location to fix it." ? Or is that PEP a long term goal, and not a description of the current status ? David
On 2017-01-30 21:50, Cory Benfield wrote:
On 30 Jan 2017, at 13:53, David Cournapeau <cournape@gmail.com> wrote:
Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?
There *are* no default cert locations on Windows or macOS that can be accessed by OpenSSL.
I cannot stress this strongly enough: you cannot provide a platform-native certificate validation logic for Python *and* use OpenSSL for certificate validation on Windows or macOS. (macOS can technically do this when you link against the system OpenSSL, at the cost of using a catastrophically insecure version of OpenSSL.)
In theory it is possible for Python and OpenSSL, too. I looked into a custom X509_LOOKUP_METHOD to locate trust anchors by subject. Steve is trying an alternative approach in https://bugs.python.org/issue28747. It ain't pretty and we are not there yet, too. Native support for SChannel and SecureTransport has some benefits. It's too bad OpenSSL lacks support for PKCS#11 Trust Assertion Objects. We could use https://p11-glue.freedesktop.org/doc/pkcs11-trust-assertions/#pkcs11-objects under Linux and the PKCS#11 under Windows and macOS. Christian
On 30Jan2017 1310, Christian Heimes wrote:
On 2017-01-30 21:50, Cory Benfield wrote:
On 30 Jan 2017, at 13:53, David Cournapeau <cournape@gmail.com> wrote:
Are there any official recommendations for downstream packagers beyond PEP 476 ? Is it "acceptable" for downstream packagers to patch python's default cert locations ?
There *are* no default cert locations on Windows or macOS that can be accessed by OpenSSL.
I cannot stress this strongly enough: you cannot provide a platform-native certificate validation logic for Python *and* use OpenSSL for certificate validation on Windows or macOS. (macOS can technically do this when you link against the system OpenSSL, at the cost of using a catastrophically insecure version of OpenSSL.)
In theory it is possible for Python and OpenSSL, too. I looked into a custom X509_LOOKUP_METHOD to locate trust anchors by subject. Steve is trying an alternative approach in https://bugs.python.org/issue28747. It ain't pretty and we are not there yet, too. Native support for SChannel and SecureTransport has some benefits.
My approach there is certainly not pretty, but IMHO it's the only feasible way to enable this on 3.6 (that is, until we get the proper TLS API). In short, I want to allow Python code to set OpenSSL's certificate validation callback. Basically, given a raw certificate, return True/False based on whether it should be trusted. I then have separate code (yet to be published) implementing the callback on Windows by calling into the WinVerifyTrust API with the HTTPS provider, which (allegedly) behaves identically to browsers using the same API (again, allegedly - I have absolutely no evidence to support this other than some manual testing). The big cons are that all the OpenSSL configuration you may do becomes useless (and as we know, there's essentially no configuration available for Windows's validation), and you're delegating the most interesting-to-exploit part of the process to someone else,perhaps unknowingly, if you import something that patches SSLContext... You still end up with an OpenSSL-wrapped socket, but I guess it could be with a certificate it doesn't like? (Also a big con is that Christian doesn't like this approach, and his risk assessment is certainly better than mine :) ) Native support for secure sockets is the best approach, but it won't land before 3.7. Cheers, Steve
On 31 Jan 2017, at 18:26, Steve Dower <steve.dower@python.org> wrote:
In short, I want to allow Python code to set OpenSSL's certificate validation callback. Basically, given a raw certificate, return True/False based on whether it should be trusted. I then have separate code (yet to be published) implementing the callback on Windows by calling into the WinVerifyTrust API with the HTTPS provider, which (allegedly) behaves identically to browsers using the same API (again, allegedly - I have absolutely no evidence to support this other than some manual testing).
For context here Steve, this is not quite what Chrome does (and I cannot stress enough that the Chrome approach is the best one I’ve seen, the folks working on it really do know what they’re doing). The reason here is a bit tricky, but essentially the validation callback is called incrementally for each step up the chain. This is not normally what a platform validation API actually wants: generally they want the entire cert chain the remote peer sent at once. Chrome, instead, essentially disables the OpenSSL cert validation entirely: they still require the certificate be presented, but override the verification callback to always say “yeah that’s cool, no big deal”. They then take the complete cert chain provided by the remote peer and pass that to the platform validation code in one shot after the handshake is complete, but before they send/receive any data on the connection. This is still safe: so long as you don’t actually expose any data before you have validated the certificates you aren’t at risk. I have actually prototyped this approach for Requests/urllib3 in the past. I wrote a small Rust extension to call into the platform-native code, and then wrapped it in a CFFI library that exposed a single callable to validate a cert chain for a specific hostname (library is here: https://github.com/python-hyper/certitude <https://github.com/python-hyper/certitude>). This could then be called from urllib3 code that used PyOpenSSL using this patch here: https://github.com/shazow/urllib3/pull/802/files <https://github.com/shazow/urllib3/pull/802/files> PLEASE DON’T ACTUALLY USE THIS CODE. I have not validated that certitude does entirely the right things with the platform APIs. This is just an example of a stripped-down version of what Chrome does, as a potential example of how to get something working for your Python use-case. Cory
Sorry, I misspoke when I said "certificate validation callback", I meant the same callback Cory uses below (name escapes me now, but it's unfortunately similar to what I said). There are two callbacks in OpenSSL, one that allows you to verify each certificate in the chain individually, and one that requires you to validate the entire chain. I do indeed take the entire chain in one go and pass it to the OS API. Christian also didn't like that I was bypassing *all* of OpenSSL's certificate handling here, but maybe there's a way to make it reliable if Chrome has done it? Top-posted from my Windows Phone -----Original Message----- From: "Cory Benfield" <cory@lukasa.co.uk> Sent: 2/1/2017 2:03 To: "Steve Dower" <steve.dower@python.org> Cc: "Christian Heimes" <christian@python.org>; "David Cournapeau" <cournape@gmail.com>; "python-dev" <python-dev@python.org> Subject: Re: [Python-Dev] SSL certificates recommendations for downstreampython packagers On 31 Jan 2017, at 18:26, Steve Dower <steve.dower@python.org> wrote: In short, I want to allow Python code to set OpenSSL's certificate validation callback. Basically, given a raw certificate, return True/False based on whether it should be trusted. I then have separate code (yet to be published) implementing the callback on Windows by calling into the WinVerifyTrust API with the HTTPS provider, which (allegedly) behaves identically to browsers using the same API (again, allegedly - I have absolutely no evidence to support this other than some manual testing). For context here Steve, this is not quite what Chrome does (and I cannot stress enough that the Chrome approach is the best one I’ve seen, the folks working on it really do know what they’re doing). The reason here is a bit tricky, but essentially the validation callback is called incrementally for each step up the chain. This is not normally what a platform validation API actually wants: generally they want the entire cert chain the remote peer sent at once. Chrome, instead, essentially disables the OpenSSL cert validation entirely: they still require the certificate be presented, but override the verification callback to always say “yeah that’s cool, no big deal”. They then take the complete cert chain provided by the remote peer and pass that to the platform validation code in one shot after the handshake is complete, but before they send/receive any data on the connection. This is still safe: so long as you don’t actually expose any data before you have validated the certificates you aren’t at risk. I have actually prototyped this approach for Requests/urllib3 in the past. I wrote a small Rust extension to call into the platform-native code, and then wrapped it in a CFFI library that exposed a single callable to validate a cert chain for a specific hostname (library is here: https://github.com/python-hyper/certitude). This could then be called from urllib3 code that used PyOpenSSL using this patch here: https://github.com/shazow/urllib3/pull/802/files PLEASE DON’T ACTUALLY USE THIS CODE. I have not validated that certitude does entirely the right things with the platform APIs. This is just an example of a stripped-down version of what Chrome does, as a potential example of how to get something working for your Python use-case. Cory
participants (5)
-
Christian Heimes -
Cory Benfield -
David Cournapeau -
Paul Moore -
Steve Dower