Verification of SSL cert and hostname made easy
Hi,
Larry has granted me a special pardon to add an outstanding fix for SSL, http://bugs.python.org/issue19509 . Right now most stdlib modules (ftplib, imaplib, nntplib, poplib, smtplib) neither support server name indication (SNI) nor check the subject name of the peer's certificate properly. The second issue is a major loop-hole because it allows man-in-the-middle attack despite CERT_REQUIRED.
With CERT_REQUIRED OpenSSL verifies that the peer's certificate is directly or indirectly signed by a trusted root certification authority. With Python 3.4 the ssl module is able to use/load the system's trusted root certs on all major systems (Linux, Mac, BSD, Windows). On Linux and BSD it requires a properly configured system openssl to locate the root certs. This usually works out of the box. On Mac Apple's openssl build is able to use the keychain API of OSX. I have added code for Windows' system store.
SSL socket code usually looks like this:
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) context.verify_mode = ssl.CERT_REQUIRED # new, by default it loads certs trusted for Purpose.SERVER_AUTH context.load_default_certs()
sock = socket.create_connection(("example.net", 443)) sslsock = context.wrap_socket(sock)
SSLContext.wrap_socket() wraps an ordinary socket into a SSLSocket. With verify_mode = CERT_REQUIRED OpenSSL ensures that the peer's SSL certificate is signed by a trusted root CA. In this example one very important step is missing. The peer may return *ANY* signed certificate for *ANY* hostname. These lines do NOT check that the certificate's information match "example.net". An attacker can use any arbitrary certificate (e.g. for "www.evil.net"), get it signed and abuse it for MitM attacks on "mail.python.org". http://docs.python.org/3/library/ssl.html#ssl.match_hostname must be used to verify the cert. It's easy to forget it...
I have thought about multiple ways to fix the issue. At first I added a new argument "check_hostname" to all affected modules and implemented the check manually. For every module I had to modify several places for SSL and STARTTLS and add / change about 10 lines. The extra lines are required to properly shutdown and close the connection when the cert doesn't match the hostname. I don't like the solution because it's tedious. Every 3rd party author has to copy the same code, too.
Then I came up with a better solution:
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) context.verify_mode = ssl.CERT_REQUIRED context.load_default_certs() context.check_hostname = True # <-- NEW
sock = socket.create_connection(("example.net", 443)) # server_hostname is already used for SNI sslsock = context.wrap_socket(sock, server_hostname="example.net")
This fix requires only a new SSLContext attribute and a small modification to SSLSocket.do_handshake():
if self.context.check_hostname: try: match_hostname(self.getpeercert(), self.server_hostname) except Exception: self.shutdown(_SHUT_RDWR) self.close() raise
Pros:
* match_hostname() is done in one central place * the cert is matched as early as possible * no extra arguments for APIs, a context object is enough * library developers just have to add server_hostname to get SNI and hostname checks at the same time * users of libraries can configure cert verification and checking on the same object * missing checks will not pass silently
Cons:
* Doesn't work with OpenSSL < 0.9.8f (released 2007) because older versions lack SNI support. The ssl module raises an exception for server_hostname if SNI is not supported.
The default settings for all stdlib modules will still be verify_mode = CERT_NONE and check_hostname = False for maximum backward compatibility. Python 3.4 comes with a new function ssl.create_default_context() that returns a new context with best practice settings and loaded root CA certs. The settings are TLS 1.0, no weak and insecure ciphers (no MD5, no RC4), no compression (CRIME attack), CERT_REQUIRED and check_hostname = True (for client side only).
http://bugs.python.org/issue19509 has a working patch for ftplib.
Comments?
Christian
On 1 Dec 2013 04:32, "Christian Heimes" christian@python.org wrote:
Hi,
Larry has granted me a special pardon to add an outstanding fix for SSL, http://bugs.python.org/issue19509 . Right now most stdlib modules (ftplib, imaplib, nntplib, poplib, smtplib) neither support server name indication (SNI) nor check the subject name of the peer's certificate properly. The second issue is a major loop-hole because it allows man-in-the-middle attack despite CERT_REQUIRED.
With CERT_REQUIRED OpenSSL verifies that the peer's certificate is directly or indirectly signed by a trusted root certification authority. With Python 3.4 the ssl module is able to use/load the system's trusted root certs on all major systems (Linux, Mac, BSD, Windows). On Linux and BSD it requires a properly configured system openssl to locate the root certs. This usually works out of the box. On Mac Apple's openssl build is able to use the keychain API of OSX. I have added code for Windows' system store.
SSL socket code usually looks like this:
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) context.verify_mode = ssl.CERT_REQUIRED # new, by default it loads certs trusted for Purpose.SERVER_AUTH context.load_default_certs()
sock = socket.create_connection(("example.net", 443)) sslsock = context.wrap_socket(sock)
SSLContext.wrap_socket() wraps an ordinary socket into a SSLSocket. With verify_mode = CERT_REQUIRED OpenSSL ensures that the peer's SSL certificate is signed by a trusted root CA. In this example one very important step is missing. The peer may return *ANY* signed certificate for *ANY* hostname. These lines do NOT check that the certificate's information match "example.net". An attacker can use any arbitrary certificate (e.g. for "www.evil.net"), get it signed and abuse it for MitM attacks on "mail.python.org". http://docs.python.org/3/library/ssl.html#ssl.match_hostname must be used to verify the cert. It's easy to forget it...
I have thought about multiple ways to fix the issue. At first I added a new argument "check_hostname" to all affected modules and implemented the check manually. For every module I had to modify several places for SSL and STARTTLS and add / change about 10 lines. The extra lines are required to properly shutdown and close the connection when the cert doesn't match the hostname. I don't like the solution because it's tedious. Every 3rd party author has to copy the same code, too.
Then I came up with a better solution:
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) context.verify_mode = ssl.CERT_REQUIRED context.load_default_certs() context.check_hostname = True # <-- NEW
sock = socket.create_connection(("example.net", 443)) # server_hostname is already used for SNI sslsock = context.wrap_socket(sock, server_hostname="example.net")
This fix requires only a new SSLContext attribute and a small modification to SSLSocket.do_handshake():
if self.context.check_hostname: try: match_hostname(self.getpeercert(), self.server_hostname) except Exception: self.shutdown(_SHUT_RDWR) self.close() raise
Pros:
- match_hostname() is done in one central place
- the cert is matched as early as possible
- no extra arguments for APIs, a context object is enough
- library developers just have to add server_hostname to get SNI and
hostname checks at the same time
- users of libraries can configure cert verification and checking on the
same object
- missing checks will not pass silently
Cons:
- Doesn't work with OpenSSL < 0.9.8f (released 2007) because older
versions lack SNI support. The ssl module raises an exception for server_hostname if SNI is not supported.
The default settings for all stdlib modules will still be verify_mode = CERT_NONE and check_hostname = False for maximum backward compatibility. Python 3.4 comes with a new function ssl.create_default_context() that returns a new context with best practice settings and loaded root CA certs. The settings are TLS 1.0, no weak and insecure ciphers (no MD5, no RC4), no compression (CRIME attack), CERT_REQUIRED and check_hostname = True (for client side only).
http://bugs.python.org/issue19509 has a working patch for ftplib.
Comments?
If Larry is OK with it as RM (and it sounds like he is), +1 from me as well.
Cheers, Nick.
Christian
Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com
Sounds good.
Is another change for asyncio needed?
On Sat, Nov 30, 2013 at 1:54 PM, Nick Coghlan ncoghlan@gmail.com wrote:
On 1 Dec 2013 04:32, "Christian Heimes" christian@python.org wrote:
Hi,
Larry has granted me a special pardon to add an outstanding fix for SSL, http://bugs.python.org/issue19509 . Right now most stdlib modules (ftplib, imaplib, nntplib, poplib, smtplib) neither support server name indication (SNI) nor check the subject name of the peer's certificate properly. The second issue is a major loop-hole because it allows man-in-the-middle attack despite CERT_REQUIRED.
With CERT_REQUIRED OpenSSL verifies that the peer's certificate is directly or indirectly signed by a trusted root certification authority. With Python 3.4 the ssl module is able to use/load the system's trusted root certs on all major systems (Linux, Mac, BSD, Windows). On Linux and BSD it requires a properly configured system openssl to locate the root certs. This usually works out of the box. On Mac Apple's openssl build is able to use the keychain API of OSX. I have added code for Windows' system store.
SSL socket code usually looks like this:
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) context.verify_mode = ssl.CERT_REQUIRED # new, by default it loads certs trusted for Purpose.SERVER_AUTH context.load_default_certs()
sock = socket.create_connection(("example.net", 443)) sslsock = context.wrap_socket(sock)
SSLContext.wrap_socket() wraps an ordinary socket into a SSLSocket. With verify_mode = CERT_REQUIRED OpenSSL ensures that the peer's SSL certificate is signed by a trusted root CA. In this example one very important step is missing. The peer may return *ANY* signed certificate for *ANY* hostname. These lines do NOT check that the certificate's information match "example.net". An attacker can use any arbitrary certificate (e.g. for "www.evil.net"), get it signed and abuse it for MitM attacks on "mail.python.org". http://docs.python.org/3/library/ssl.html#ssl.match_hostname must be used to verify the cert. It's easy to forget it...
I have thought about multiple ways to fix the issue. At first I added a new argument "check_hostname" to all affected modules and implemented the check manually. For every module I had to modify several places for SSL and STARTTLS and add / change about 10 lines. The extra lines are required to properly shutdown and close the connection when the cert doesn't match the hostname. I don't like the solution because it's tedious. Every 3rd party author has to copy the same code, too.
Then I came up with a better solution:
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) context.verify_mode = ssl.CERT_REQUIRED context.load_default_certs() context.check_hostname = True # <-- NEW
sock = socket.create_connection(("example.net", 443)) # server_hostname is already used for SNI sslsock = context.wrap_socket(sock, server_hostname="example.net")
This fix requires only a new SSLContext attribute and a small modification to SSLSocket.do_handshake():
if self.context.check_hostname: try: match_hostname(self.getpeercert(), self.server_hostname) except Exception: self.shutdown(_SHUT_RDWR) self.close() raise
Pros:
- match_hostname() is done in one central place
- the cert is matched as early as possible
- no extra arguments for APIs, a context object is enough
- library developers just have to add server_hostname to get SNI and
hostname checks at the same time
- users of libraries can configure cert verification and checking on the
same object
- missing checks will not pass silently
Cons:
- Doesn't work with OpenSSL < 0.9.8f (released 2007) because older
versions lack SNI support. The ssl module raises an exception for server_hostname if SNI is not supported.
The default settings for all stdlib modules will still be verify_mode = CERT_NONE and check_hostname = False for maximum backward compatibility. Python 3.4 comes with a new function ssl.create_default_context() that returns a new context with best practice settings and loaded root CA certs. The settings are TLS 1.0, no weak and insecure ciphers (no MD5, no RC4), no compression (CRIME attack), CERT_REQUIRED and check_hostname = True (for client side only).
http://bugs.python.org/issue19509 has a working patch for ftplib.
Comments?
If Larry is OK with it as RM (and it sounds like he is), +1 from me as well.
On Sat, 30 Nov 2013 19:29:37 +0100 Christian Heimes christian@python.org wrote:
This fix requires only a new SSLContext attribute and a small modification to SSLSocket.do_handshake():
if self.context.check_hostname: try: match_hostname(self.getpeercert(), self.server_hostname) except Exception: self.shutdown(_SHUT_RDWR) self.close() raise
Small nit: what happens if the server_hostname is None (i.e. wasn't passed to context.wrap_socket())?
The default settings for all stdlib modules will still be verify_mode = CERT_NONE and check_hostname = False for maximum backward compatibility. Python 3.4 comes with a new function ssl.create_default_context() that returns a new context with best practice settings and loaded root CA certs. The settings are TLS 1.0, no weak and insecure ciphers (no MD5, no RC4), no compression (CRIME attack), CERT_REQUIRED and check_hostname = True (for client side only).
Sounds fine to me, thanks.
Regards
Antoine.
Am 30.11.2013 23:51, schrieb Antoine Pitrou:
Small nit: what happens if the server_hostname is None (i.e. wasn't passed to context.wrap_socket())?
The code will raise an exception. My patch already implements a more verbose ValueError that explains the cause of the problem. It's flaw in code, that calls context.wrap_socket. Erroneous code will no longer pass silently.
The patch also ensures a valid combination of verify_mode and check_hostname:
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) context.check_hostname = True
Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: check_hostname needs a SSL context with either CERT_OPTIONAL or CERT_REQUIRED
context.verify_mode = ssl.CERT_REQUIRED context.check_hostname = True context.verify_mode = ssl.CERT_NONE
Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: Cannot set verify_mode to CERT_NONE when check_hostname is enabled.
It's only a limitation of the Python API, getpeercert() returns {} for an unverified cert. OpenSSL can still returns the cert, though.
Christian
On Sun, 01 Dec 2013 02:53:32 +0100 Christian Heimes christian@python.org wrote:
Am 30.11.2013 23:51, schrieb Antoine Pitrou:
Small nit: what happens if the server_hostname is None (i.e. wasn't passed to context.wrap_socket())?
The code will raise an exception. My patch already implements a more verbose ValueError that explains the cause of the problem. It's flaw in code, that calls context.wrap_socket. Erroneous code will no longer pass silently.
The patch also ensures a valid combination of verify_mode and check_hostname:
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) context.check_hostname = True
Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: check_hostname needs a SSL context with either CERT_OPTIONAL or CERT_REQUIRED
context.verify_mode = ssl.CERT_REQUIRED context.check_hostname = True context.verify_mode = ssl.CERT_NONE
Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: Cannot set verify_mode to CERT_NONE when check_hostname is enabled.
So I have to set attributes in a given order? I find this silly.
Regards
Antoine.
On 1 December 2013 20:37, Antoine Pitrou solipsis@pitrou.net wrote:
On Sun, 01 Dec 2013 02:53:32 +0100 Christian Heimes christian@python.org wrote:
Am 30.11.2013 23:51, schrieb Antoine Pitrou:
Small nit: what happens if the server_hostname is None (i.e. wasn't passed to context.wrap_socket())?
The code will raise an exception. My patch already implements a more verbose ValueError that explains the cause of the problem. It's flaw in code, that calls context.wrap_socket. Erroneous code will no longer pass silently.
The patch also ensures a valid combination of verify_mode and check_hostname:
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) context.check_hostname = True
Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: check_hostname needs a SSL context with either CERT_OPTIONAL or CERT_REQUIRED
context.verify_mode = ssl.CERT_REQUIRED context.check_hostname = True context.verify_mode = ssl.CERT_NONE
Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: Cannot set verify_mode to CERT_NONE when check_hostname is enabled.
So I have to set attributes in a given order? I find this silly.
Perhaps a cleaner option would be to make check_hostname read only, and add a secure-by-default method that allows all verification related settings to be adjusted at once:
def set_verify_mode(mode=ssl.CERT_REQUIRED, check_hostname=True): ...
That way the consistency check would be directly on the method arguments, rather than involving the current context state. Any future settings that required a verified cert could then use a similar model.
If we don't do that, then I think Christian's approach is a reasonable compromise given the late stage of the release cycle - it ensures the context can't get into the inconsistent verify_mode=CERT_NONE and check_hostname=True state, and leaves our options completely open for 3.5:
- add a "configure all security related settings at once" set_verify_mode method as suggested above - allow the context to get into an inconsistent state, check it in wrap_socket (bad due to point of exception != point of error) - have setting check_hostname potentially affect verify_mode and vice-versa (bad due to being implicit magic)
We wouldn't be able to make check_hostname read-only the way we would if we added the configuration method now, but that doesn't bother me that much as long as the setting consistency invariant is enforced.
Cheers, Nick.
On Sun, 1 Dec 2013 21:33:06 +1000 Nick Coghlan ncoghlan@gmail.com wrote:
If we don't do that, then I think Christian's approach is a reasonable compromise given the late stage of the release cycle - it ensures the context can't get into the inconsistent verify_mode=CERT_NONE and check_hostname=True state, and leaves our options completely open for 3.5:
I would prefer the check to be made when the the socket is created, i.e. the wrap_socket() call.
Regards
Antoine.
On 1 December 2013 21:40, Antoine Pitrou solipsis@pitrou.net wrote:
On Sun, 1 Dec 2013 21:33:06 +1000 Nick Coghlan ncoghlan@gmail.com wrote:
If we don't do that, then I think Christian's approach is a reasonable compromise given the late stage of the release cycle - it ensures the context can't get into the inconsistent verify_mode=CERT_NONE and check_hostname=True state, and leaves our options completely open for 3.5:
I would prefer the check to be made when the the socket is created, i.e. the wrap_socket() call.
That was my initial reaction, but then I realised it creates a situation where the exception is raised at a point that differs from the source of the error (the bug is in the way the context was configured, but the exception won't be thrown until you actually try to wrap a socket). So I now agree with Christian that it's better to prevent the creation of the internally inconsistent SSL context in the first place, rather than delaying the detection of the inconsistency until the context is actually used.
I think a read-only attribute and a combined setter method is a better way to achieve that (since it avoids the quirky "order of setting attributes matters" behaviour), but I'm also OK with read/write properties that internally enforce of the state invariant.
If we decide the invariant enforcement in 3.4 is too strict, we can change our minds and relax things in 3.5. By contrast, if we allow the invariant to be broken in 3.4, we're locked in by backwards compatibility concerns and can't change our minds in the future.
Cheers, Nick.
Am 01.12.2013 12:33, schrieb Nick Coghlan:
Perhaps a cleaner option would be to make check_hostname read only, and add a secure-by-default method that allows all verification related settings to be adjusted at once:
def set_verify_mode(mode=ssl.CERT_REQUIRED, check_hostname=True): ...That way the consistency check would be directly on the method arguments, rather than involving the current context state. Any future settings that required a verified cert could then use a similar model.
I don't like to add yet another API element to SSLContext. I'd rather extend SSLContext.__init__ to take two additional parameters:
class SSLContext(_SSLContext): def __init__(self, protocol, verify_mode=ssl.CERT_NONE, check_hostname=False): self.protocol = protocol self.verify_mode = verify_mode self.check_hostname = check_hostname
I also like to point out (again) that the requirement is only a limitation of our API. OpenSSL is able to acquire and return the peer's certificate with any mode. In the past somebody decided to map 'no certificate' to None and 'no verification' to empty dict.
If we don't do that, then I think Christian's approach is a reasonable compromise given the late stage of the release cycle - it ensures the context can't get into the inconsistent verify_mode=CERT_NONE and check_hostname=True state, and leaves our options completely open for 3.5:
You are right, I'm trying to aim for the simplest and smallest solution possible. I'm well aware that the order of API calls is a limitation and that it feels a bit awkward, too. In my opinion there is no way this API can be screwed up. :) Any limitation can be lifted for 3.5 but we can't make it more restrict in future versions.
And there is ssl.create_default_context(), too. It creates a context with all security-related bits flipped on.
Christian
On 30 nov. 2013, at 19:29, Christian Heimes christian@python.org wrote:
With CERT_REQUIRED OpenSSL verifies that the peer's certificate is directly or indirectly signed by a trusted root certification authority. With Python 3.4 the ssl module is able to use/load the system's trusted root certs on all major systems (Linux, Mac, BSD, Windows). On Linux and BSD it requires a properly configured system openssl to locate the root certs. This usually works out of the box. On Mac Apple's openssl build is able to use the keychain API of OSX. I have added code for Windows' system store.
Note that only Apple's build of OpenSSL integrates with keychain, other builds don't. The patch for keychain integration is on Apple's open source site but that isn't very helpful because that code uses a private API to do most of the work.
This almost certainly means that users of fink, macports and the like cannot use the system keystore.
It is probably possible to use the Keychain API to verify certificates, I haven't seriously looked into that yet and there is a risk of using higher level APIs: those tend to not like calling fork without calling execv soon after and that could break existing scripts.
Ronald
participants (5)
-
Antoine Pitrou -
Christian Heimes -
Guido van Rossum -
Nick Coghlan -
Ronald Oussoren