On 1 Dec 2013 04:32, "Christian Heimes" wrote:

Hi,

Larry has granted me a special pardon to add an outstanding fix for SSL, http://bugs.python.org/issue19509 . Right now most stdlib modules (ftplib, imaplib, nntplib, poplib, smtplib) neither support server name indication (SNI) nor check the subject name of the peer's certificate properly. The second issue is a major loop-hole because it allows man-in-the-middle attack despite CERT_REQUIRED.

With CERT_REQUIRED OpenSSL verifies that the peer's certificate is directly or indirectly signed by a trusted root certification authority. With Python 3.4 the ssl module is able to use/load the system's trusted root certs on all major systems (Linux, Mac, BSD, Windows). On Linux and BSD it requires a properly configured system openssl to locate the root certs. This usually works out of the box. On Mac Apple's openssl build is able to use the keychain API of OSX. I have added code for Windows' system store.

SSL socket code usually looks like this:

context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) context.verify_mode = ssl.CERT_REQUIRED # new, by default it loads certs trusted for Purpose.SERVER_AUTH context.load_default_certs()

sock = socket.create_connection(("example.net", 443)) sslsock = context.wrap_socket(sock)

SSLContext.wrap_socket() wraps an ordinary socket into a SSLSocket. With verify_mode = CERT_REQUIRED OpenSSL ensures that the peer's SSL certificate is signed by a trusted root CA. In this example one very important step is missing. The peer may return *ANY* signed certificate for *ANY* hostname. These lines do NOT check that the certificate's information match "example.net". An attacker can use any arbitrary certificate (e.g. for "www.evil.net"), get it signed and abuse it for MitM attacks on "mail.python.org". http://docs.python.org/3/library/ssl.html#ssl.match_hostname must be used to verify the cert. It's easy to forget it...

I have thought about multiple ways to fix the issue. At first I added a new argument "check_hostname" to all affected modules and implemented the check manually. For every module I had to modify several places for SSL and STARTTLS and add / change about 10 lines. The extra lines are required to properly shutdown and close the connection when the cert doesn't match the hostname. I don't like the solution because it's tedious. Every 3rd party author has to copy the same code, too.

Then I came up with a better solution:

context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) context.verify_mode = ssl.CERT_REQUIRED context.load_default_certs() context.check_hostname = True # <-- NEW

sock = socket.create_connection(("example.net", 443)) # server_hostname is already used for SNI sslsock = context.wrap_socket(sock, server_hostname="example.net")

This fix requires only a new SSLContext attribute and a small modification to SSLSocket.do_handshake():

if self.context.check_hostname: try: match_hostname(self.getpeercert(), self.server_hostname) except Exception: self.shutdown(_SHUT_RDWR) self.close() raise

Pros:

* match_hostname() is done in one central place * the cert is matched as early as possible * no extra arguments for APIs, a context object is enough * library developers just have to add server_hostname to get SNI and hostname checks at the same time * users of libraries can configure cert verification and checking on the same object * missing checks will not pass silently

Cons:

* Doesn't work with OpenSSL < 0.9.8f (released 2007) because older versions lack SNI support. The ssl module raises an exception for server_hostname if SNI is not supported.

The default settings for all stdlib modules will still be verify_mode = CERT_NONE and check_hostname = False for maximum backward compatibility. Python 3.4 comes with a new function ssl.create_default_context() that returns a new context with best practice settings and loaded root CA certs. The settings are TLS 1.0, no weak and insecure ciphers (no MD5, no RC4), no compression (CRIME attack), CERT_REQUIRED and check_hostname = True (for client side only).

http://bugs.python.org/issue19509 has a working patch for ftplib.

Comments?

If Larry is OK with it as RM (and it sounds like he is), +1 from me as well. Cheers, Nick.

Christian

_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:

https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com

On Sat, 30 Nov 2013 19:29:37 +0100 Christian Heimes wrote:

This fix requires only a new SSLContext attribute and a small modification to SSLSocket.do_handshake():

if self.context.check_hostname: try: match_hostname(self.getpeercert(), self.server_hostname) except Exception: self.shutdown(_SHUT_RDWR) self.close() raise

Small nit: what happens if the server_hostname is None (i.e. wasn't passed to context.wrap_socket())?

The default settings for all stdlib modules will still be verify_mode = CERT_NONE and check_hostname = False for maximum backward compatibility. Python 3.4 comes with a new function ssl.create_default_context() that returns a new context with best practice settings and loaded root CA certs. The settings are TLS 1.0, no weak and insecure ciphers (no MD5, no RC4), no compression (CRIME attack), CERT_REQUIRED and check_hostname = True (for client side only).

Sounds fine to me, thanks. Regards Antoine.

On Sun, 01 Dec 2013 02:53:32 +0100 Christian Heimes wrote:

Am 30.11.2013 23:51, schrieb Antoine Pitrou:

...

Small nit: what happens if the server_hostname is None (i.e. wasn't passed to context.wrap_socket())?

The code will raise an exception. My patch already implements a more verbose ValueError that explains the cause of the problem. It's flaw in code, that calls context.wrap_socket. Erroneous code will no longer pass silently.

The patch also ensures a valid combination of verify_mode and check_hostname:

...

...

...

context = ssl.SSLContext(ssl.PROTOCOL_TLSv1) context.check_hostname = True Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: check_hostname needs a SSL context with either CERT_OPTIONAL or CERT_REQUIRED context.verify_mode = ssl.CERT_REQUIRED context.check_hostname = True context.verify_mode = ssl.CERT_NONE Traceback (most recent call last): File "<stdin>", line 1, in <module> ValueError: Cannot set verify_mode to CERT_NONE when check_hostname is enabled.

So I have to set attributes in a given order? I find this silly. Regards Antoine.

On Sun, 1 Dec 2013 21:33:06 +1000 Nick Coghlan wrote:

If we don't do that, then I think Christian's approach is a reasonable compromise given the late stage of the release cycle - it ensures the context can't get into the inconsistent verify_mode=CERT_NONE and check_hostname=True state, and leaves our options completely open for 3.5:

I would prefer the check to be made when the the socket is created, i.e. the wrap_socket() call. Regards Antoine.

Am 01.12.2013 12:33, schrieb Nick Coghlan:

Perhaps a cleaner option would be to make check_hostname read only, and add a secure-by-default method that allows all verification related settings to be adjusted at once:

def set_verify_mode(mode=ssl.CERT_REQUIRED, check_hostname=True): ...

That way the consistency check would be directly on the method arguments, rather than involving the current context state. Any future settings that required a verified cert could then use a similar model.

I don't like to add yet another API element to SSLContext. I'd rather extend SSLContext.__init__ to take two additional parameters: class SSLContext(_SSLContext): def __init__(self, protocol, verify_mode=ssl.CERT_NONE, check_hostname=False): self.protocol = protocol self.verify_mode = verify_mode self.check_hostname = check_hostname I also like to point out (again) that the requirement is only a limitation of our API. OpenSSL is able to acquire and return the peer's certificate with any mode. In the past somebody decided to map 'no certificate' to None and 'no verification' to empty dict.

If we don't do that, then I think Christian's approach is a reasonable compromise given the late stage of the release cycle - it ensures the context can't get into the inconsistent verify_mode=CERT_NONE and check_hostname=True state, and leaves our options completely open for 3.5:

You are right, I'm trying to aim for the simplest and smallest solution possible. I'm well aware that the order of API calls is a limitation and that it feels a bit awkward, too. In my opinion there is no way this API can be screwed up. :) Any limitation can be lifted for 3.5 but we can't make it more restrict in future versions. And there is ssl.create_default_context(), too. It creates a context with all security-related bits flipped on. Christian

On 30 nov. 2013, at 19:29, Christian Heimes wrote:

With CERT_REQUIRED OpenSSL verifies that the peer's certificate is directly or indirectly signed by a trusted root certification authority. With Python 3.4 the ssl module is able to use/load the system's trusted root certs on all major systems (Linux, Mac, BSD, Windows). On Linux and BSD it requires a properly configured system openssl to locate the root certs. This usually works out of the box. On Mac Apple's openssl build is able to use the keychain API of OSX. I have added code for Windows' system store.

Note that only Apple's build of OpenSSL integrates with keychain, other builds don't. The patch for keychain integration is on Apple's open source site but that isn't very helpful because that code uses a private API to do most of the work. This almost certainly means that users of fink, macports and the like cannot use the system keystore. It is probably possible to use the Keychain API to verify certificates, I haven't seriously looked into that yet and there is a risk of using higher level APIs: those tend to not like calling fork without calling execv soon after and that could break existing scripts. Ronald