[Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Cory Benfield]

> On 1 Jun 2017, at 20:59, Steve Dower <steve.dower at python.org> wrote:
> 
> On 01Jun2017 1010, Nathaniel Smith wrote:
>> I believe that for answering this question about the ssl module, it's really only Linux users that matter, since pip/requests/everyone else pushing for this only want to use ssl.MemoryBIO on Linux. Their plan on Windows/MacOS (IIUC) is to stop using the ssl module entirely in favor of new ctypes bindings for their respective native TLS libraries.
>> (And yes, in principle it might be possible to write new ctypes-based bindings for openssl, but (a) this whole project is already teetering on the verge of being impossible given the resources available, so adding any major extra deliverable is likely to sink the whole thing, and (b) compared to the proprietary libraries, openssl is *much* harder and riskier to wrap at the ctypes level because it has different/incompatible ABIs depending on its micro version and the vendor who distributed it. This is why manylinux packages that need openssl have to ship their own, but pip can't and shouldn't ship its own openssl for many hopefully obvious reasons.)
> 
> How much of a stop-gap would it be (for Windows at least) to override OpenSSL's certificate validation with a call into the OS? This leaves most of the work with OpenSSL, but lets the OS say yes/no to the certificates based on its own configuration.
> 
> For Windows, this is under 100 lines of C code in (probably) _ssl, and while I think an SChannel based approach is the better way to go long-term,[1] offering platform-specific certificate validation as the default in 2.7 is far more palatable than backporting new public API.

It’s entirely do-able. This is where I reveal just how long I’ve been fretting over this problem: https://pypi.python.org/pypi/certitude <https://pypi.python.org/pypi/certitude>. Ignore the description, it’s wildly out-of-date: let me summarise the library instead.

Certitude is a Python library that uses CFFI and Rust to call into the system certificate validation libraries on macOS and Windows using a single unified API. Under the covers it has a whole bunch of Rust code that translates from what OpenSSL can give you (a list of certificates in the peer cert chain in DER format) and into what those two operating systems expect. The Rust code for Windows is here[1] and is about as horrifying a chunk of Rust as you can imagine seeing (the Windows API does not translate very neatly into Rust so the word “unsafe” appears a lot), but it does appear to work, at least in the mainline cases and in the few tests I have. The macOS code is here[2] and is moderately less horrifying, containing no instances of the word “unsafe”.

I lifted this approach from Chrome, because at the moment this is what they do: they use their custom fork of OpenSSL (BoringSSL) to do the actual TLS protocol manipulation, but hand the cert chain verification off to platform-native libraries on Windows and macOS.

I have never productised this library because ultimately I never had the time to spend writing a sufficiently broad test-case to confirm to me that it worked in all cases. There are very real risks in calling these APIs directly because if you get it wrong it’s easy to fail open.

It should be noted that right now certitude only works with, surprise, PyOpenSSL. Partly this is because the standard library does not expose SSL_get_peer_cert_chain, but even if it did that wouldn’t be enough as OpenSSL with VERIFY_NONE does not actually *save* the peer cert chain anywhere. That means even with PyOpenSSL the only way to get the peer cert chain is to hook into the verify callback and save off the certs as they come in, a gloriously absurd solution that is impossible with pure-Python code from the ssl module.

While this approach could work with _ssl.c, it ultimately doesn’t resolve the issue. It involves writing a substantial amount of new code which needs to be maintained by the ssl module maintainers. All of this code needs to be tested *thoroughly*, because python-dev would be accepting responsibility for the incredibly damaging potential CVEs in that code. And it doesn’t get python-dev out of the business of shipping OpenSSL on macOS and Windows, meaning that python-dev continues to bear the burden of OpenSSL CVEs, as well as the brand new CVEs that it is at risk of introducing.

Oh, and it can’t be backported to Python 2.7 or any of the bugfix-only Python 3 releases, and as I just noted the ssl module has never made it possible to use this approach from outside CPython. So it’s strictly just as bad as the situation PEP 543 is in, but with more C code. Doesn’t sound like a winning description to me. ;)

Cory

[1]: https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/windows.rs <https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/windows.rs>
[2]: https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/osx.rs <https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/osx.rs>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170602/7c8f269c/attachment-0001.html>