[Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

[Victor Stinner]

Thanks Cory for the long explanation. Let me try to summarize (tell me
if I'm wrong).

We have 3 options:

* Do nothing: reject the PEP 546 and let each project handles security
on its own (current status co)
* Write *new* C code, maybe using certitude as a starting point, to
offload certifcate validation on Windows and macOS
* Backport existing code from master to 2.7: MemoryBIO and SSLObject

Writing new code seems more risky and error-prone than backporting
already "battle-tested" MemoryBIO from master. I also expect that
writing code to validate certificate will be longer than the "100
lines of C code in (probably)" expected by Steve Dower.

rust-certitude counts around 700 lines of Rust and 80 lines of Python
code. But maybe I misunderstood the purpose of certitude: Steve Dower
asked to only validate a certificate, not load or export CA.

I counted 150 Python lines for SSLObject and 230 C lines for MemoryBIO.

Since the long term plan is to not use stdlib ssl but a new
implementation on Windows and macOS, it seems worthless to backport
MemoryBIO on Python 2.7. The PEP 546 (backport MemoryBIO) is a
practical solution to provide a *smooth* transition from ssl to a new
TLS API. The experience showed that hard changes like "run 2to3 and
drop your Python 2 code" doesn't work in practice. Users want a
transition plan with small steps.

Victor

2017-06-02 11:08 GMT+02:00 Cory Benfield <cory at lukasa.co.uk>:
>
> On 1 Jun 2017, at 20:59, Steve Dower <steve.dower at python.org> wrote:
>
> On 01Jun2017 1010, Nathaniel Smith wrote:
>
> I believe that for answering this question about the ssl module, it's really
> only Linux users that matter, since pip/requests/everyone else pushing for
> this only want to use ssl.MemoryBIO on Linux. Their plan on Windows/MacOS
> (IIUC) is to stop using the ssl module entirely in favor of new ctypes
> bindings for their respective native TLS libraries.
> (And yes, in principle it might be possible to write new ctypes-based
> bindings for openssl, but (a) this whole project is already teetering on the
> verge of being impossible given the resources available, so adding any major
> extra deliverable is likely to sink the whole thing, and (b) compared to the
> proprietary libraries, openssl is *much* harder and riskier to wrap at the
> ctypes level because it has different/incompatible ABIs depending on its
> micro version and the vendor who distributed it. This is why manylinux
> packages that need openssl have to ship their own, but pip can't and
> shouldn't ship its own openssl for many hopefully obvious reasons.)
>
>
> How much of a stop-gap would it be (for Windows at least) to override
> OpenSSL's certificate validation with a call into the OS? This leaves most
> of the work with OpenSSL, but lets the OS say yes/no to the certificates
> based on its own configuration.
>
> For Windows, this is under 100 lines of C code in (probably) _ssl, and while
> I think an SChannel based approach is the better way to go long-term,[1]
> offering platform-specific certificate validation as the default in 2.7 is
> far more palatable than backporting new public API.
>
>
> It’s entirely do-able. This is where I reveal just how long I’ve been
> fretting over this problem: https://pypi.python.org/pypi/certitude. Ignore
> the description, it’s wildly out-of-date: let me summarise the library
> instead.
>
> Certitude is a Python library that uses CFFI and Rust to call into the
> system certificate validation libraries on macOS and Windows using a single
> unified API. Under the covers it has a whole bunch of Rust code that
> translates from what OpenSSL can give you (a list of certificates in the
> peer cert chain in DER format) and into what those two operating systems
> expect. The Rust code for Windows is here[1] and is about as horrifying a
> chunk of Rust as you can imagine seeing (the Windows API does not translate
> very neatly into Rust so the word “unsafe” appears a lot), but it does
> appear to work, at least in the mainline cases and in the few tests I have.
> The macOS code is here[2] and is moderately less horrifying, containing no
> instances of the word “unsafe”.
>
> I lifted this approach from Chrome, because at the moment this is what they
> do: they use their custom fork of OpenSSL (BoringSSL) to do the actual TLS
> protocol manipulation, but hand the cert chain verification off to
> platform-native libraries on Windows and macOS.
>
> I have never productised this library because ultimately I never had the
> time to spend writing a sufficiently broad test-case to confirm to me that
> it worked in all cases. There are very real risks in calling these APIs
> directly because if you get it wrong it’s easy to fail open.
>
> It should be noted that right now certitude only works with, surprise,
> PyOpenSSL. Partly this is because the standard library does not expose
> SSL_get_peer_cert_chain, but even if it did that wouldn’t be enough as
> OpenSSL with VERIFY_NONE does not actually *save* the peer cert chain
> anywhere. That means even with PyOpenSSL the only way to get the peer cert
> chain is to hook into the verify callback and save off the certs as they
> come in, a gloriously absurd solution that is impossible with pure-Python
> code from the ssl module.
>
> While this approach could work with _ssl.c, it ultimately doesn’t resolve
> the issue. It involves writing a substantial amount of new code which needs
> to be maintained by the ssl module maintainers. All of this code needs to be
> tested *thoroughly*, because python-dev would be accepting responsibility
> for the incredibly damaging potential CVEs in that code. And it doesn’t get
> python-dev out of the business of shipping OpenSSL on macOS and Windows,
> meaning that python-dev continues to bear the burden of OpenSSL CVEs, as
> well as the brand new CVEs that it is at risk of introducing.
>
> Oh, and it can’t be backported to Python 2.7 or any of the bugfix-only
> Python 3 releases, and as I just noted the ssl module has never made it
> possible to use this approach from outside CPython. So it’s strictly just as
> bad as the situation PEP 543 is in, but with more C code. Doesn’t sound like
> a winning description to me. ;)
>
> Cory
>
> [1]:
> https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/windows.rs
> [2]:
> https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/osx.rs
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
>