[Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

Antoine Pitrou solipsis at pitrou.net
Fri Jun 2 08:06:22 EDT 2017

Thanks you for all the explanations.  So to summarize my opinion, I'm
still -0.5 on this PEP.  I would also like to see the issues Jython,
Ubuntu et al. have mentioned solved before this is accepted.



On Fri, 2 Jun 2017 11:42:58 +0200
Victor Stinner <victor.stinner at gmail.com> wrote:
> Thanks Cory for the long explanation. Let me try to summarize (tell me
> if I'm wrong).
> We have 3 options:
> * Do nothing: reject the PEP 546 and let each project handles security
> on its own (current status co)
> * Write *new* C code, maybe using certitude as a starting point, to
> offload certifcate validation on Windows and macOS
> * Backport existing code from master to 2.7: MemoryBIO and SSLObject
> Writing new code seems more risky and error-prone than backporting
> already "battle-tested" MemoryBIO from master. I also expect that
> writing code to validate certificate will be longer than the "100
> lines of C code in (probably)" expected by Steve Dower.
> rust-certitude counts around 700 lines of Rust and 80 lines of Python
> code. But maybe I misunderstood the purpose of certitude: Steve Dower
> asked to only validate a certificate, not load or export CA.
> I counted 150 Python lines for SSLObject and 230 C lines for MemoryBIO.
> Since the long term plan is to not use stdlib ssl but a new
> implementation on Windows and macOS, it seems worthless to backport
> MemoryBIO on Python 2.7. The PEP 546 (backport MemoryBIO) is a
> practical solution to provide a *smooth* transition from ssl to a new
> TLS API. The experience showed that hard changes like "run 2to3 and
> drop your Python 2 code" doesn't work in practice. Users want a
> transition plan with small steps.
> Victor
> 2017-06-02 11:08 GMT+02:00 Cory Benfield <cory at lukasa.co.uk>:
> >
> > On 1 Jun 2017, at 20:59, Steve Dower <steve.dower at python.org> wrote:
> >
> > On 01Jun2017 1010, Nathaniel Smith wrote:
> >
> > I believe that for answering this question about the ssl module, it's really
> > only Linux users that matter, since pip/requests/everyone else pushing for
> > this only want to use ssl.MemoryBIO on Linux. Their plan on Windows/MacOS
> > (IIUC) is to stop using the ssl module entirely in favor of new ctypes
> > bindings for their respective native TLS libraries.
> > (And yes, in principle it might be possible to write new ctypes-based
> > bindings for openssl, but (a) this whole project is already teetering on the
> > verge of being impossible given the resources available, so adding any major
> > extra deliverable is likely to sink the whole thing, and (b) compared to the
> > proprietary libraries, openssl is *much* harder and riskier to wrap at the
> > ctypes level because it has different/incompatible ABIs depending on its
> > micro version and the vendor who distributed it. This is why manylinux
> > packages that need openssl have to ship their own, but pip can't and
> > shouldn't ship its own openssl for many hopefully obvious reasons.)
> >
> >
> > How much of a stop-gap would it be (for Windows at least) to override
> > OpenSSL's certificate validation with a call into the OS? This leaves most
> > of the work with OpenSSL, but lets the OS say yes/no to the certificates
> > based on its own configuration.
> >
> > For Windows, this is under 100 lines of C code in (probably) _ssl, and while
> > I think an SChannel based approach is the better way to go long-term,[1]
> > offering platform-specific certificate validation as the default in 2.7 is
> > far more palatable than backporting new public API.
> >
> >
> > It’s entirely do-able. This is where I reveal just how long I’ve been
> > fretting over this problem: https://pypi.python.org/pypi/certitude. Ignore
> > the description, it’s wildly out-of-date: let me summarise the library
> > instead.
> >
> > Certitude is a Python library that uses CFFI and Rust to call into the
> > system certificate validation libraries on macOS and Windows using a single
> > unified API. Under the covers it has a whole bunch of Rust code that
> > translates from what OpenSSL can give you (a list of certificates in the
> > peer cert chain in DER format) and into what those two operating systems
> > expect. The Rust code for Windows is here[1] and is about as horrifying a
> > chunk of Rust as you can imagine seeing (the Windows API does not translate
> > very neatly into Rust so the word “unsafe” appears a lot), but it does
> > appear to work, at least in the mainline cases and in the few tests I have.
> > The macOS code is here[2] and is moderately less horrifying, containing no
> > instances of the word “unsafe”.
> >
> > I lifted this approach from Chrome, because at the moment this is what they
> > do: they use their custom fork of OpenSSL (BoringSSL) to do the actual TLS
> > protocol manipulation, but hand the cert chain verification off to
> > platform-native libraries on Windows and macOS.
> >
> > I have never productised this library because ultimately I never had the
> > time to spend writing a sufficiently broad test-case to confirm to me that
> > it worked in all cases. There are very real risks in calling these APIs
> > directly because if you get it wrong it’s easy to fail open.
> >
> > It should be noted that right now certitude only works with, surprise,
> > PyOpenSSL. Partly this is because the standard library does not expose
> > SSL_get_peer_cert_chain, but even if it did that wouldn’t be enough as
> > OpenSSL with VERIFY_NONE does not actually *save* the peer cert chain
> > anywhere. That means even with PyOpenSSL the only way to get the peer cert
> > chain is to hook into the verify callback and save off the certs as they
> > come in, a gloriously absurd solution that is impossible with pure-Python
> > code from the ssl module.
> >
> > While this approach could work with _ssl.c, it ultimately doesn’t resolve
> > the issue. It involves writing a substantial amount of new code which needs
> > to be maintained by the ssl module maintainers. All of this code needs to be
> > tested *thoroughly*, because python-dev would be accepting responsibility
> > for the incredibly damaging potential CVEs in that code. And it doesn’t get
> > python-dev out of the business of shipping OpenSSL on macOS and Windows,
> > meaning that python-dev continues to bear the burden of OpenSSL CVEs, as
> > well as the brand new CVEs that it is at risk of introducing.
> >
> > Oh, and it can’t be backported to Python 2.7 or any of the bugfix-only
> > Python 3 releases, and as I just noted the ssl module has never made it
> > possible to use this approach from outside CPython. So it’s strictly just as
> > bad as the situation PEP 543 is in, but with more C code. Doesn’t sound like
> > a winning description to me. ;)
> >
> > Cory
> >
> > [1]:
> > https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/windows.rs
> > [2]:
> > https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/osx.rs
> >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
> >  
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe: https://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org

More information about the Python-Dev mailing list