[Python-Dev] RFC: Backport ssl.MemoryBIO and ssl.SSLObject to Python 2.7

Ben Darnell ben at bendarnell.com
Fri Jun 2 09:43:21 EDT 2017


The PEP's rationale is now "This PEP will help facilitate the future
adoption
of :pep:`543` across all supported Python versions, which will improve
security
for both Python 2 and Python 3 users."

What exactly are these security improvements? My understanding (which may
well be incorrect) is that the security improvements come in the future
when the PEP 543 APIs are implemented on top of the various platform-native
security libraries. These changes will not be present until some future 3.x
release, and will not be backported to 2.7 (without another PEP, which I
expect would be even more contentious than this one). What then is the
security improvement for Python 2 users?

In Tornado, I have not felt any urgency to replace wrap_socket with
MemoryBIO. Is there a security-related reason I should do so sooner rather
than later? (I'd also urge Cory and any other wrap_socket skeptics on the
requests team to reconsider - Tornado's SSLIOStream works well. The
asynchronous use of wrap_socket isn't so subtle and risky with buffering)

-Ben

On Fri, Jun 2, 2017 at 8:07 AM Antoine Pitrou <solipsis at pitrou.net> wrote:

>
> Thanks you for all the explanations.  So to summarize my opinion, I'm
> still -0.5 on this PEP.  I would also like to see the issues Jython,
> Ubuntu et al. have mentioned solved before this is accepted.
>
> Regards
>
> Antoine.
>
>
>
> On Fri, 2 Jun 2017 11:42:58 +0200
> Victor Stinner <victor.stinner at gmail.com> wrote:
> > Thanks Cory for the long explanation. Let me try to summarize (tell me
> > if I'm wrong).
> >
> > We have 3 options:
> >
> > * Do nothing: reject the PEP 546 and let each project handles security
> > on its own (current status co)
> > * Write *new* C code, maybe using certitude as a starting point, to
> > offload certifcate validation on Windows and macOS
> > * Backport existing code from master to 2.7: MemoryBIO and SSLObject
> >
> > Writing new code seems more risky and error-prone than backporting
> > already "battle-tested" MemoryBIO from master. I also expect that
> > writing code to validate certificate will be longer than the "100
> > lines of C code in (probably)" expected by Steve Dower.
> >
> > rust-certitude counts around 700 lines of Rust and 80 lines of Python
> > code. But maybe I misunderstood the purpose of certitude: Steve Dower
> > asked to only validate a certificate, not load or export CA.
> >
> > I counted 150 Python lines for SSLObject and 230 C lines for MemoryBIO.
> >
> > Since the long term plan is to not use stdlib ssl but a new
> > implementation on Windows and macOS, it seems worthless to backport
> > MemoryBIO on Python 2.7. The PEP 546 (backport MemoryBIO) is a
> > practical solution to provide a *smooth* transition from ssl to a new
> > TLS API. The experience showed that hard changes like "run 2to3 and
> > drop your Python 2 code" doesn't work in practice. Users want a
> > transition plan with small steps.
> >
> > Victor
> >
> > 2017-06-02 11:08 GMT+02:00 Cory Benfield <cory at lukasa.co.uk>:
> > >
> > > On 1 Jun 2017, at 20:59, Steve Dower <steve.dower at python.org> wrote:
> > >
> > > On 01Jun2017 1010, Nathaniel Smith wrote:
> > >
> > > I believe that for answering this question about the ssl module, it's
> really
> > > only Linux users that matter, since pip/requests/everyone else pushing
> for
> > > this only want to use ssl.MemoryBIO on Linux. Their plan on
> Windows/MacOS
> > > (IIUC) is to stop using the ssl module entirely in favor of new ctypes
> > > bindings for their respective native TLS libraries.
> > > (And yes, in principle it might be possible to write new ctypes-based
> > > bindings for openssl, but (a) this whole project is already teetering
> on the
> > > verge of being impossible given the resources available, so adding any
> major
> > > extra deliverable is likely to sink the whole thing, and (b) compared
> to the
> > > proprietary libraries, openssl is *much* harder and riskier to wrap at
> the
> > > ctypes level because it has different/incompatible ABIs depending on
> its
> > > micro version and the vendor who distributed it. This is why manylinux
> > > packages that need openssl have to ship their own, but pip can't and
> > > shouldn't ship its own openssl for many hopefully obvious reasons.)
> > >
> > >
> > > How much of a stop-gap would it be (for Windows at least) to override
> > > OpenSSL's certificate validation with a call into the OS? This leaves
> most
> > > of the work with OpenSSL, but lets the OS say yes/no to the
> certificates
> > > based on its own configuration.
> > >
> > > For Windows, this is under 100 lines of C code in (probably) _ssl, and
> while
> > > I think an SChannel based approach is the better way to go
> long-term,[1]
> > > offering platform-specific certificate validation as the default in
> 2.7 is
> > > far more palatable than backporting new public API.
> > >
> > >
> > > It’s entirely do-able. This is where I reveal just how long I’ve been
> > > fretting over this problem: https://pypi.python.org/pypi/certitude.
> Ignore
> > > the description, it’s wildly out-of-date: let me summarise the library
> > > instead.
> > >
> > > Certitude is a Python library that uses CFFI and Rust to call into the
> > > system certificate validation libraries on macOS and Windows using a
> single
> > > unified API. Under the covers it has a whole bunch of Rust code that
> > > translates from what OpenSSL can give you (a list of certificates in
> the
> > > peer cert chain in DER format) and into what those two operating
> systems
> > > expect. The Rust code for Windows is here[1] and is about as
> horrifying a
> > > chunk of Rust as you can imagine seeing (the Windows API does not
> translate
> > > very neatly into Rust so the word “unsafe” appears a lot), but it does
> > > appear to work, at least in the mainline cases and in the few tests I
> have.
> > > The macOS code is here[2] and is moderately less horrifying,
> containing no
> > > instances of the word “unsafe”.
> > >
> > > I lifted this approach from Chrome, because at the moment this is what
> they
> > > do: they use their custom fork of OpenSSL (BoringSSL) to do the actual
> TLS
> > > protocol manipulation, but hand the cert chain verification off to
> > > platform-native libraries on Windows and macOS.
> > >
> > > I have never productised this library because ultimately I never had
> the
> > > time to spend writing a sufficiently broad test-case to confirm to me
> that
> > > it worked in all cases. There are very real risks in calling these APIs
> > > directly because if you get it wrong it’s easy to fail open.
> > >
> > > It should be noted that right now certitude only works with, surprise,
> > > PyOpenSSL. Partly this is because the standard library does not expose
> > > SSL_get_peer_cert_chain, but even if it did that wouldn’t be enough as
> > > OpenSSL with VERIFY_NONE does not actually *save* the peer cert chain
> > > anywhere. That means even with PyOpenSSL the only way to get the peer
> cert
> > > chain is to hook into the verify callback and save off the certs as
> they
> > > come in, a gloriously absurd solution that is impossible with
> pure-Python
> > > code from the ssl module.
> > >
> > > While this approach could work with _ssl.c, it ultimately doesn’t
> resolve
> > > the issue. It involves writing a substantial amount of new code which
> needs
> > > to be maintained by the ssl module maintainers. All of this code needs
> to be
> > > tested *thoroughly*, because python-dev would be accepting
> responsibility
> > > for the incredibly damaging potential CVEs in that code. And it
> doesn’t get
> > > python-dev out of the business of shipping OpenSSL on macOS and
> Windows,
> > > meaning that python-dev continues to bear the burden of OpenSSL CVEs,
> as
> > > well as the brand new CVEs that it is at risk of introducing.
> > >
> > > Oh, and it can’t be backported to Python 2.7 or any of the bugfix-only
> > > Python 3 releases, and as I just noted the ssl module has never made it
> > > possible to use this approach from outside CPython. So it’s strictly
> just as
> > > bad as the situation PEP 543 is in, but with more C code. Doesn’t
> sound like
> > > a winning description to me. ;)
> > >
> > > Cory
> > >
> > > [1]:
> > >
> https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/windows.rs
> > > [2]:
> > >
> https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/osx.rs
> > >
> > > _______________________________________________
> > > Python-Dev mailing list
> > > Python-Dev at python.org
> > > https://mail.python.org/mailman/listinfo/python-dev
> > > Unsubscribe:
> > >
> https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com
> > >
> > _______________________________________________
> > Python-Dev mailing list
> > Python-Dev at python.org
> > https://mail.python.org/mailman/listinfo/python-dev
> > Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org
>
>
>
> _______________________________________________
> Python-Dev mailing list
> Python-Dev at python.org
> https://mail.python.org/mailman/listinfo/python-dev
> Unsubscribe:
> https://mail.python.org/mailman/options/python-dev/ben%40bendarnell.com
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20170602/9d28a3bb/attachment-0001.html>


More information about the Python-Dev mailing list