The PEP's rationale is now "This PEP will help facilitate the future
adoption
of :pep:`543` across all supported Python versions, which will improve
security
for both Python 2 and Python 3 users."
What exactly are these security improvements? My understanding (which may
well be incorrect) is that the security improvements come in the future
when the PEP 543 APIs are implemented on top of the various platform-native
security libraries. These changes will not be present until some future 3.x
release, and will not be backported to 2.7 (without another PEP, which I
expect would be even more contentious than this one). What then is the
security improvement for Python 2 users?
In Tornado, I have not felt any urgency to replace wrap_socket with
MemoryBIO. Is there a security-related reason I should do so sooner rather
than later? (I'd also urge Cory and any other wrap_socket skeptics on the
requests team to reconsider - Tornado's SSLIOStream works well. The
asynchronous use of wrap_socket isn't so subtle and risky with buffering)
-Ben
On Fri, Jun 2, 2017 at 8:07 AM Antoine Pitrou
Thanks you for all the explanations. So to summarize my opinion, I'm still -0.5 on this PEP. I would also like to see the issues Jython, Ubuntu et al. have mentioned solved before this is accepted.
Regards
Antoine.
Thanks Cory for the long explanation. Let me try to summarize (tell me if I'm wrong).
We have 3 options:
* Do nothing: reject the PEP 546 and let each project handles security on its own (current status co) * Write *new* C code, maybe using certitude as a starting point, to offload certifcate validation on Windows and macOS * Backport existing code from master to 2.7: MemoryBIO and SSLObject
Writing new code seems more risky and error-prone than backporting already "battle-tested" MemoryBIO from master. I also expect that writing code to validate certificate will be longer than the "100 lines of C code in (probably)" expected by Steve Dower.
rust-certitude counts around 700 lines of Rust and 80 lines of Python code. But maybe I misunderstood the purpose of certitude: Steve Dower asked to only validate a certificate, not load or export CA.
I counted 150 Python lines for SSLObject and 230 C lines for MemoryBIO.
Since the long term plan is to not use stdlib ssl but a new implementation on Windows and macOS, it seems worthless to backport MemoryBIO on Python 2.7. The PEP 546 (backport MemoryBIO) is a practical solution to provide a *smooth* transition from ssl to a new TLS API. The experience showed that hard changes like "run 2to3 and drop your Python 2 code" doesn't work in practice. Users want a transition plan with small steps.
Victor
2017-06-02 11:08 GMT+02:00 Cory Benfield
: On 1 Jun 2017, at 20:59, Steve Dower
wrote: On 01Jun2017 1010, Nathaniel Smith wrote:
I believe that for answering this question about the ssl module, it's
really
only Linux users that matter, since pip/requests/everyone else pushing for this only want to use ssl.MemoryBIO on Linux. Their plan on Windows/MacOS (IIUC) is to stop using the ssl module entirely in favor of new ctypes bindings for their respective native TLS libraries. (And yes, in principle it might be possible to write new ctypes-based bindings for openssl, but (a) this whole project is already teetering on the verge of being impossible given the resources available, so adding any major extra deliverable is likely to sink the whole thing, and (b) compared to the proprietary libraries, openssl is *much* harder and riskier to wrap at
ctypes level because it has different/incompatible ABIs depending on its micro version and the vendor who distributed it. This is why manylinux packages that need openssl have to ship their own, but pip can't and shouldn't ship its own openssl for many hopefully obvious reasons.)
How much of a stop-gap would it be (for Windows at least) to override OpenSSL's certificate validation with a call into the OS? This leaves most of the work with OpenSSL, but lets the OS say yes/no to the certificates based on its own configuration.
For Windows, this is under 100 lines of C code in (probably) _ssl, and while I think an SChannel based approach is the better way to go long-term,[1] offering platform-specific certificate validation as the default in 2.7 is far more palatable than backporting new public API.
It’s entirely do-able. This is where I reveal just how long I’ve been fretting over this problem: https://pypi.python.org/pypi/certitude. Ignore the description, it’s wildly out-of-date: let me summarise the library instead.
Certitude is a Python library that uses CFFI and Rust to call into the system certificate validation libraries on macOS and Windows using a single unified API. Under the covers it has a whole bunch of Rust code that translates from what OpenSSL can give you (a list of certificates in
peer cert chain in DER format) and into what those two operating systems expect. The Rust code for Windows is here[1] and is about as horrifying a chunk of Rust as you can imagine seeing (the Windows API does not
very neatly into Rust so the word “unsafe” appears a lot), but it does appear to work, at least in the mainline cases and in the few tests I have. The macOS code is here[2] and is moderately less horrifying, containing no instances of the word “unsafe”.
I lifted this approach from Chrome, because at the moment this is what
do: they use their custom fork of OpenSSL (BoringSSL) to do the actual TLS protocol manipulation, but hand the cert chain verification off to platform-native libraries on Windows and macOS.
I have never productised this library because ultimately I never had
time to spend writing a sufficiently broad test-case to confirm to me
it worked in all cases. There are very real risks in calling these APIs directly because if you get it wrong it’s easy to fail open.
It should be noted that right now certitude only works with, surprise, PyOpenSSL. Partly this is because the standard library does not expose SSL_get_peer_cert_chain, but even if it did that wouldn’t be enough as OpenSSL with VERIFY_NONE does not actually *save* the peer cert chain anywhere. That means even with PyOpenSSL the only way to get the peer cert chain is to hook into the verify callback and save off the certs as
come in, a gloriously absurd solution that is impossible with
On Fri, 2 Jun 2017 11:42:58 +0200 Victor Stinner
wrote: the the translate they the that they pure-Python code from the ssl module.
While this approach could work with _ssl.c, it ultimately doesn’t resolve the issue. It involves writing a substantial amount of new code which needs to be maintained by the ssl module maintainers. All of this code needs to be tested *thoroughly*, because python-dev would be accepting responsibility for the incredibly damaging potential CVEs in that code. And it doesn’t get python-dev out of the business of shipping OpenSSL on macOS and Windows, meaning that python-dev continues to bear the burden of OpenSSL CVEs, as well as the brand new CVEs that it is at risk of introducing.
Oh, and it can’t be backported to Python 2.7 or any of the bugfix-only Python 3 releases, and as I just noted the ssl module has never made it possible to use this approach from outside CPython. So it’s strictly just as bad as the situation PEP 543 is in, but with more C code. Doesn’t sound like a winning description to me. ;)
Cory
[1]:
https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/wind...
[2]:
https://github.com/Lukasa/rust-certitude/blob/master/rust-certitude/src/osx....
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe:
https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.co...
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gma...
_______________________________________________ Python-Dev mailing list Python-Dev@python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/ben%40bendarnell.com