Twisted Web Agent configurable Elliptic Curves settings
Hello,
The other day, we had a Scrapy user report an issue connecting to https://www.skelbiu.lt/ with OpenSSL 1.1 [1]
To not mix scrapy's things with Twisted Web, I used this (adapted from official docs):
#--------------- from __future__ import print_function
from twisted.internet import reactor from twisted.web.client import Agent from twisted.web.http_headers import Headers
agent = Agent(reactor)
d = agent.request( 'GET', 'https://www.skelbiu.lt/', Headers({'User-Agent': ['Twisted Web Client Example']}), None)
def cbResponse(ignored): print('Response received') d.addCallback(cbResponse)
def cbShutdown(ignored): print(ignored) reactor.stop() d.addBoth(cbShutdown)
reactor.run() #---------------
And I did get a Handshake failure too:
$ python twistedtest.py [Failure instance: Traceback (failure with no frames): <class 'twisted.web._newclient.ResponseNeverReceived'>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]>] ]
It seems this happens (at least) with OpenSSL 1.1.0e (currently in Debian 9 sid [2]) It does not happen (for me) with OpenSSL 1.0.2g for example.
I dug into this this afternoon and narrowed it down to the use of _defaultCurveName = u"prime256v1" in twisted.internet._sslverify.py
I tried patching the current trunk with _defaultCurveName = u"secp384r1" (the EC that ssllabs.com reports) and it did work.
Looking at ClientHello messages for openssl 1.0.2 and 1.1 [4]: with 1.1, only 1 Elliptic Curve is sent by Twisted Web Agent, secp256r1
openssl v1.1 client uses 4 by default: ecdh_x25519, secp256r1, secp521r1, secp384r1
I was wondering what is the proper way to configure requested Elliptic Curves. I haven't seen any interface for this, contrary to ciphers with acceptableCiphers.
Thank you for your input.
Best, Paul.
[1] https://github.com/scrapy/scrapy/issues/2717 [2] https://packages.debian.org/fr/source/sid/openssl [3] https://github.com/twisted/twisted/blob/78679af87e349721a167f35bef239e192e91... [4] https://github.com/scrapy/scrapy/issues/2717#issuecomment-297464034
On Apr 26, 2017, at 10:32 AM, Paul Tremberth paul.tremberth@gmail.com wrote:
Hello,
The other day, we had a Scrapy user report an issue connecting to https://www.skelbiu.lt/ https://www.skelbiu.lt/ with OpenSSL 1.1 [1]
Thanks for passing this issue on! Always nice to have users engaging directly with us rather than trying janky workarounds :).
To not mix scrapy's things with Twisted Web, I used this (adapted from official docs):
That's a good example to use, yes.
And I did get a Handshake failure too:
$ python twistedtest.py [Failure instance: Traceback (failure with no frames): <class 'twisted.web._newclient.ResponseNeverReceived'>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]>] ]It seems this happens (at least) with OpenSSL 1.1.0e (currently in Debian 9 sid [2]) It does not happen (for me) with OpenSSL 1.0.2g for example.
What platform are you on? How do you know what version of OpenSSL you're using? (It can sometimes be quite tricky to suss out what OpenSSL twisted is using unless you know the internals fairly well, unfortunately; 'twist --version' really ought to print it out.)
I dug into this this afternoon and narrowed it down to the use of _defaultCurveName = u"prime256v1" in twisted.internet._sslverify.py
I tried patching the current trunk with _defaultCurveName = u"secp384r1" (the EC that ssllabs.com http://ssllabs.com/ reports) and it did work.
Looking at ClientHello messages for openssl 1.0.2 and 1.1 [4]: with 1.1, only 1 Elliptic Curve is sent by Twisted Web Agent, secp256r1
What about with 1.0.2?
openssl v1.1 client uses 4 by default: ecdh_x25519, secp256r1, secp521r1, secp384r1
I was wondering what is the proper way to configure requested Elliptic Curves. I haven't seen any interface for this, contrary to ciphers with acceptableCiphers.
It's possible that there should be an interface for this, but, your issue should not be fixed with an API to work around this bug. It should be fixed with a fix for this bug.
My understanding is that Twisted just configures one curve, whereas OpenSSL configures 4.
Twisted should just configure all 4 (unless there's some security reason not to match OpenSSL's behavior, which we should probably check on).
Just guessing based on what I see here, I imagine that would mean getting rid of the _ecCurve attribute, and instead having an _ecCurves = something; then, getting rid of the call to SSL_CTX_set_tmp_ecdh and instead using SSL_CTX_set1_curves, as https://github.com/openssl/openssl/commit/6977e8ee4a718a76351ba5275a9f0be4e5... https://github.com/openssl/openssl/commit/6977e8ee4a718a76351ba5275a9f0be4e530eab5 seems to indicate that's how you have to request multiple curves at once.
Twisted should just fix this, not make it configurable. Later, for security testing purposes, we may want to make it a fine-grained configurable thing, but for right now the priority should be getting correct behavior into a release.
Thank you for your input.
Thanks for using Twisted!
What platform are you on? How do you know what version of OpenSSL you're using? (It can sometimes be quite tricky to suss out what OpenSSL twisted is
using unless you know the internals fairly well, unfortunately; 'twist --version' really ought to print it out.)
I'm on (X)Ubuntu 16.10. And the system uses 1.0.2g-1ubuntu9.1. But I built my own static wheel version of cryptography following https://cryptography.io/en/latest/installation/#static-wheels , with OPENSSL_VERSION="1.1.0e" and pip installed the wheel in another virtualenv to test it, along with twisted[tls]
I checked OpenSSL version using this (that's what scrapy also does when printing versions): $ python -c "import OpenSSL; print(OpenSSL.SSL.SSLeay_version(OpenSSL.SSL.SSLEAY_VERSION))" OpenSSL 1.1.0e 16 Feb 2017
What about with 1.0.2?
I just tested it again this morning with my system's default openssl:
$ python -c "import OpenSSL; print(OpenSSL.SSL.SSLeay_version(OpenSSL.SSL.SSLEAY_VERSION))" OpenSSL 1.0.2g 1 Mar 2016
with the same snippet using Twisted Web Agent, this is what wireshark shows in the ClientHello:
Extension: elliptic_curves Type: elliptic_curves (0x000a) Length: 28 Elliptic Curves Length: 26 Elliptic curves (13 curves) Elliptic curve: secp256r1 (0x0017) Elliptic curve: secp521r1 (0x0019) Elliptic curve: brainpoolP512r1 (0x001c) Elliptic curve: brainpoolP384r1 (0x001b) Elliptic curve: secp384r1 (0x0018) Elliptic curve: brainpoolP256r1 (0x001a) Elliptic curve: secp256k1 (0x0016) Elliptic curve: sect571r1 (0x000e) Elliptic curve: sect571k1 (0x000d) Elliptic curve: sect409k1 (0x000b) Elliptic curve: sect409r1 (0x000c) Elliptic curve: sect283k1 (0x0009) Elliptic curve: sect283r1 (0x000a)
participants (2)
-
Glyph Lefkowitz -
Paul Tremberth