Hello,
The other day, we had a Scrapy user report an issue connecting to
https://www.skelbiu.lt/ with OpenSSL 1.1 [1]
Thanks for passing this issue on! Always nice to have users engaging directly with us rather than trying janky workarounds :).
To not mix scrapy's things with Twisted Web, I used this (adapted from official docs):
That's a good example to use, yes.
And I did get a Handshake failure too:
$ python twistedtest.py
[Failure instance: Traceback (failure with no frames): <class 'twisted.web._newclient.ResponseNeverReceived'>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]>]
]
It seems this happens (at least) with OpenSSL 1.1.0e (currently in Debian 9 sid [2])
It does not happen (for me) with OpenSSL 1.0.2g for example.
What platform are you on? How do you know what version of OpenSSL you're using? (It can sometimes be quite tricky to suss out what OpenSSL twisted is using unless you know the internals fairly well, unfortunately; 'twist --version' really ought to print it out.)
I dug into this this afternoon and narrowed it down to the use of
_defaultCurveName = u"prime256v1"
in twisted.internet._sslverify.py
I tried patching the current trunk with _defaultCurveName = u"secp384r1" (the EC that
ssllabs.com reports)
and it did work.
Looking at ClientHello messages for openssl 1.0.2 and 1.1 [4]:
with 1.1, only 1 Elliptic Curve is sent by Twisted Web Agent, secp256r1
What about with 1.0.2?
openssl v1.1 client uses 4 by default: ecdh_x25519, secp256r1, secp521r1, secp384r1
I was wondering what is the proper way to configure requested Elliptic Curves.
I haven't seen any interface for this, contrary to ciphers with acceptableCiphers.
It's possible that there should be an interface for this, but, your issue should not be fixed with an API to work around this bug. It should be fixed with a fix for this bug.
My understanding is that Twisted just configures one curve, whereas OpenSSL configures 4.
Twisted should just configure all 4 (unless there's some security reason not to match OpenSSL's behavior, which we should probably check on).
Twisted should just fix this, not make it configurable. Later, for security testing purposes, we may want to make it a fine-grained configurable thing, but for right now the priority should be getting correct behavior into a release.
Thank you for your input.