On Apr 26, 2017, at 10:32 AM, Paul Tremberth wrote:

Hello,

The other day, we had a Scrapy user report an issue connecting to https://www.skelbiu.lt/ https://www.skelbiu.lt/ with OpenSSL 1.1 [1]

Thanks for passing this issue on! Always nice to have users engaging directly with us rather than trying janky workarounds :).

To not mix scrapy's things with Twisted Web, I used this (adapted from official docs):

That's a good example to use, yes.

And I did get a Handshake failure too:

$ python twistedtest.py [Failure instance: Traceback (failure with no frames): : [] ]

It seems this happens (at least) with OpenSSL 1.1.0e (currently in Debian 9 sid [2]) It does not happen (for me) with OpenSSL 1.0.2g for example.

What platform are you on? How do you know what version of OpenSSL you're using? (It can sometimes be quite tricky to suss out what OpenSSL twisted is using unless you know the internals fairly well, unfortunately; 'twist --version' really ought to print it out.)

I dug into this this afternoon and narrowed it down to the use of _defaultCurveName = u"prime256v1" in twisted.internet._sslverify.py

I tried patching the current trunk with _defaultCurveName = u"secp384r1" (the EC that ssllabs.com http://ssllabs.com/ reports) and it did work.

Looking at ClientHello messages for openssl 1.0.2 and 1.1 [4]: with 1.1, only 1 Elliptic Curve is sent by Twisted Web Agent, secp256r1

What about with 1.0.2?

openssl v1.1 client uses 4 by default: ecdh_x25519, secp256r1, secp521r1, secp384r1

I was wondering what is the proper way to configure requested Elliptic Curves. I haven't seen any interface for this, contrary to ciphers with acceptableCiphers.

It's possible that there should be an interface for this, but, your issue should not be fixed with an API to work around this bug. It should be fixed with a fix for this bug. My understanding is that Twisted just configures one curve, whereas OpenSSL configures 4. Twisted should just configure all 4 (unless there's some security reason not to match OpenSSL's behavior, which we should probably check on). Just guessing based on what I see here, I imagine that would mean getting rid of the _ecCurve attribute, and instead having an _ecCurves = something; then, getting rid of the call to SSL_CTX_set_tmp_ecdh and instead using SSL_CTX_set1_curves, as https://github.com/openssl/openssl/commit/6977e8ee4a718a76351ba5275a9f0be4e5... https://github.com/openssl/openssl/commit/6977e8ee4a718a76351ba5275a9f0be4e5... seems to indicate that's how you have to request multiple curves at once. Twisted should just fix this, not make it configurable. Later, for security testing purposes, we may want to make it a fine-grained configurable thing, but for right now the priority should be getting correct behavior into a release.

Thank you for your input.

Thanks for using Twisted!