Mailman 3 May 2017 - Twisted-web

Twisted Web Agent configurable Elliptic Curves settings
by Paul Tremberth Dec. 16, 2021

Dec. 16, 2021

Hello, The other day, we had a Scrapy user report an issue connecting to https://www.skelbiu.lt/ with OpenSSL 1.1 [1] To not mix scrapy's things with Twisted Web, I used this (adapted from official docs): #--------------- from __future__ import print_function from twisted.internet import reactor from twisted.web.client import Agent from twisted.web.http_headers import Headers agent = Agent(reactor) d = agent.request( 'GET', 'https://www.skelbiu.lt/', Headers({'User-Agent': ['Twisted Web Client Example']}), None) def cbResponse(ignored): print('Response received') d.addCallback(cbResponse) def cbShutdown(ignored): print(ignored) reactor.stop() d.addBoth(cbShutdown) reactor.run() #--------------- And I did get a Handshake failure too: $ python twistedtest.py [Failure instance: Traceback (failure with no frames): <class 'twisted.web._newclient.ResponseNeverReceived'>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]>] ] It seems this happens (at least) with OpenSSL 1.1.0e (currently in Debian 9 sid [2]) It does not happen (for me) with OpenSSL 1.0.2g for example. I dug into this this afternoon and narrowed it down to the use of _defaultCurveName = u"prime256v1" in twisted.internet._sslverify.py I tried patching the current trunk with _defaultCurveName = u"secp384r1" (the EC that ssllabs.com reports) and it did work. Looking at ClientHello messages for openssl 1.0.2 and 1.1 [4]: with 1.1, only 1 Elliptic Curve is sent by Twisted Web Agent, secp256r1 openssl v1.1 client uses 4 by default: ecdh_x25519, secp256r1, secp521r1, secp384r1 I was wondering what is the proper way to configure requested Elliptic Curves. I haven't seen any interface for this, contrary to ciphers with acceptableCiphers. Thank you for your input. Best, Paul. [1] https://github.com/scrapy/scrapy/issues/2717 [2] https://packages.debian.org/fr/source/sid/openssl [3] https://github.com/twisted/twisted/blob/78679af87e349721a167f35bef239e192e9… [4] https://github.com/scrapy/scrapy/issues/2717#issuecomment-297464034

2 2

Test
by Philippe Lafoucrière Aug. 5, 2021

Aug. 5, 2021

Hi all just a test

3 2

Question regarding custom contextFactory for twisted.web.client.Agent
by Jason Litzinger July 5, 2017

July 5, 2017

Hello, I have an optimization question in the realm of "before I do something foolish..." regarding the contextFactory passed to twisted.web.client.Agent. Background: I have (several) twisted applications running on a Cortex-A5 system, one of which interfaces with a web service using the twisted.web.client.Agent API*. The problem I've observed is that requests to this web service result in a higher than desired CPU load. To understand the source of the load I isolated the server interactions and profiled. Investigation: The profile data revealed the load in question stemmed from TLS related calls, specifically optionsForClientTLS, which is called once per connection. In looking (briefly) at what optionsForClientTLS does, it seemed (perhaps wrongly) that there was an opportunity for optimization in my specific case. The optimization is to create a custom contextFactory that caches the connection creator since the host is not changing. I prototyped an implementation and the load was reduced by a factor of ten. Question: Is it safe/sane to reuse a connection creator if the host to which I'm connecting is not going to change? Thanks, -Jason Litzinger * Technically this code is inside an open source third party SDK, but were I in their shoes I'd write it the same way, so I've no issues with their implementation.

3 15