Mailman 3 June 2018 - Twisted-web

Twisted Web Agent configurable Elliptic Curves settings
by Paul Tremberth Dec. 16, 2021

Dec. 16, 2021

Hello, The other day, we had a Scrapy user report an issue connecting to https://www.skelbiu.lt/ with OpenSSL 1.1 [1] To not mix scrapy's things with Twisted Web, I used this (adapted from official docs): #--------------- from __future__ import print_function from twisted.internet import reactor from twisted.web.client import Agent from twisted.web.http_headers import Headers agent = Agent(reactor) d = agent.request( 'GET', 'https://www.skelbiu.lt/', Headers({'User-Agent': ['Twisted Web Client Example']}), None) def cbResponse(ignored): print('Response received') d.addCallback(cbResponse) def cbShutdown(ignored): print(ignored) reactor.stop() d.addBoth(cbShutdown) reactor.run() #--------------- And I did get a Handshake failure too: $ python twistedtest.py [Failure instance: Traceback (failure with no frames): <class 'twisted.web._newclient.ResponseNeverReceived'>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]>] ] It seems this happens (at least) with OpenSSL 1.1.0e (currently in Debian 9 sid [2]) It does not happen (for me) with OpenSSL 1.0.2g for example. I dug into this this afternoon and narrowed it down to the use of _defaultCurveName = u"prime256v1" in twisted.internet._sslverify.py I tried patching the current trunk with _defaultCurveName = u"secp384r1" (the EC that ssllabs.com reports) and it did work. Looking at ClientHello messages for openssl 1.0.2 and 1.1 [4]: with 1.1, only 1 Elliptic Curve is sent by Twisted Web Agent, secp256r1 openssl v1.1 client uses 4 by default: ecdh_x25519, secp256r1, secp521r1, secp384r1 I was wondering what is the proper way to configure requested Elliptic Curves. I haven't seen any interface for this, contrary to ciphers with acceptableCiphers. Thank you for your input. Best, Paul. [1] https://github.com/scrapy/scrapy/issues/2717 [2] https://packages.debian.org/fr/source/sid/openssl [3] https://github.com/twisted/twisted/blob/78679af87e349721a167f35bef239e192e9… [4] https://github.com/scrapy/scrapy/issues/2717#issuecomment-297464034

2 2

Test
by Philippe Lafoucrière Aug. 5, 2021

Aug. 5, 2021

Hi all just a test

3 2

Changing the order of the request header lines that get sent.
by richard77＠airmail.cc June 30, 2018

June 30, 2018

Hello subscribers I am using scrapy 1.4.0 which makes use of twisted 17.9.0 to send and receive HTTP requests and responses (https://github.com/scrapy/scrapy/blob/master/scrapy/core/downloader/handler… ScrapyAgent.download_request()). I would like to influence the order in which the request header lines get sent. With scrapy the request header is always handled as a python dictionary and therefore can not be sorted. So I was looking at the twisted source code to find the place where it sends the request header and discontinues to use a dictionary (where I would be able to modify the order at this place). Unfortunately I couldn't find it till now. Agent.request() gets the request headers as a dictionary and then returns self._requestWithEndpoint() with the headers dictionary in https://github.com/twisted/twisted/blob/trunk/src/twisted/web/client.py on line 1655. _AgentBase._requestWithEndpoint() on line 1419 then seems to send the request but where can I change the order of the headers? Hopefully you can help me with the problem.

1 0