Mailman 3 August 2018 - Twisted-web

Twisted Web Agent configurable Elliptic Curves settings
by Paul Tremberth 17 Dec '21

17 Dec '21

Hello, The other day, we had a Scrapy user report an issue connecting to https://www.skelbiu.lt/ with OpenSSL 1.1 [1] To not mix scrapy's things with Twisted Web, I used this (adapted from official docs): #--------------- from __future__ import print_function from twisted.internet import reactor from twisted.web.client import Agent from twisted.web.http_headers import Headers agent = Agent(reactor) d = agent.request( 'GET', 'https://www.skelbiu.lt/', Headers({'User-Agent': ['Twisted Web Client Example']}), None) def cbResponse(ignored): print('Response received') d.addCallback(cbResponse) def cbShutdown(ignored): print(ignored) reactor.stop() d.addBoth(cbShutdown) reactor.run() #--------------- And I did get a Handshake failure too: $ python twistedtest.py [Failure instance: Traceback (failure with no frames): <class 'twisted.web._newclient.ResponseNeverReceived'>: [<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines', 'ssl3_read_bytes', 'sslv3 alert handshake failure')]>] ] It seems this happens (at least) with OpenSSL 1.1.0e (currently in Debian 9 sid [2]) It does not happen (for me) with OpenSSL 1.0.2g for example. I dug into this this afternoon and narrowed it down to the use of _defaultCurveName = u"prime256v1" in twisted.internet._sslverify.py I tried patching the current trunk with _defaultCurveName = u"secp384r1" (the EC that ssllabs.com reports) and it did work. Looking at ClientHello messages for openssl 1.0.2 and 1.1 [4]: with 1.1, only 1 Elliptic Curve is sent by Twisted Web Agent, secp256r1 openssl v1.1 client uses 4 by default: ecdh_x25519, secp256r1, secp521r1, secp384r1 I was wondering what is the proper way to configure requested Elliptic Curves. I haven't seen any interface for this, contrary to ciphers with acceptableCiphers. Thank you for your input. Best, Paul. [1] https://github.com/scrapy/scrapy/issues/2717 [2] https://packages.debian.org/fr/source/sid/openssl [3] https://github.com/twisted/twisted/blob/78679af87e349721a167f35bef239e192e9… [4] https://github.com/scrapy/scrapy/issues/2717#issuecomment-297464034

2 2

Test
by Philippe Lafoucrière 05 Aug '21

05 Aug '21

Hi all just a test

3 2

Re: [Twisted-web] Twisted-web Digest, Vol 151, Issue 2
by Jason Harrison 09 Aug '18

09 Aug '18

On Thu, 9 Aug 2018 at 09:31 am Jean-Paul Calderone < exarkun(a)twistedmatrix.com> wrote: > In the code you posted, DelayedResponse is the root resource. The root > resource *always* has to provide a child resource. If someone requests > /foo then the child is "foo". If someone requests "/" then the child is > "". > > isLeaf = True is a shortcut that stops traversal and uses the current > resource as the requested resource. Without this, DelayedResponse must > have a child or it can only produce a 404. > > Maybe that explains it? If there are examples in the documentation that > have root resources with no children and without isLeaf = True then perhaps > they are also broken and should be fixed. If you see any, please point > them out. > > Thank you for the explanation! Let's see if I understand your explanation and the variation in the examples. https://twistedmatrix.com/documents/current/web/howto/web-in-60/custom-code… does not have isLeaf = True because it has: root = Resource() root.putChild(b"buy", PaymentRequired()) factory = Site(root) https://twistedmatrix.com/documents/current/web/howto/web-in-60/dynamic-con… does have isLeaf = True because it does not have a .putChild() The explanation about isLeaf could be improved by: Setting isLeaf to True indicates that ClockPage resources will never have any children. Without this, the ClockPage resources must have a child or it can only produce a 404. https://twistedmatrix.com/documents/current/web/howto/web-in-60/dynamic-dis… does not have isLeaf = True because it defines getChild for the root class Calendar So does not have isLeaf = True because it was written as an .rpy script?? http://twistedmatrix.com/documents/current/web/howto/web-in-60/asynchronous… Thank you again, -Jason

2 1

Re: [Twisted-web] Asynchronous responses example from documentation -- why import time??
by Jason Harrison 09 Aug '18

09 Aug '18

> On Wed, 8 Aug 2018 at 11:58 AM Jean-Paul Calderone < > exarkun(a)twistedmatrix.com> wrote: One possible explanation is that it has nothing to do with import time per > se. Instead, *any* change to your source file would have fixed the > problem. This could be because you had "stale" pyc files lying around > (cached bytecode the interpreter *thought* was up-to-date with your source > but was actually outdated). It could also be that your server process was > left running and was still using your old code. Then, for some reason, > after you added the import time your server got the new version of the code > (perhaps you're relying on an auto-reloader and it missed a change, for > example, or you just forgot to restart the server yourself). > > import time itself is definitely not a requirement for arbitrary Twisted > Web-based programs to return a response other than 404. > > Jean-Paul > > Hi Jean-Paul, I made many changes to the example in order to get it to work. So perhaps it's a caching issue, but I can't find any caches. Here's what I'm developing on: - OSX 10.13.6 - I'm using Sublime text, no auto reloaders afaik - python3 --version : Python 3.7.0 - pip3 list: Package Version -------------- ------- attrs 18.1.0 Automat 0.7.0 constantly 15.1.0 dlib 19.15.0 hyperlink 18.0.0 idna 2.7 incremental 17.5.0 numpy 1.15.0 pip 18.0 PyHamcrest 1.9.0 setuptools 39.2.0 six 1.11.0 Twisted 18.7.0 wheel 0.31.1 zope.interface 4.5.0 - running the server using ./asynchronous.py - I don't have any .pyc files in the script directory, possibly because I "chmod +x" the source file and included the #! at the top - I added logging and while the main program (setup of Twisted reactor) logged, the DelayedResource class did not - I added parts of the example at https://twistedmatrix.com/documents/current/web/howto/web-in-60/dynamic-con… - I changed "resource = DelayedResource()" to "resource = ClockPage()" - I changed the render_GET handler to return the Clock page example output - etc until I stopped getting a 404 - I restarted the server many many times I removed and backed out every change until I got a 404, concluding it was the "import time" line. Any other ideas?

2 2

Asynchronous responses example from documentation -- why import time??
by Jason Harrison 08 Aug '18

08 Aug '18

Hi, I am learning twisted-web by attempting each of the examples in the documentation and making any (small) changes necessary for Python 3. However when I got to the Aysnchronous Responses example I was not able to get it to work. The example is at: https://twistedmatrix.com/documents/current/web/howto/web-in-60/asynchronou… I have adapted it to be a self-contained "server" rather than a .rpy snippet but the response was always a 404. After much head scratching and comparison to other examples, I discovered that by adding "import time" to my "server" it started working. Why is "import time" required. What is happening without the "import time"? -Jason Cell: 604 644 8611 Email: drjasonharrison(a)gmail.com LinkedIn: http://www.linkedin.com/in/drjasonharrison Twitter: http://twitter.com/drjasonharrison #!/usr/bin/env python3 # https://twistedmatrix.com/documents/current/web/howto/web-in-60/asynchronou… # Asynchronous responses, # # try # - http://localhost:8888 import sys from twisted.web.server import Site from twisted.web.resource import Resource from twisted.internet import reactor, endpoints from twisted.web.resource import NoResource from twisted.web.server import NOT_DONE_YET from twisted.python import log import time #??? log.startLogging(sys.stdout) observer = log.PythonLoggingObserver() observer.start() class DelayedResource(Resource): def _delayedRender(self, request): log.msg("_delayedRender request {request!r}", request = request) request.write(b"<html><body>Sorry to keep you waiting.</body></html>") request.finish() def render_GET(self, request): log.msg("get request {request!r}", request = request) reactor.callLater(5, self._delayedRender, request) return NOT_DONE_YET resource = DelayedResource() factory = Site(resource) endpoint = endpoints.TCP4ServerEndpoint(reactor, 8888) endpoint.listen(factory) log.msg("running reactor") reactor.run()

2 1