Hello,
The other day, we had a Scrapy user report an issue connecting to
https://www.skelbiu.lt/ with OpenSSL 1.1 [1]
To not mix scrapy's things with Twisted Web, I used this (adapted from
official docs):
#---------------
from __future__ import print_function
from twisted.internet import reactor
from twisted.web.client import Agent
from twisted.web.http_headers import Headers
agent = Agent(reactor)
d = agent.request(
'GET',
'https://www.skelbiu.lt/',
Headers({'User-Agent': ['Twisted Web Client Example']}),
None)
def cbResponse(ignored):
print('Response received')
d.addCallback(cbResponse)
def cbShutdown(ignored):
print(ignored)
reactor.stop()
d.addBoth(cbShutdown)
reactor.run()
#---------------
And I did get a Handshake failure too:
$ python twistedtest.py
[Failure instance: Traceback (failure with no frames): <class
'twisted.web._newclient.ResponseNeverReceived'>:
[<twisted.python.failure.Failure OpenSSL.SSL.Error: [('SSL routines',
'ssl3_read_bytes', 'sslv3 alert handshake failure')]>]
]
It seems this happens (at least) with OpenSSL 1.1.0e (currently in Debian 9
sid [2])
It does not happen (for me) with OpenSSL 1.0.2g for example.
I dug into this this afternoon and narrowed it down to the use of
_defaultCurveName = u"prime256v1"
in twisted.internet._sslverify.py
I tried patching the current trunk with _defaultCurveName = u"secp384r1"
(the EC that ssllabs.com reports)
and it did work.
Looking at ClientHello messages for openssl 1.0.2 and 1.1 [4]:
with 1.1, only 1 Elliptic Curve is sent by Twisted Web Agent, secp256r1
openssl v1.1 client uses 4 by default: ecdh_x25519, secp256r1, secp521r1,
secp384r1
I was wondering what is the proper way to configure requested Elliptic
Curves.
I haven't seen any interface for this, contrary to ciphers with
acceptableCiphers.
Thank you for your input.
Best,
Paul.
[1] https://github.com/scrapy/scrapy/issues/2717
[2] https://packages.debian.org/fr/source/sid/openssl
[3]
https://github.com/twisted/twisted/blob/78679af87e349721a167f35bef239e192e9…
[4] https://github.com/scrapy/scrapy/issues/2717#issuecomment-297464034
On Thu, 9 Aug 2018 at 09:31 am Jean-Paul Calderone <
exarkun(a)twistedmatrix.com> wrote:
> In the code you posted, DelayedResponse is the root resource. The root
> resource *always* has to provide a child resource. If someone requests
> /foo then the child is "foo". If someone requests "/" then the child is
> "".
>
> isLeaf = True is a shortcut that stops traversal and uses the current
> resource as the requested resource. Without this, DelayedResponse must
> have a child or it can only produce a 404.
>
> Maybe that explains it? If there are examples in the documentation that
> have root resources with no children and without isLeaf = True then perhaps
> they are also broken and should be fixed. If you see any, please point
> them out.
>
>
Thank you for the explanation! Let's see if I understand your explanation
and the variation in the examples.
https://twistedmatrix.com/documents/current/web/howto/web-in-60/custom-code…
does not have isLeaf = True because it has:
root = Resource()
root.putChild(b"buy", PaymentRequired())
factory = Site(root)
https://twistedmatrix.com/documents/current/web/howto/web-in-60/dynamic-con…
does have isLeaf = True because it does not have a .putChild()
The explanation about isLeaf could be improved by:
Setting isLeaf to True indicates that ClockPage resources will never have
any children.
Without this, the ClockPage resources must have a child or it can only
produce a 404.
https://twistedmatrix.com/documents/current/web/howto/web-in-60/dynamic-dis…
does not have isLeaf = True because it defines getChild for the root class
Calendar
So does not have isLeaf = True because it was written as an .rpy script??
http://twistedmatrix.com/documents/current/web/howto/web-in-60/asynchronous…
Thank you again,
-Jason
> On Wed, 8 Aug 2018 at 11:58 AM Jean-Paul Calderone <
> exarkun(a)twistedmatrix.com> wrote:
One possible explanation is that it has nothing to do with import time per
> se. Instead, *any* change to your source file would have fixed the
> problem. This could be because you had "stale" pyc files lying around
> (cached bytecode the interpreter *thought* was up-to-date with your source
> but was actually outdated). It could also be that your server process was
> left running and was still using your old code. Then, for some reason,
> after you added the import time your server got the new version of the code
> (perhaps you're relying on an auto-reloader and it missed a change, for
> example, or you just forgot to restart the server yourself).
>
> import time itself is definitely not a requirement for arbitrary Twisted
> Web-based programs to return a response other than 404.
>
> Jean-Paul
>
>
Hi Jean-Paul,
I made many changes to the example in order to get it to work. So perhaps
it's a caching issue, but I can't find any caches. Here's what I'm
developing on:
- OSX 10.13.6
- I'm using Sublime text, no auto reloaders afaik
- python3 --version : Python 3.7.0
- pip3 list:
Package Version
-------------- -------
attrs 18.1.0
Automat 0.7.0
constantly 15.1.0
dlib 19.15.0
hyperlink 18.0.0
idna 2.7
incremental 17.5.0
numpy 1.15.0
pip 18.0
PyHamcrest 1.9.0
setuptools 39.2.0
six 1.11.0
Twisted 18.7.0
wheel 0.31.1
zope.interface 4.5.0
- running the server using ./asynchronous.py
- I don't have any .pyc files in the script directory, possibly because I
"chmod +x" the source file and included the #! at the top
- I added logging and while the main program (setup of Twisted reactor)
logged, the DelayedResource class did not
- I added parts of the example at
https://twistedmatrix.com/documents/current/web/howto/web-in-60/dynamic-con…
- I changed "resource = DelayedResource()" to "resource = ClockPage()"
- I changed the render_GET handler to return the Clock page example output
- etc until I stopped getting a 404
- I restarted the server many many times
I removed and backed out every change until I got a 404, concluding it was
the "import time" line.
Any other ideas?
Hi,
I am learning twisted-web by attempting each of the examples in the
documentation and making any (small) changes necessary for Python 3.
However when I got to the Aysnchronous Responses example I was not able to
get it to work. The example is at:
https://twistedmatrix.com/documents/current/web/howto/web-in-60/asynchronou…
I have adapted it to be a self-contained "server" rather than a .rpy
snippet but the response was always a 404. After much head scratching and
comparison to other examples, I discovered that by adding "import time" to
my "server" it started working.
Why is "import time" required. What is happening without the "import time"?
-Jason
Cell: 604 644 8611
Email: drjasonharrison(a)gmail.com
LinkedIn: http://www.linkedin.com/in/drjasonharrison
Twitter: http://twitter.com/drjasonharrison
#!/usr/bin/env python3
#
https://twistedmatrix.com/documents/current/web/howto/web-in-60/asynchronou…
# Asynchronous responses,
#
# try
# - http://localhost:8888
import sys
from twisted.web.server import Site
from twisted.web.resource import Resource
from twisted.internet import reactor, endpoints
from twisted.web.resource import NoResource
from twisted.web.server import NOT_DONE_YET
from twisted.python import log
import time #???
log.startLogging(sys.stdout)
observer = log.PythonLoggingObserver()
observer.start()
class DelayedResource(Resource):
def _delayedRender(self, request):
log.msg("_delayedRender request {request!r}", request = request)
request.write(b"<html><body>Sorry to keep you
waiting.</body></html>")
request.finish()
def render_GET(self, request):
log.msg("get request {request!r}", request = request)
reactor.callLater(5, self._delayedRender, request)
return NOT_DONE_YET
resource = DelayedResource()
factory = Site(resource)
endpoint = endpoints.TCP4ServerEndpoint(reactor, 8888)
endpoint.listen(factory)
log.msg("running reactor")
reactor.run()