[Twisted-Python] bug or my code? (getPage + Twisted Webserver = Exception)
Hi All, I have a flask application that I am running on a https twisted webserver and forwarding the port to a url. Everything works well. I can goto the url using firefox, accept the certificate and get in. I have a small nose test to test if the url is up. Here is the test: @deferred(timeout = 31) @defer.inlineCallbacks def test_if_forwarding_url_is_ready(self): """ furl: Check if url is accessible """ # First wait for 10 seconds yield task.deferLater(reactor, 10, lambda:None) delay = 2 secureurl = Config.General.url # Every two seconds do a query and find out if you can get # data from the mapped url. If you can, then exit for i in range(15): got_data = False print "Trial :", i try: output = yield getPage(secureurl) got_data = True except Exception as E: print "Exception occurred:", E got_data = False if got_data == True: assert "secure magic" in output return yield task.deferLater(reactor, delay, lambda:None) # All 15 trials to get data from the url failed assert False When the test is run, it fails "most of the time" and outputs: Exception occurred: [('SSL routines', 'SSL23_READ', 'ssl handshake failure')] But if I run the test and right away goto the secureurl manually using firefox or chrome, as soon as I am on the webpage, the test passes! Sometimes I've seen the test will pass after 8 trials, if that is when I can type the url in firefox and press enter! Any ideas what is causing this and how to fix it? Maybe I am doing something stupid and perhaps someone can help me rewrite this code. Thanks, --Ram
On 3 Jul, 09:54 pm, mobilebackup77@gmail.com wrote:
Hi All,
I have a flask application that I am running on a https twisted webserver and forwarding the port to a url. Everything works well. I can goto the url using firefox, accept the certificate and get in.
I have a small nose test to test if the url is up. Here is the test:
@deferred(timeout = 31) @defer.inlineCallbacks def test_if_forwarding_url_is_ready(self): """ furl: Check if url is accessible """ # First wait for 10 seconds yield task.deferLater(reactor, 10, lambda:None) delay = 2 secureurl = Config.General.url # Every two seconds do a query and find out if you can get # data from the mapped url. If you can, then exit for i in range(15): got_data = False print "Trial :", i try: output = yield getPage(secureurl) got_data = True except Exception as E: print "Exception occurred:", E got_data = False if got_data == True: assert "secure magic" in output return yield task.deferLater(reactor, delay, lambda:None) # All 15 trials to get data from the url failed assert False
When the test is run, it fails "most of the time" and outputs:
Exception occurred: [('SSL routines', 'SSL23_READ', 'ssl handshake failure')]
But if I run the test and right away goto the secureurl manually using firefox or chrome, as soon as I am on the webpage, the test passes! Sometimes I've seen the test will pass after 8 trials, if that is when I can type the url in firefox and press enter!
Any ideas what is causing this and how to fix it? Maybe I am doing something stupid and perhaps someone can help me rewrite this code.
nose used to support Twisted with some gross, arguably incorrect code. I don't know if this is still the case, but it's perhaps something to investigate. Apart from that, getPage uses Twisted's regular ("standard"? "only"?) SSL support to do HTTPS. There *shouldn't* be anything particular to getPage that would cause the handshake to fail, so either Twisted's SSL support is broken in general in whatever configuration you're using, or the problem isn't in Twisted. Jean-Paul
Thanks for the quick reply Jean-Paul. I just removed nose from the equation. And now, only firefox or chrome can actually get to the webpages/Twisted webserver, but getPage keeps printing that exception. It seems I've reduced the problem to getPage. I ran a separate thread and used urlopen, and even that can actually get the https webpage hosted on the twisted server. It seems getPage's https support/tests need to be extended perhaps? (I wish I could help find the bug). I know I need to isolate the bug from my project into a simple example that can illustrate the bug. Unfortunately, I'm not sure I've the time to do that right now. On Wed, Jul 4, 2012 at 7:09 AM, <exarkun@twistedmatrix.com> wrote:
On 3 Jul, 09:54 pm, mobilebackup77@gmail.com wrote:
Hi All,
I have a flask application that I am running on a https twisted webserver and forwarding the port to a url. Everything works well. I can goto the url using firefox, accept the certificate and get in.
I have a small nose test to test if the url is up. Here is the test:
@deferred(timeout = 31) @defer.inlineCallbacks def test_if_forwarding_url_is_ready(self): """ furl: Check if url is accessible """ # First wait for 10 seconds yield task.deferLater(reactor, 10, lambda:None) delay = 2 secureurl = Config.General.url # Every two seconds do a query and find out if you can get # data from the mapped url. If you can, then exit for i in range(15): got_data = False print "Trial :", i try: output = yield getPage(secureurl) got_data = True except Exception as E: print "Exception occurred:", E got_data = False if got_data == True: assert "secure magic" in output return yield task.deferLater(reactor, delay, lambda:None) # All 15 trials to get data from the url failed assert False
When the test is run, it fails "most of the time" and outputs:
Exception occurred: [('SSL routines', 'SSL23_READ', 'ssl handshake failure')]
But if I run the test and right away goto the secureurl manually using firefox or chrome, as soon as I am on the webpage, the test passes! Sometimes I've seen the test will pass after 8 trials, if that is when I can type the url in firefox and press enter!
Any ideas what is causing this and how to fix it? Maybe I am doing something stupid and perhaps someone can help me rewrite this code.
nose used to support Twisted with some gross, arguably incorrect code. I don't know if this is still the case, but it's perhaps something to investigate.
Apart from that, getPage uses Twisted's regular ("standard"? "only"?) SSL support to do HTTPS. There *shouldn't* be anything particular to getPage that would cause the handshake to fail, so either Twisted's SSL support is broken in general in whatever configuration you're using, or the problem isn't in Twisted.
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Le Jul 4, 2012 à 1:19 PM, Me Myself <mobilebackup77@gmail.com> a écrit :
I know I need to isolate the bug from my project into a simple example that can illustrate the bug. Unfortunately, I'm not sure I've the time to do that right now.
It's OK, any time you have the time to do it, we will be ready to handle the bug :). Please just put it on a to-do list or something so that you don't forget. -glyph
The server I am using uses TLS SNI. Does getPage support this, or has any plans in the near future to support this? Do any other python packages (request/urllib3/...) support TLS SNI yet? Also, is there a way to do HTTP CONNECT using twisted before I do the SSL handshake. I think that should solve my problem. Thanks, --Ram On Wed, Jul 4, 2012 at 5:43 PM, Glyph <glyph@twistedmatrix.com> wrote:
Le Jul 4, 2012 à 1:19 PM, Me Myself <mobilebackup77@gmail.com> a écrit :
I know I need to isolate the bug from my project into a simple example that can illustrate the bug. Unfortunately, I'm not sure I've the time to do that right now.
It's OK, any time you have the time to do it, we will be ready to handle the bug :). Please just put it on a to-do list or something so that you don't forget.
-glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 04:25 pm, mobilebackup77@gmail.com wrote:
The server I am using uses TLS SNI. Does getPage support this, or has any plans in the near future to support this? Do any other python packages (request/urllib3/...) support TLS SNI yet?
Sure. Just pass in an appropriately configured SSL context factory. See the pyOpenSSL documentation for details on how to configure SNI.
Also, is there a way to do HTTP CONNECT using twisted before I do the SSL handshake. I think that should solve my problem.
Twisted doesn't yet have any support for HTTP CONNECT. Jean-Paul
Can you please show me an example whenever you get a chance? Thanks, --Ram. On Tue, Jul 17, 2012 at 3:50 PM, <exarkun@twistedmatrix.com> wrote:
On 04:25 pm, mobilebackup77@gmail.com wrote:
The server I am using uses TLS SNI. Does getPage support this, or has any plans in the near future to support this? Do any other python packages (request/urllib3/...) support TLS SNI yet?
Sure. Just pass in an appropriately configured SSL context factory.
See the pyOpenSSL documentation for details on how to configure SNI.
Also, is there a way to do HTTP CONNECT using twisted before I do the SSL handshake. I think that should solve my problem.
Twisted doesn't yet have any support for HTTP CONNECT.
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Ram, Have you checked out the ssl[1] howto? The section on "Client with certificates" gives an example on how to make your own SSL Context Class. 1) http://twistedmatrix.com/documents/current/core/howto/ssl.html Regards, Justin Venus On Tue, Jul 17, 2012 at 3:01 PM, Me Myself <mobilebackup77@gmail.com> wrote:
Can you please show me an example whenever you get a chance?
Thanks, --Ram.
On Tue, Jul 17, 2012 at 3:50 PM, <exarkun@twistedmatrix.com> wrote:
On 04:25 pm, mobilebackup77@gmail.com wrote:
The server I am using uses TLS SNI. Does getPage support this, or has any plans in the near future to support this? Do any other python packages (request/urllib3/...) support TLS SNI yet?
Sure. Just pass in an appropriately configured SSL context factory.
See the pyOpenSSL documentation for details on how to configure SNI.
Also, is there a way to do HTTP CONNECT using twisted before I do the SSL handshake. I think that should solve my problem.
Twisted doesn't yet have any support for HTTP CONNECT.
Jean-Paul
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
-- Justin Venus justin.venus@gmail.com
On Jul 17, 2012, at 12:50 PM, exarkun@twistedmatrix.com wrote:
On 04:25 pm, mobilebackup77@gmail.com wrote:
The server I am using uses TLS SNI. Does getPage support this, or has any plans in the near future to support this? Do any other python packages (request/urllib3/...) support TLS SNI yet?
Sure. Just pass in an appropriately configured SSL context factory.
See the pyOpenSSL documentation for details on how to configure SNI.
Also, if you're interested in making some higher-level wrappers for Twisted, it would be great to improve twisted.web.client.Agent to support SNI for HTTPS URLs. This is the server-side ticket but it has links to a few client-side things: <http://twistedmatrix.com/trac/ticket/4887>. -g
participants (4)
-
exarkun＠twistedmatrix.com -
Glyph -
Justin Venus -
Me Myself