[Twisted-Python] Twisted 16.3.0 Prerelease 2 Announcement
Hi everyone, Here's another prerelease in the 16.3 series -- fixing a 16.2 regression in HTTP timeouts not working. For more information, check the NEWS file (link provided below). As usual, it's available for download -- go here (https://twistedmatrix.com/Releases/pre/16.3.0pre2/) to get the prerelease tarballs and the full NEWS file. If you want to install it right away, run: pip install https://twistedmatrix.com/Releases/pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2 A reminder that if you would like to try out the newly-landed HTTP/2 support, run: pip install -U https://twistedmatrix.com/Releases/pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2#egg=Twisted[http2,tls] This will download the new HTTP/2 dependencies and the TLS requirements as well. Please let me know if you have any issues, as well as if you don't! If everything works well, that's a good thing for me to know :) Twisted Regards, Amber Brown (HawkOwl)
HTTP2 support sounds really exciting.
Please let me know if you have any issues, as well as if you don't! If everything works well, that's a good thing for me to know :)
I played around with this today and found out that the command you recommend:
pip install -U https://twistedmatrix.com/Releases /pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2#egg=Twisted[http2,tls]
pip install -U https://twistedmatrix.com/Releases/pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2#egg=Twisted[http2,tls] Downloading/unpacking Twisted[http2,tls] from https://twistedmatrix.com/Releases/pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2 Downloading Twisted-16.3.0rc2.tar.bz2 (2.9MB): 2.9MB downloaded Running setup.py (path:/home/pawel/.virtualenvs/foo3/build/Twisted/setup.py) egg_info for
does NOT install dependencies when ran on Python 3, I had to manually install h2 to HTTP2 support to work. It works ok on Python 2. My installation logs on Python 3.4 look like this: package Twisted warning: no previously-included files matching '*.misc' found under directory 'twisted' warning: no previously-included files matching '*.bugfix' found under directory 'twisted' warning: no previously-included files matching '*.doc' found under directory 'twisted' warning: no previously-included files matching '*.feature' found under directory 'twisted' warning: no previously-included files matching '*.removal' found under directory 'twisted' warning: no previously-included files matching 'NEWS' found under directory 'twisted' warning: no previously-included files matching 'README' found under directory 'twisted' warning: no previously-included files matching 'topfiles' found under directory 'twisted' warning: no previously-included files found matching 'twisted/topfiles/CREDITS' warning: no previously-included files found matching 'twisted/topfiles/ChangeLog.Old' warning: no previously-included files found matching 'bin/_preamble.py' warning: no previously-included files found matching 'admin' warning: no previously-included files found matching 'bin/admin' warning: no previously-included files matching '*' found under directory 'admin' warning: no previously-included files matching '*' found under directory 'bin/admin' warning: no previously-included files found matching 'docs/historic/2003' warning: no previously-included files matching '*' found under directory 'docs/historic/2003' Installing extra requirements: 'http2,tls' Requirement already up-to-date: zope.interface>=4.0.2 in /home/pawel/.virtualenvs/foo3/lib/python3.4/site-packages (from Twisted[http2,tls]) Requirement already up-to-date: setuptools in /home/pawel/.virtualenvs/foo3/lib/python3.4/site-packages (from zope.interface>=4.0.2->Twisted[http2,tls]) Installing collected packages: Twisted Running setup.py install for Twisted warning: no previously-included files matching '*.misc' found under directory 'twisted' warning: no previously-included files matching '*.bugfix' found under directory 'twisted' warning: no previously-included files matching '*.doc' found under directory 'twisted' warning: no previously-included files matching '*.feature' found under directory 'twisted' warning: no previously-included files matching '*.removal' found under directory 'twisted' warning: no previously-included files matching 'NEWS' found under directory 'twisted' warning: no previously-included files matching 'README' found under directory 'twisted' warning: no previously-included files matching 'topfiles' found under directory 'twisted' warning: no previously-included files found matching 'twisted/topfiles/CREDITS' warning: no previously-included files found matching 'twisted/topfiles/ChangeLog.Old' warning: no previously-included files found matching 'bin/_preamble.py' warning: no previously-included files found matching 'admin' warning: no previously-included files found matching 'bin/admin' warning: no previously-included files matching '*' found under directory 'admin' warning: no previously-included files matching '*' found under directory 'bin/admin' warning: no previously-included files found matching 'docs/historic/2003' warning: no previously-included files matching '*' found under directory 'docs/historic/2003' warning: PickyBuildPy: byte-compiling is disabled, skipping. changing mode of build/scripts-3.4/trial from 664 to 775 changing mode of build/scripts-3.4/twistd from 664 to 775 warning: install_lib: byte-compiling is disabled, skipping. changing mode of /home/pawel/.virtualenvs/foo3/bin/twistd to 775 changing mode of /home/pawel/.virtualenvs/foo3/bin/trial to 775 Could not find .egg-info directory in install record for Twisted[http2,tls] from https://twistedmatrix.com/Releases/pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2 Successfully installed Twisted Cleaning up... 2016-06-28 14:44 GMT+02:00 Amber "Hawkie" Brown <hawkowl@atleastfornow.net>:
Hi everyone,
Here's another prerelease in the 16.3 series -- fixing a 16.2 regression in HTTP timeouts not working.
For more information, check the NEWS file (link provided below).
As usual, it's available for download -- go here ( https://twistedmatrix.com/Releases/pre/16.3.0pre2/) to get the prerelease tarballs and the full NEWS file. If you want to install it right away, run:
pip install https://twistedmatrix.com/Releases/pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2
A reminder that if you would like to try out the newly-landed HTTP/2 support, run:
pip install -U https://twistedmatrix.com/Releases/pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2#egg=Twisted[http2,tls]
This will download the new HTTP/2 dependencies and the TLS requirements as well.
Please let me know if you have any issues, as well as if you don't! If everything works well, that's a good thing for me to know :)
Twisted Regards, Amber Brown (HawkOwl)
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 3 July 2016 at 11:15, Paweł Miech <pawelmhm@gmail.com> wrote:
HTTP2 support sounds really exciting.
Please let me know if you have any issues, as well as if you don't! If everything works well, that's a good thing for me to know :)
I played around with this today and found out that the command you recommend:
pip install -U https://twistedmatrix.com/Releases /pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2#egg=Twisted[http2,tls]
does NOT install dependencies when ran on Python 3, I had to manually install h2 to HTTP2 support to work. It works ok on Python 2. My installation logs on Python 3.4 look like this:
AFAIK this is a known issue : see https://github.com/twisted/twisted/blob/trunk/docs/installation/howto/option... -- Adi Roiban
AFAIK this is a known issue :
Ah thanks, that's ok. One other thing I noticed a propos HTTP 2 is that it seems that reading relatively large file results in error: "priority.priority.MissingStreamError: 'Stream 1 not in tree'". I created simple gist to recreate this issue see here: https://gist.github.com/pawelmhm/3aa7e4f3a0e322364dcb75e3f0a32da4 Resource is launched with Python 3.4, to test it on the client I used curl with http2 support. Data file <https://drive.google.com/file/d/0B6myg3n6dqcVcXpPdkJCNUJLOTA/view?pref=2&pli=1> is not huge (570kb) but the error is raised when I send it as a whole, when I only send some smaller chunk (e.g. first 100 items from file) it seems to work ok. It seems that Python-hyper is all right with this file, perhaps curl --http2 does something weird. 2016-07-03 12:50 GMT+02:00 Adi Roiban <adi@roiban.ro>:
On 3 July 2016 at 11:15, Paweł Miech <pawelmhm@gmail.com> wrote:
HTTP2 support sounds really exciting.
Please let me know if you have any issues, as well as if you don't! If everything works well, that's a good thing for me to know :)
I played around with this today and found out that the command you recommend:
pip install -U https://twistedmatrix.com/Releases /pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2#egg=Twisted[http2,tls]
does NOT install dependencies when ran on Python 3, I had to manually install h2 to HTTP2 support to work. It works ok on Python 2. My installation logs on Python 3.4 look like this:
AFAIK this is a known issue :
see https://github.com/twisted/twisted/blob/trunk/docs/installation/howto/option...
-- Adi Roiban
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Good catch Paweł. I have opened this issue as Twisted issue #8558: https://twistedmatrix.com/trac/ticket/8558 <https://twistedmatrix.com/trac/ticket/8558>. I believe I know what the fix is and it’s fairly simple, so I’ll try to address this quickly and see if we can ship the fix in either the next pre-release or in the actual 16.3 release. Cory
On 3 Jul 2016, at 13:47, Paweł Miech <pawelmhm@gmail.com <mailto:pawelmhm@gmail.com>> wrote:
AFAIK this is a known issue :
Ah thanks, that's ok.
One other thing I noticed a propos HTTP 2 is that it seems that reading relatively large file results in error: "priority.priority.MissingStreamError: 'Stream 1 not in tree'". I created simple gist to recreate this issue see here: https://gist.github.com/pawelmhm/3aa7e4f3a0e322364dcb75e3f0a32da4 <https://gist.github.com/pawelmhm/3aa7e4f3a0e322364dcb75e3f0a32da4> Resource is launched with Python 3.4, to test it on the client I used curl with http2 support. Data file <https://drive.google.com/file/d/0B6myg3n6dqcVcXpPdkJCNUJLOTA/view?pref=2&pli=1> is not huge (570kb) but the error is raised when I send it as a whole, when I only send some smaller chunk (e.g. first 100 items from file) it seems to work ok. It seems that Python-hyper is all right with this file, perhaps curl --http2 does something weird.
2016-07-03 12:50 GMT+02:00 Adi Roiban <adi@roiban.ro <mailto:adi@roiban.ro>>:
On 3 July 2016 at 11:15, Paweł Miech <pawelmhm@gmail.com <mailto:pawelmhm@gmail.com>> wrote: HTTP2 support sounds really exciting.
Please let me know if you have any issues, as well as if you don't! If everything works well, that's a good thing for me to know :)
I played around with this today and found out that the command you recommend:
pip install -U https://twistedmatrix.com/Releases/pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2#egg=Twisted[http2,tls] <https://twistedmatrix.com/Releases/pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2#egg=Twisted[http2,tls]> does NOT install dependencies when ran on Python 3, I had to manually install h2 to HTTP2 support to work. It works ok on Python 2. My installation logs on Python 3.4 look like this:
AFAIK this is a known issue :
see https://github.com/twisted/twisted/blob/trunk/docs/installation/howto/option... <https://github.com/twisted/twisted/blob/trunk/docs/installation/howto/option...>
-- Adi Roiban
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python <http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python>
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com <mailto:Twisted-Python@twistedmatrix.com> http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Sun, Jul 3, 2016 at 3:15 AM, Paweł Miech <pawelmhm@gmail.com> wrote:
HTTP2 support sounds really exciting.
Please let me know if you have any issues, as well as if you don't! If everything works well, that's a good thing for me to know :)
I played around with this today and found out that the command you recommend:
pip install -U https://twistedmatrix.com/Releases /pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2#egg=Twisted[http2,tls]
does NOT install dependencies when ran on Python 3, I had to manually install h2 to HTTP2 support to work. It works ok on Python 2. My installation logs on Python 3.4 look like this:
I think I've fixed that in trunk with this: https://github.com/twisted/twisted/pull/287 -- Craig
Thanks for fixing this. Did anyone actually manage to make HTTP2 in Twisted work with Google-Chrome? I tried to do this today, and it seems this is surprisingly difficult. It turns out that Chrome requires ALPN and it dropped support for NPN. ALPN is only supported with OpenSSL 1.0.2 or above, which by default is not available in most systems. This is discussed here <https://www.nginx.com/blog/supporting-http2-google-chrome-users/>. I tried setting up docker image with Ubuntu 16.04 that has required version of OpenSSL, but it seems that Chrome still doesn't like it. It returns ERR_SPDY_INADEQUATE_TRANSPORT_SECURITY error and fails to load page. Looking up this error I found out this might be happening because some TSL ciphers are blacklisted in HTTP2, there is nice answer about this here <https://serverfault.com/questions/712808/chrome-reports-err-spdy-inadequate-...> it links to this part of HTTP2 spec https://http2.github.io/http2-spec/#rfc.section.9.2.2 My question is: should user deal with this kind of stuff themselves? If some ciphers are blacklisted in HTTP2 shouldn't this be handled somewhere in Twisted? E.g. perhaps there should be some Http2SSLContextFactory? If you'd like to reproduce this I did some sample repo here: https://github.com/pawelmhm/sf-books-http2 it contains dockerfile that builds from Ubuntu 16.04 and runs simple Twisted HTTP 2 resource. 2016-07-04 13:48 GMT+02:00 Craig Rodrigues <rodrigc@crodrigues.org>:
On Sun, Jul 3, 2016 at 3:15 AM, Paweł Miech <pawelmhm@gmail.com> wrote:
HTTP2 support sounds really exciting.
Please let me know if you have any issues, as well as if you don't! If everything works well, that's a good thing for me to know :)
I played around with this today and found out that the command you recommend:
pip install -U https://twistedmatrix.com/Releases /pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2#egg=Twisted[http2,tls]
does NOT install dependencies when ran on Python 3, I had to manually install h2 to HTTP2 support to work. It works ok on Python 2. My installation logs on Python 3.4 look like this:
I think I've fixed that in trunk with this:
https://github.com/twisted/twisted/pull/287
-- Craig
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Works for me with txacme and a lets: cert IIRC, when I was trying to use a self signed cert on my local network I got the ERR_SPDY_INADEQUATE_TRANSPORT_SECURITY error. DJM On 9 July 2016 at 18:30, Paweł Miech <pawelmhm@gmail.com> wrote:
Thanks for fixing this.
Did anyone actually manage to make HTTP2 in Twisted work with Google-Chrome? I tried to do this today, and it seems this is surprisingly difficult. It turns out that Chrome requires ALPN and it dropped support for NPN. ALPN is only supported with OpenSSL 1.0.2 or above, which by default is not available in most systems. This is discussed here <https://www.nginx.com/blog/supporting-http2-google-chrome-users/>. I tried setting up docker image with Ubuntu 16.04 that has required version of OpenSSL, but it seems that Chrome still doesn't like it. It returns ERR_SPDY_INADEQUATE_TRANSPORT_SECURITY error and fails to load page. Looking up this error I found out this might be happening because some TSL ciphers are blacklisted in HTTP2, there is nice answer about this here <https://serverfault.com/questions/712808/chrome-reports-err-spdy-inadequate-...> it links to this part of HTTP2 spec https://http2.github.io/http2-spec/#rfc.section.9.2.2
My question is: should user deal with this kind of stuff themselves? If some ciphers are blacklisted in HTTP2 shouldn't this be handled somewhere in Twisted? E.g. perhaps there should be some Http2SSLContextFactory? If you'd like to reproduce this I did some sample repo here: https://github.com/pawelmhm/sf-books-http2 it contains dockerfile that builds from Ubuntu 16.04 and runs simple Twisted HTTP 2 resource.
2016-07-04 13:48 GMT+02:00 Craig Rodrigues <rodrigc@crodrigues.org>:
On Sun, Jul 3, 2016 at 3:15 AM, Paweł Miech <pawelmhm@gmail.com> wrote:
HTTP2 support sounds really exciting.
Please let me know if you have any issues, as well as if you don't! If everything works well, that's a good thing for me to know :)
I played around with this today and found out that the command you recommend:
pip install -U https://twistedmatrix.com/Releases /pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2#egg=Twisted[http2,tls]
does NOT install dependencies when ran on Python 3, I had to manually install h2 to HTTP2 support to work. It works ok on Python 2. My installation logs on Python 3.4 look like this:
I think I've fixed that in trunk with this:
https://github.com/twisted/twisted/pull/287
-- Craig
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
Hmm, I have it working fine (Python 2.7/3.5, w/ Cryptography wheels on OS X)... The default ciphers in Twisted are: ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+3DES:!aNULL:!MD5:!DSS So I am not sure why it's not picking up "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256" first... - Amber
On 10 Jul 2016, at 01:30, Paweł Miech <pawelmhm@gmail.com> wrote:
Thanks for fixing this.
Did anyone actually manage to make HTTP2 in Twisted work with Google-Chrome? I tried to do this today, and it seems this is surprisingly difficult. It turns out that Chrome requires ALPN and it dropped support for NPN. ALPN is only supported with OpenSSL 1.0.2 or above, which by default is not available in most systems. This is discussed here. I tried setting up docker image with Ubuntu 16.04 that has required version of OpenSSL, but it seems that Chrome still doesn't like it. It returns ERR_SPDY_INADEQUATE_TRANSPORT_SECURITY error and fails to load page. Looking up this error I found out this might be happening because some TSL ciphers are blacklisted in HTTP2, there is nice answer about this here it links to this part of HTTP2 spec https://http2.github.io/http2-spec/#rfc.section.9.2.2
My question is: should user deal with this kind of stuff themselves? If some ciphers are blacklisted in HTTP2 shouldn't this be handled somewhere in Twisted? E.g. perhaps there should be some Http2SSLContextFactory? If you'd like to reproduce this I did some sample repo here: https://github.com/pawelmhm/sf-books-http2 it contains dockerfile that builds from Ubuntu 16.04 and runs simple Twisted HTTP 2 resource.
2016-07-04 13:48 GMT+02:00 Craig Rodrigues <rodrigc@crodrigues.org>: On Sun, Jul 3, 2016 at 3:15 AM, Paweł Miech <pawelmhm@gmail.com> wrote: HTTP2 support sounds really exciting.
Please let me know if you have any issues, as well as if you don't! If everything works well, that's a good thing for me to know :)
I played around with this today and found out that the command you recommend:
pip install -U https://twistedmatrix.com/Releases/pre/16.3.0pre2/Twisted-16.3.0rc2.tar.bz2#egg=Twisted[http2,tls]
does NOT install dependencies when ran on Python 3, I had to manually install h2 to HTTP2 support to work. It works ok on Python 2. My installation logs on Python 3.4 look like this:
I think I've fixed that in trunk with this:
https://github.com/twisted/twisted/pull/287
-- Craig
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Jul 9, 2016, at 10:30 AM, Paweł Miech <pawelmhm@gmail.com> wrote:
My question is: should user deal with this kind of stuff themselves? If some ciphers are blacklisted in HTTP2 shouldn't this be handled somewhere in Twisted?
As others have already said, this should work out of the box, and I'm not sure why it isn't for you, especially that you've gone to the extra trouble of building a Docker image and retrieving recent enough versions of every relevant layer of the stack. However, to answer this question generally: this should absolutely be handled by Twisted. In fact, even if we're doing the right thing already except in your one configuration, we should go a step beyond and provide tooling and logging to clearly explain to system operators why they won't get HTTP/2 if their dependencies are out of date. -glyph
On 11 Jul 2016, at 01:45, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Jul 9, 2016, at 10:30 AM, Paweł Miech <pawelmhm@gmail.com <mailto:pawelmhm@gmail.com>> wrote:
My question is: should user deal with this kind of stuff themselves? If some ciphers are blacklisted in HTTP2 shouldn't this be handled somewhere in Twisted?
As others have already said, this should work out of the box, and I'm not sure why it isn't for you, especially that you've gone to the extra trouble of building a Docker image and retrieving recent enough versions of every relevant layer of the stack.
However, to answer this question generally: this should absolutely be handled by Twisted. In fact, even if we're doing the right thing already except in your one configuration, we should go a step beyond and provide tooling and logging to clearly explain to system operators why they won't get HTTP/2 if their dependencies are out of date.
This turns out to be trickier than you’d expect. PyOpenSSL does not expose any of the APIs for us to programmatically detect what ciphers are available to the OpenSSL we have installed. Cryptography exposes only one: SSL_get_ciphers. This is not really the one we want, because it lists all *possible* ciphers, rather than the ones that are actually enabled for a given connection. This makes it very difficult for us to conclude that we’d want to use HTTP/2 but we cannot because of a lack of cipher support. Now, Twisted *could* add code to introspect the HTTP/2 TLS configuration and optionally terminate the connection in the same manner that Chrome does. Currently I’ve not done that because it’s not been hugely needed, but we could do that. The reality is, though, that Twisted can’t unconditionally not use those ciphers because it needs to support HTTP/1.1 as well as HTTP/2, and HTTP/1.1 does not have those same restrictions. What would be looking for here? Out of the box, Twisted should do the very best it can, but right now it seems like the only thing we could do is detect when HTTP/2 is literally impossible to support (e.g. when there is no TLS 1.2 support). With that said, those versions *completely* overlap with the versions where OpenSSL doesn’t support ALPN. Regardless, Twisted’s default cipher ordering is appropriate for HTTP/2 (it prefers ECDHE AES GCM, which is what is required). So I’m not sure what more we could do. Cory
On Jul 11, 2016, at 3:35 AM, Cory Benfield <cory@lukasa.co.uk <mailto:cory@lukasa.co.uk>> wrote:
On 11 Jul 2016, at 01:45, Glyph Lefkowitz <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
On Jul 9, 2016, at 10:30 AM, Paweł Miech <pawelmhm@gmail.com <mailto:pawelmhm@gmail.com>> wrote:
My question is: should user deal with this kind of stuff themselves? If some ciphers are blacklisted in HTTP2 shouldn't this be handled somewhere in Twisted?
As others have already said, this should work out of the box, and I'm not sure why it isn't for you, especially that you've gone to the extra trouble of building a Docker image and retrieving recent enough versions of every relevant layer of the stack.
However, to answer this question generally: this should absolutely be handled by Twisted. In fact, even if we're doing the right thing already except in your one configuration, we should go a step beyond and provide tooling and logging to clearly explain to system operators why they won't get HTTP/2 if their dependencies are out of date.
This turns out to be trickier than you’d expect.
PyOpenSSL does not expose any of the APIs for us to programmatically detect what ciphers are available to the OpenSSL we have installed. Cryptography exposes only one: SSL_get_ciphers. This is not really the one we want, because it lists all *possible* ciphers, rather than the ones that are actually enabled for a given connection. This makes it very difficult for us to conclude that we’d want to use HTTP/2 but we cannot because of a lack of cipher support.
So pyOpenSSL/Cryptography doesn't have SSL_get_current_cipher anywhere?
Now, Twisted *could* add code to introspect the HTTP/2 TLS configuration and optionally terminate the connection in the same manner that Chrome does. Currently I’ve not done that because it’s not been hugely needed, but we could do that. The reality is, though, that Twisted can’t unconditionally not use those ciphers because it needs to support HTTP/1.1 as well as HTTP/2, and HTTP/1.1 does not have those same restrictions.
The main interest I think we have is to placate Chrome, to ensure it can speak HTTP/2 if it's possible, and to explain why it's not possible, if it's not.
What would be looking for here? Out of the box, Twisted should do the very best it can, but right now it seems like the only thing we could do is detect when HTTP/2 is literally impossible to support (e.g. when there is no TLS 1.2 support). With that said, those versions *completely* overlap with the versions where OpenSSL doesn’t support ALPN.
In the same way that we complain about service_identity perhaps we should complain about OpenSSL?
Regardless, Twisted’s default cipher ordering is appropriate for HTTP/2 (it prefers ECDHE AES GCM, which is what is required). So I’m not sure what more we could do.
Yeah, I'm curious why the OP was having this problem. -glyph
Thanks for input everyone! @Cory
right now it seems like the only thing we could do is detect when HTTP/2 is literally impossible to support (e.g. when there is no TLS 1.2 support)
This seems to suggest that Ubuntu 16.04 (the system I'm testing) does not support ciphers required by HTTP2. But nginx article about HTTP2 lists ubuntu as only linux like system that is able to support HTTP2 over ALPN which is required by Chrome: https://www.nginx.com/blog/supporting-http2-google-chrome-users/ I decided to verify tnginx statements and I tried to set up nginx with HTTP2 on ubuntu 16.04. It turns out this is possible and it works ok. I just followed this article here: https://www.digitalocean.com/community/tutorials/how-to-set-up-nginx-with-ht... This means that in principle Ubuntu 16.04 should be able to support HTTP2 and it has required TLS ciphers. So the problem here is not about lack of OS support. Looking into this nginx article they recommend two things that are part of manual setup which (maybe?) are required? 1) They say ciphers should be set to ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5; This long string does not mean much to me, but reading email from Amber again I see it differs slightly from what she says Twisted uses. But one thing I'm wondering about is how do you guys know which ciphers are set in Twisted? Looking into source code of DefaultOpenSSLContextFactory I see context is created here: https://github.com/twisted/twisted/blob/3455a902fb15e732ee43b59f4d82a66b1053... I dont see any point where there is a call that sets ciphers. Maybe this is done somewhere else? I tried grepping source for string mentioned by Amber but cant find it. 2) they ask user to generate DHE key and provide this to nginx configuration. When I compare my nginx with Twisted using openssl I see that ciphers in response differ. For example this is what my nginx cipher is:
openssl s_client -connect localhost:443 ... ... ... New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
And here is cipher for Twisted
openssl s_client -connect localhost:8080
New, TLSv1/SSLv3, Cipher is AES256-GCM-SHA384 When I check HTTP2 cipher black list I see AES256-GCM-SHA384 is there on this list see here https://http2.github.io/http2-spec/#BadCipherSuites 2016-07-11 21:22 GMT+02:00 Glyph Lefkowitz <glyph@twistedmatrix.com>:
On Jul 11, 2016, at 3:35 AM, Cory Benfield <cory@lukasa.co.uk> wrote:
On 11 Jul 2016, at 01:45, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
On Jul 9, 2016, at 10:30 AM, Paweł Miech <pawelmhm@gmail.com> wrote:
My question is: should user deal with this kind of stuff themselves? If some ciphers are blacklisted in HTTP2 shouldn't this be handled somewhere in Twisted?
As others have already said, this *should* work out of the box, and I'm not sure why it isn't for you, especially that you've gone to the extra trouble of building a Docker image and retrieving recent enough versions of every relevant layer of the stack.
However, to answer this question generally: this should *absolutely* be handled by Twisted. In fact, even if we're doing the right thing already except in your one configuration, we should go a step beyond and provide tooling and logging to clearly explain to system operators why they won't get HTTP/2 if their dependencies are out of date.
This turns out to be trickier than you’d expect.
PyOpenSSL does not expose any of the APIs for us to programmatically detect what ciphers are available to the OpenSSL we have installed. Cryptography exposes only one: SSL_get_ciphers. This is not really the one we want, because it lists all *possible* ciphers, rather than the ones that are actually enabled for a given connection. This makes it very difficult for us to conclude that we’d want to use HTTP/2 but we cannot because of a lack of cipher support.
So pyOpenSSL/Cryptography doesn't have SSL_get_current_cipher anywhere?
Now, Twisted *could* add code to introspect the HTTP/2 TLS configuration and optionally terminate the connection in the same manner that Chrome does. Currently I’ve not done that because it’s not been hugely needed, but we could do that. The reality is, though, that Twisted can’t unconditionally not use those ciphers because it needs to support HTTP/1.1 as well as HTTP/2, and HTTP/1.1 does not have those same restrictions.
The main interest I think we have is to placate Chrome, to ensure it can speak HTTP/2 if it's possible, and to explain why it's not possible, if it's not.
What would be looking for here? Out of the box, Twisted should do the very best it can, but right now it seems like the only thing we could do is detect when HTTP/2 is literally impossible to support (e.g. when there is no TLS 1.2 support). With that said, those versions *completely* overlap with the versions where OpenSSL doesn’t support ALPN.
In the same way that we complain about service_identity perhaps we should complain about OpenSSL?
Regardless, Twisted’s default cipher ordering is appropriate for HTTP/2 (it prefers ECDHE AES GCM, which is what is required). So I’m not sure what more we could do.
Yeah, I'm curious why the OP was having this problem.
-glyph
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Mon, Jul 11, 2016 at 2:04 PM, Paweł Miech <pawelmhm@gmail.com> wrote:
1) They say ciphers should be set to ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
This long string does not mean much to me, but reading email from Amber again I see it differs slightly from what she says Twisted uses. But one thing I'm wondering about is how do you guys know which ciphers are set in Twisted? Looking into source code of DefaultOpenSSLContextFactory I see context is created here: https://github.com/twisted/twisted/blob/3455a902fb15e732ee43b59f4d82a66b1053... I dont see any point where there is a call that sets ciphers. Maybe this is done somewhere else? I tried grepping source for string mentioned by Amber but cant find it.
In an earlier e-mail you mentioned that you were using Python 3. Is that still true? In the Windows Python 3 build which was recently enabled, I saw these warnings: c:\buildslave\win2012r2-64-py3_5\Twisted\twisted\internet\_sslverify.py:1799: DeprecationWarning: str for cipher_list is no longer accepted, use bytes c:\buildslave\win2012r2-64-py3_5\Twisted\twisted\internet\_sslverify.py:1656: DeprecationWarning: str for buf is no longer accepted, use bytes c:\buildslave\win2012r2-64-py3_5\Twisted\twisted\internet\_sslverify.py:1660: DeprecationWarning: str for cipher_list is no longer accepted, use bytes I am not sure if this is related to your problem, but it struck me that you mentioned a problem with ciphers, and I saw this warning just now. -- Craig
In an earlier e-mail you mentioned that you were using Python 3. Is that still true?
I can reproduce this in Python 2.7.11 and Python 3.5.2. In both of them Chrome responds with ERR_SPDY_INADEQUATE_TRANSPORT_SECURITY. When I test with curl with verbose flag I see that it also shows information about ciphers used: Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH ... SSL connection using TLSv1.2 / AES256-GCM-SHA384 I see ciphers are set at this point here: https://github.com/twisted/twisted/blob/556f0f24df2eba2f38ec7f0fa422c4aa7df0... and Twisted cipher is described here: https://github.com/twisted/twisted/blob/556f0f24df2eba2f38ec7f0fa422c4aa7df0... so probably this is the area to look for in case there is something going awry in setting ciphers. One thing to note is that I use DefaultOpenSSLContextFactory and do something like this: context_factory = DefaultOpenSSLContextFactory("key.pem", "cert.pem") reactor.listenSSL(8080, site, context_factory) Twisted docs for SSL https://twistedmatrix.com/documents/current/core/howto/ssl.html suggest to try something like this: certData = getModule(__name__).filePath.sibling('server.pem').getContent() certificate = ssl.PrivateCertificate.loadPEM(certData) factory = protocol.Factory.forProtocol(echoserv.Echo) reactor.listenSSL(8000, factory, certificate.options()) but those code samples from docs appeared broken. I was not able to run them I was planning to review those docs later, find out what is wrong and create PR for that. Is it possible that using DefaultOpenSSLContextFactory instead of certificate.options() affects something here? I can see my Twisted-SSL code works ok in Chrome with HTTP 1.1 ( I can see green "secure" icon in url bar and confirm that requests flies all right with ssl in dev tools) only fails with HTTP2. This seems to suggest that using DefaultSSLContextFactory is ok (even if it's not documented officially), but maybe execution path is different for contextFactory and certificate.options()? 2016-07-12 1:47 GMT+02:00 Glyph Lefkowitz <glyph@twistedmatrix.com>:
On Jul 11, 2016, at 4:42 PM, Craig Rodrigues <rodrigc@crodrigues.org> wrote:
In an earlier e-mail you mentioned that you were using Python 3. Is that still true?
Seconded - it would be very interesting to know if switching to python 2 fixes your issue. :)
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 11 Jul 2016, at 22:04, Paweł Miech <pawelmhm@gmail.com> wrote:
This seems to suggest that Ubuntu 16.04 (the system I'm testing) does not support ciphers required by HTTP2. But nginx article about HTTP2 lists ubuntu as only linux like system that is able to support HTTP2 over ALPN which is required by Chrome: https://www.nginx.com/blog/supporting-http2-google-chrome-users/ <https://www.nginx.com/blog/supporting-http2-google-chrome-users/> Sorry. To be clear, I was not responding to your specific needs but discussing Glyph’s wider point about alerting when bad configuration is present.
I decided to verify tnginx statements and I tried to set up nginx with HTTP2 on ubuntu 16.04. It turns out this is possible and it works ok. I just followed this article here: https://www.digitalocean.com/community/tutorials/how-to-set-up-nginx-with-ht... <https://www.digitalocean.com/community/tutorials/how-to-set-up-nginx-with-ht...> This means that in principle Ubuntu 16.04 should be able to support HTTP2 and it has required TLS ciphers.
So the problem here is not about lack of OS support.
Looking into this nginx article they recommend two things that are part of manual setup which (maybe?) are required?
1) They say ciphers should be set to ssl_ciphers EECDH+CHACHA20:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
This long string does not mean much to me, but reading email from Amber again I see it differs slightly from what she says Twisted uses. But one thing I'm wondering about is how do you guys know which ciphers are set in Twisted? Looking into source code of DefaultOpenSSLContextFactory I see context is created here: https://github.com/twisted/twisted/blob/3455a902fb15e732ee43b59f4d82a66b1053... <https://github.com/twisted/twisted/blob/3455a902fb15e732ee43b59f4d82a66b1053...> I dont see any point where there is a call that sets ciphers. Maybe this is done somewhere else? I tried grepping source for string mentioned by Amber but cant find it.
Ok, this is your problem. DefaultOpenSSLContextFactory should have been deprecated a long time ago. It’s insecure, and in particular does not set a cipher string, so it uses DEFAULT. That will have all kinds of messed up priorities. For that reason, you should adjust your code to use OpenSSLCertificateOptions or, even better, use the TLS endpoint directly. The TL;DR is: yes, it seems that DefaultOpenSSLContextFactory produces a context that is genuinely unacceptable for HTTP/2. Cory
DefaultOpenSSLContextFactory should have been deprecated a long time ago. It’s insecure, and in particular does not set a cipher string, so it uses DEFAULT. That will have all kinds of messed up priorities. For that reason, you should adjust your code to use OpenSSLCertificateOptions or, even better, use the TLS endpoint directly.The TL;DR is: yes, it seems that DefaultOpenSSLContextFactory produces a context that is genuinely unacceptable for HTTP/2.
Indeed it all works fine with endpoints. Thanks! I was not aware that DefaultOpenSSLContextFactory is deprecated. There is no warning about it anywhere. It seems that is is very widely used by users, I just did some github search now and found around 5k occurences of people using it: https://github.com/search?utf8=%E2%9C%93&q=defaultopensslcontextfactory&type=Code&ref=searchresults If you google for "ssl in twisted" you will also find articles that recommend it. Since so many people use it, maybe it could be updated to be more secure? If it does not make sense to update it then perhaps it would be good to deprecate it so that it does not confuse users? 2016-07-12 9:56 GMT+02:00 Tristan Seligmann <mithrandi@mithrandi.net>:
On Tue, 12 Jul 2016 at 09:43 Cory Benfield <cory@lukasa.co.uk> wrote:
For that reason, you should adjust your code to use OpenSSLCertificateOptions or, even better, use the TLS endpoint directly.
The exported name of this class is actually just "CertificateOptions", fwiw.
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 12 Jul 2016, at 09:33, Paweł Miech <pawelmhm@gmail.com> wrote:
If you google for "ssl in twisted" you will also find articles that recommend it. Since so many people use it, maybe it could be updated to be more secure? If it does not make sense to update it then perhaps it would be good to deprecate it so that it does not confuse users?
Agreed. I’m planning to begin the deprecation process, though it will take a little while as we need to remove all uses of it from within the Twisted codebase itself, as well as from the documentation. That turns out to be a bigger task than expected!
Agreed. I’m planning to begin the deprecation process, though it will take a little while as we need to remove all uses of it from within the Twisted codebase itself, as well as from the documentation. That turns out to be a bigger task than expected!
+1 One final point that I glossed over earlier
To be clear, I was not responding to your specific needs but discussing Glyph’s wider point about alerting when bad configuration is present.
When using Twisted endpoints (e.g. serverFromString) the problem with bad openssl configuration is not bad. If OS does not support ALPN (OpenSSL versions below 1.0.2) so in vast majority of Linux systems currently in use Chrome connection simply falls back to HTTP 1.1 (I tested this on Ubuntu 14.04), This means there is no error and content is served, so it's some sort of graceful degradation. This behavior is identical to nginx. I'm not sure if Twisted can and should do something about this. Maybe it can print some warning or maybe it can just let users know in documentation that HTTP2 support via ALPN (which is required in Chrome) requires Openssl 1.0.2? Adding warnings to code might require some extra development but it does not look that difficult. If you think about this, you probably dont need to check ciphers available in system, you can probably only check OpenSSL version available and check if client attempts to use ALPN. 2016-07-12 17:13 GMT+02:00 Cory Benfield <cory@lukasa.co.uk>:
On 12 Jul 2016, at 09:33, Paweł Miech <pawelmhm@gmail.com> wrote:
If you google for "ssl in twisted" you will also find articles that recommend it. Since so many people use it, maybe it could be updated to be more secure? If it does not make sense to update it then perhaps it would be good to deprecate it so that it does not confuse users?
Agreed. I’m planning to begin the deprecation process, though it will take a little while as we need to remove all uses of it from within the Twisted codebase itself, as well as from the documentation. That turns out to be a bigger task than expected!
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 12 Jul 2016, at 17:42, Paweł Miech <pawelmhm@gmail.com> wrote:
Agreed. I’m planning to begin the deprecation process, though it will take a little while as we need to remove all uses of it from within the Twisted codebase itself, as well as from the documentation. That turns out to be a bigger task than expected!
+1
One final point that I glossed over earlier
To be clear, I was not responding to your specific needs but discussing Glyph’s wider point about alerting when bad configuration is present.
When using Twisted endpoints (e.g. serverFromString) the problem with bad openssl configuration is not bad. If OS does not support ALPN (OpenSSL versions below 1.0.2) so in vast majority of Linux systems currently in use Chrome connection simply falls back to HTTP 1.1 (I tested this on Ubuntu 14.04), This means there is no error and content is served, so it's some sort of graceful degradation. This behavior is identical to nginx. I'm not sure if Twisted can and should do something about this. Maybe it can print some warning or maybe it can just let users know in documentation that HTTP2 support via ALPN (which is required in Chrome) requires Openssl 1.0.2? Adding warnings to code might require some extra development but it does not look that difficult. If you think about this, you probably dont need to check ciphers available in system, you can probably only check OpenSSL version available and check if client attempts to use ALPN.
We can actually do better than that. The way the Twisted APIs are constructed, it knows if it’s got NPN, ALPN, neither, or both. So Twisted is capable of warning in a situation where it has protocols to advertise/negotiate, but no mechanism with which to do it. Unfortunately, I’m not sure of a way of doing it that isn’t intrusive: users opt in to HTTP/2 only by having the HTTP/2 dependencies installed, which they may have for other reasons (they’re common code used by other tools). That means that you could have a situation where you have the HTTP/2 dependencies installed, install Twisted, and then get spammed with warnings because you have older OpenSSL’s. I’m definitely open to it, but I’m not sure that the user experience is good. If anyone has suggestions of how to get a better UX, I’m open to it. Cory
On Jul 12, 2016, at 12:43 AM, Cory Benfield <cory@lukasa.co.uk> wrote:
DefaultOpenSSLContextFactory should have been deprecated a long time ago.
2 years ago, to be precise: https://twistedmatrix.com/trac/ticket/6923 Someone fixing this would be tremendously useful. -glyph
On 12 Jul 2016, at 22:04, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
2 years ago, to be precise:
https://twistedmatrix.com/trac/ticket/6923 <https://twistedmatrix.com/trac/ticket/6923>
Someone fixing this would be tremendously useful.
-glyph
I tried to get started on this yesterday. Unfortunately, the stack of work that requires this means I’ll be chasing after this for a while. Specifically: - To deprecate ContextManager, we need to remove all instances of it from the code and documentation. - That’s fine, except that code/docs that used ContextManager used {connect,listen}SSL and friends (because those were the appropriate APIs) - Which means that we actually need to adjust a huge swathe of docs to use endpoints in order to remove the use of ContextManager. I’ve gotten started on this, but sadly the {connect,listen} paradigm for SSL is extremely widespread in the Twisted documentation. This means I’m going to generate *several* sizeable tickets that require review just to get to a place where we can actually put the darn deprecation marker on those classes. I’ve begun by tackling the tutorial in #8588 (https://twistedmatrix.com/trac/ticket/8588 <https://twistedmatrix.com/trac/ticket/8588>). There are further questions about the pedagogical value of this as “The Twisted Tutorial”, but for now I just want to bring it into 2014. Anyway, I’ll be spending my Twisted time on this for a while I suspect. This will delay HTTP/2 client support, unfortunately. =( Cory
Anyway, I’ll be spending my Twisted time on this for a while I suspect. This will delay HTTP/2 client support, unfortunately. =(
Isn't it better to get HTTP2 client support and just document things better for HTTP2? Or maybe even backport some features from CertificateOptions to factory? DefaultSSLContextFactory seems to work ok for cases outside HTTP2. It is not evidently broken. It is probably less secure than twisted.internet.ssl.CertificateOptions but is really broken beyond repair? 2016-07-13 10:52 GMT+02:00 Cory Benfield <cory@lukasa.co.uk>:
On 12 Jul 2016, at 22:04, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
2 years ago, to be precise:
https://twistedmatrix.com/trac/ticket/6923
Someone fixing this would be tremendously useful.
-glyph
I tried to get started on this yesterday. Unfortunately, the stack of work that requires this means I’ll be chasing after this for a while. Specifically:
- To deprecate ContextManager, we need to remove all instances of it from the code and documentation. - That’s fine, except that code/docs that used ContextManager used {connect,listen}SSL and friends (because those were the appropriate APIs) - Which means that we actually need to adjust a huge swathe of docs to use endpoints in order to remove the use of ContextManager.
I’ve gotten started on this, but sadly the {connect,listen} paradigm for SSL is extremely widespread in the Twisted documentation. This means I’m going to generate *several* sizeable tickets that require review just to get to a place where we can actually put the darn deprecation marker on those classes.
I’ve begun by tackling the tutorial in #8588 ( https://twistedmatrix.com/trac/ticket/8588). There are further questions about the pedagogical value of this as “The Twisted Tutorial”, but for now I just want to bring it into 2014.
Anyway, I’ll be spending my Twisted time on this for a while I suspect. This will delay HTTP/2 client support, unfortunately. =(
Cory
_______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com http://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 13 Jul 2016, at 10:00, Paweł Miech <pawelmhm@gmail.com> wrote:
Anyway, I’ll be spending my Twisted time on this for a while I suspect. This will delay HTTP/2 client support, unfortunately. =(
Isn't it better to get HTTP2 client support and just document things better for HTTP2? Or maybe even backport some features from CertificateOptions to factory? DefaultSSLContextFactory seems to work ok for cases outside HTTP2. It is not evidently broken. It is probably less secure than twisted.internet.ssl.CertificateOptions but is really broken beyond repair?
Generally speaking I’d say it isn’t better, for a couple of reasons. Firstly, it rarely works well to document one’s way out of a usability problem. This is doubly-true when the *rest* of the documentation is contrary to what your new documentation would say. For example, the Twisted Web howto client documentation uses ClientContextFactory, which will be utterly unsuitable for HTTP/2. More generally, having two different ways to do TLS, one of which is substantially less secure and powerful than the other, is a real problem. For example, DefaultSSLContextFactory literally only works for HTTP/2 servers by chance: it was never actually *designed* to work with ALPN and only managed to do so because we refactored the implementation to have the TLSMemoryBIOFactory apply the ALPN/NPN logic. In essence, by sheer bad luck we managed to change the HTTP/2 implementation to accidentally work with the old method, when it was never planned to do so. Worse, though, is that while the ContextFactory isn’t that bad for servers, it’s *terrible* for clients. In particular, the ClientContextFactory does not use SNI, does not validate hostnames, and generally speaking does not produce secure TLS. That means I’d want to prevent the HTTP/2 client from using the ClientContextFactory *anyway*: it’s really genuinely terrible and needs to be burned with fire. If you’re interested in speeding up the arrival of HTTP/2 client support, then, the best way to do that is to help out with the deprecation effort. I’ve got patches open for the majority of the docs problems, and will be starting to work on the code problems over the next few days. All of these patches will require review, and other people writing patches will also speed things up. Basically, I’m disinclined to want to prolong the lifetime of something that was supposed to go away two years ago. Twisted has a lot of things in it that were *supposed* to be deprecated but were never *actually* deprecated, and that kind of soft deprecation ends up causing the kind of problem we’ve bumped into here, whereby a lot of people have code that “works” with current features, but new features are designed only with an eye to the best practice. On a personal level, I want to push for Twisted to *actually* deprecate things that are soft deprecated. That has the best long-term effect on the project, by reducing the amount of code that needs to be maintained and encouraging users to move towards features that function better. Cory
My earlier replies to this thread were pretty terse, so just to expand on it:
On Jul 13, 2016, at 3:39 AM, Cory Benfield <cory@lukasa.co.uk> wrote:
On 13 Jul 2016, at 10:00, Paweł Miech <pawelmhm@gmail.com <mailto:pawelmhm@gmail.com>> wrote:
Anyway, I’ll be spending my Twisted time on this for a while I suspect. This will delay HTTP/2 client support, unfortunately. =(
Isn't it better to get HTTP2 client support and just document things better for HTTP2? Or maybe even backport some features from CertificateOptions to factory? DefaultSSLContextFactory seems to work ok for cases outside HTTP2.
It really doesn't.
It is not evidently broken.
For clients, it's horribly broken and provides no security whatsoever. For servers, it's badly configured enough that you will have problems with SSLLabs and other automated testing things.
It is probably less secure than twisted.internet.ssl.CertificateOptions but is really broken beyond repair?
Yes, it really is. The only reason we didn't just delete it immediately in a compatibility-breaking exception is that if you knew what you were doing, it was the only way to reasonably be secure in the past, and we did not want to punish those users who had done the best they could with what Twisted was giving them. It was possible to make something secure with it, if you absolutely knew what you were doing and you layered a bunch of things on top of it.
Generally speaking I’d say it isn’t better, for a couple of reasons.
Firstly, it rarely works well to document one’s way out of a usability problem. This is doubly-true when the *rest* of the documentation is contrary to what your new documentation would say. For example, the Twisted Web howto client documentation uses ClientContextFactory, which will be utterly unsuitable for HTTP/2.
Not to mention the fact that it provides no security. Client-side, anything other than optionsForClientTLS is probably completely unacceptable. Again… it was possible to add your own security to ClientContextFactory but that is the only reason we didn't delet it.
More generally, having two different ways to do TLS, one of which is substantially less secure and powerful than the other, is a real problem. For example, DefaultSSLContextFactory literally only works for HTTP/2 servers by chance: it was never actually *designed* to work with ALPN and only managed to do so because we refactored the implementation to have the TLSMemoryBIOFactory apply the ALPN/NPN logic. In essence, by sheer bad luck we managed to change the HTTP/2 implementation to accidentally work with the old method, when it was never planned to do so.
In fairness, the fact that random accidents like this happen is as much a fault of OpenSSL's ... idiosyncratic, shall we say, API, where promiscuous mutable data sharing is completely the norm, as it is a fault of Twisted's bad early misinterpretation of said API.
Worse, though, is that while the ContextFactory isn’t that bad for servers, it’s *terrible* for clients. In particular, the ClientContextFactory does not use SNI, does not validate hostnames, and generally speaking does not produce secure TLS. That means I’d want to prevent the HTTP/2 client from using the ClientContextFactory *anyway*: it’s really genuinely terrible and needs to be burned with fire.
TERRIBLE TERRIBLE TERRIBLE FOR CLIENTS DON'T USE IT
If you’re interested in speeding up the arrival of HTTP/2 client support, then, the best way to do that is to help out with the deprecation effort. I’ve got patches open for the majority of the docs problems, and will be starting to work on the code problems over the next few days. All of these patches will require review, and other people writing patches will also speed things up.
+1000 thank you for saying this.
Basically, I’m disinclined to want to prolong the lifetime of something that was supposed to go away two years ago. Twisted has a lot of things in it that were *supposed* to be deprecated but were never *actually* deprecated, and that kind of soft deprecation ends up causing the kind of problem we’ve bumped into here, whereby a lot of people have code that “works” with current features, but new features are designed only with an eye to the best practice. On a personal level, I want to push for Twisted to *actually* deprecate things that are soft deprecated. That has the best long-term effect on the project, by reducing the amount of code that needs to be maintained and encouraging users to move towards features that function better.
Basically I just wanted to chime in to make it clear that I _absolutely_ agree with _every_ part of this and we should _immediately_ move to implement all of these plans. Cory, I also really appreciate you going through and taking the time to clear up the messaging around these APIs in our documentation and our implementations to help users understand what's going on so you don't need to be a member of the Twisted cabal to get good security and modern protocol features. Thanks, -glyph
On 11 Jul 2016, at 20:22, Glyph Lefkowitz <glyph@twistedmatrix.com> wrote:
So pyOpenSSL/Cryptography doesn't have SSL_get_current_cipher anywhere?
get_current_cipher isn’t helpful. In particular, it puts us in an awkward place where we have a connection that has been negotiated for HTTP/2, but we cannot use it. The only action Twisted can meaningfully take at that point is to log and tear the connection down, which doesn’t really solve our problems. We can do that, for sure, but it wouldn’t be much clearer than what happened here. Cory
On Jul 12, 2016, at 12:45 AM, Cory Benfield <cory@lukasa.co.uk <mailto:cory@lukasa.co.uk>> wrote:
On 11 Jul 2016, at 20:22, Glyph Lefkowitz <glyph@twistedmatrix.com <mailto:glyph@twistedmatrix.com>> wrote:
So pyOpenSSL/Cryptography doesn't have SSL_get_current_cipher anywhere?
get_current_cipher isn’t helpful. In particular, it puts us in an awkward place where we have a connection that has been negotiated for HTTP/2, but we cannot use it. The only action Twisted can meaningfully take at that point is to log and tear the connection down, which doesn’t really solve our problems.
We can do that, for sure, but it wouldn’t be much clearer than what happened here.
Just generally we should probably be logging this (at INFO or somesuch) regardless, so that interested parties can extract which cipher suites are actually in use. But perhaps not relevant to this problem, really. -glyph
participants (8)
-
Adi Roiban -
Amber "Hawkie" Brown -
Cory Benfield -
Craig Rodrigues -
Donal McMullan -
Glyph Lefkowitz -
Paweł Miech -
Tristan Seligmann