[Twisted-Python] txsni + alpn + acme (letsencrypt)
![](https://secure.gravatar.com/avatar/d7ff36e4d7c8060fadaa7c20f4a5649e.jpg?s=120&d=mm&r=g)
Hello. Can you help me to learn to debug tls problems in twisted? I was disappointed that txacme, an automatic way to get certificates for twisted web, stopped working, so I'm trying to add a responder for the new challenge type. It sends a special certificate if the CA negotiates acme using alpn, proving domain control, then you get a certificate. Seems to work with openssl s_client but letsencrypt says "no application protocol". Anyone know where that error comes from? https://github.com/glyph/txsni/pull/26
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Mar 23, 2019, at 7:21 AM, Daniel Holth <dholth@gmail.com> wrote:
Hello. Can you help me to learn to debug tls problems in twisted?
Hi Daniel! Thanks so much for trying to improve this aspect of the Twisted ecosystem.
I was disappointed that txacme, an automatic way to get certificates for twisted web, stopped working, so I'm trying to add a responder for the new challenge type.
This situation is definitely bad; as JP tersely put it on https://github.com/twisted/txacme/issues/148, "The published documentation makes it appear as though the server endpoint works though it has apparently been broken for about a year". But to prevent onlookers from thinking the situation is even worse than it is, please allow me to clarify: txacme still "works" in the sense that it supports the http-01 challenge (which is still supported for the forseeable future, AFAIK), and its challenge/response implementation also powers https://pypi.org/project/lancer/ <https://pypi.org/project/lancer/>, which uses the dns-01 challenge. However, its endpoints (and particularly its string endpoint plugin) doesn't support this challenge directly. The thing that's broken is sni-01, which was disabled globally by Let's Encrypt for security reasons: https://www.zdnet.com/article/lets-encrypt-disables-tls-sni-01-validation/ <https://www.zdnet.com/article/lets-encrypt-disables-tls-sni-01-validation/> One approach to fixing this is to implement a string endpoint syntax that listens on port 80 and supports HTTP-01 rather than trying to make TLS support TLS-ALPN-01. See discussion here: https://github.com/twisted/txacme/issues/129 <https://github.com/twisted/txacme/issues/129>. Brian Warner already has a branch here: https://github.com/warner/txacme/tree/129-http01-endpoint <https://github.com/warner/txacme/tree/129-http01-endpoint> but has not made a PR yet. Perhaps you could resurrect this code, polish anything that needs polishing, and get it integrated and released :).
It sends a special certificate if the CA negotiates acme using alpn, proving domain control, then you get a certificate.
Even if you're not hitting this specific problem, continuing down this road (implementing the ALPN challenge using OpenSSL) is unfortunately a bad idea. Fundamentally, the ALPN callback in OpenSSL is poorly designed, and optimized for the very specific negotiation process that HTTP/2 requires. The guidance from people familiar with OpenSSL is - and I know this sounds like a joke, I wish I were kidding here, but no, this is seriously the recommendation - implement your own parser for the ClientHello, extract the ALPN field yourself, then construct an OpenSSL connection object based on what you find there, and hand it the bytes you just parsed. Given the mechanics of the challenge message (i.e.: it's going to be in the initial clienthello) you could probably cheese it with a regex just to get something working. I hope this was helpful. Good luck, and please keep us posted on your efforts! -g
![](https://secure.gravatar.com/avatar/d7ff36e4d7c8060fadaa7c20f4a5649e.jpg?s=120&d=mm&r=g)
Wow! Such broken. I was starting to get suspicious of openssl myself. Poor documentation about the rules on context switching and whether doing things in a certain order should trigger callbacks. At least you can get a cert when the ALPN / ACME certificate (and DEFAULT?) is the only one provided by twisted. If the several attempts they make came from the same IP address that might be one way to hack it. If it gets that bad I'll put the ClientHello regex next to the regex-based pkcs parser from my rsalette library :) Fixing the http-01 challenge is a very rational suggestion. Thanks! Daniel
![](https://secure.gravatar.com/avatar/d7ff36e4d7c8060fadaa7c20f4a5649e.jpg?s=120&d=mm&r=g)
HOLY REGEX BATMAN class _ConnectionProxy(object): def bio_write(self, buf): if ACME_TLS_1 in buf: self.acme_tls_1 = True self.bio_write = self._obj.bio_write return self._obj.bio_write(buf) Now we can choose the acme certificate store in the sni callback and make letsencrypt happy!
![](https://secure.gravatar.com/avatar/d7ff36e4d7c8060fadaa7c20f4a5649e.jpg?s=120&d=mm&r=g)
All we have to do is have some kind of per connection certificate store or flag. If acme is in the first packet and the special certificate exists, send it. Otherwise send the normal certificate, for a very short window of possible brokenness. Letsencrypt may or may not require correct alpn negotiation. Should be simple. I'm happy running the acme client separately and listing my domain instead of doing it all on demand inside twisted. On Sat, Mar 23, 2019, 23:59 Glyph <glyph@twistedmatrix.com> wrote:
![](https://secure.gravatar.com/avatar/d7ff36e4d7c8060fadaa7c20f4a5649e.jpg?s=120&d=mm&r=g)
Pull request for txsni acme https://github.com/glyph/txsni/pull/28 On Sun, Mar 24, 2019, 16:33 Glyph <glyph@twistedmatrix.com> wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Thanks! I put some review comments on it. I would encourage others with interest in this area to have a look; I might not get back to this for a couple of weeks, but I'd be happy to give people collaborator permissions on the repo if they'd like to help out. (Frankly it's probably time that this project grew up and moved over to the Twisted org anyway, given that txacme depends on it...) -g
![](https://secure.gravatar.com/avatar/d7ff36e4d7c8060fadaa7c20f4a5649e.jpg?s=120&d=mm&r=g)
Let me know if you're able to try getting a https certificate in this way: Using tls-alpn-01 negotiation with txsni (acme branch) and the dehydrated letsencrypt client: Install txsni (acme branch): pip install git+https://github.com/dholth/txsni@acme#egg=txsni Unpack dehydrated acme client shell script from https://dehydrated.io/ Make and enter config directory: mkdir -p ~/etc/dehydrated cd ~/etc/dyhydrated Use acmesni to listen on port 443: authbind twist web --listen acmesni:~/etc/dehydrated:tcp6:443 & Make config file: echo CHALLENGETYPE="tls-alpn-01" > config Create letsencrypt account: dehydrated --register --accept-terms Request certificate (replace example.com with your fqnd): dehydrated -c -d example.com Letsencrypt should request a special certificate from the twisted web server to prove domain control, and then dehydrated installs the new certificate for 'example.com' in ~/etc/dehydrated/certs/... https://example.com will be ready to go. Authbind, to listen on privileged ports in linux without root: (install authbind); touch /etc/authbind/byport/{80,443}; chown <username> /etc/authbind/byport/*; chmod u+x /etc/authbind/byport/* On Sun, Mar 24, 2019 at 9:17 PM Daniel Holth <dholth@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
In fairness, they do realize that this is a bit of a mess, and eventually one hopes there will be something better: https://github.com/openssl/openssl/issues/6109 <https://github.com/openssl/openssl/issues/6109>
What IP addresses does Let’s Encrypt use to validate my web server? We don’t publish a list of IP addresses we use to validate, because they may change at any time. In the future we may validate from multiple IP addresses at once. Source: https://letsencrypt.org/docs/faq/#what-ip-addresses-does-let-s-encrypt-use-t... <https://letsencrypt.org/docs/faq/#what-ip-addresses-does-let-s-encrypt-use-t...>
If it gets that bad I'll put the ClientHello regex next to the regex-based pkcs parser from my rsalette library :)
Oh no :-(. Don't do RSA in pure python, that's an invitation to timing attacks.
Fixing the http-01 challenge is a very rational suggestion.
Thanks! If you could get Warner's patch over the finish line, that would probably be the best, most practical step forward.
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
On Mar 23, 2019, at 7:21 AM, Daniel Holth <dholth@gmail.com> wrote:
Hello. Can you help me to learn to debug tls problems in twisted?
Hi Daniel! Thanks so much for trying to improve this aspect of the Twisted ecosystem.
I was disappointed that txacme, an automatic way to get certificates for twisted web, stopped working, so I'm trying to add a responder for the new challenge type.
This situation is definitely bad; as JP tersely put it on https://github.com/twisted/txacme/issues/148, "The published documentation makes it appear as though the server endpoint works though it has apparently been broken for about a year". But to prevent onlookers from thinking the situation is even worse than it is, please allow me to clarify: txacme still "works" in the sense that it supports the http-01 challenge (which is still supported for the forseeable future, AFAIK), and its challenge/response implementation also powers https://pypi.org/project/lancer/ <https://pypi.org/project/lancer/>, which uses the dns-01 challenge. However, its endpoints (and particularly its string endpoint plugin) doesn't support this challenge directly. The thing that's broken is sni-01, which was disabled globally by Let's Encrypt for security reasons: https://www.zdnet.com/article/lets-encrypt-disables-tls-sni-01-validation/ <https://www.zdnet.com/article/lets-encrypt-disables-tls-sni-01-validation/> One approach to fixing this is to implement a string endpoint syntax that listens on port 80 and supports HTTP-01 rather than trying to make TLS support TLS-ALPN-01. See discussion here: https://github.com/twisted/txacme/issues/129 <https://github.com/twisted/txacme/issues/129>. Brian Warner already has a branch here: https://github.com/warner/txacme/tree/129-http01-endpoint <https://github.com/warner/txacme/tree/129-http01-endpoint> but has not made a PR yet. Perhaps you could resurrect this code, polish anything that needs polishing, and get it integrated and released :).
It sends a special certificate if the CA negotiates acme using alpn, proving domain control, then you get a certificate.
Even if you're not hitting this specific problem, continuing down this road (implementing the ALPN challenge using OpenSSL) is unfortunately a bad idea. Fundamentally, the ALPN callback in OpenSSL is poorly designed, and optimized for the very specific negotiation process that HTTP/2 requires. The guidance from people familiar with OpenSSL is - and I know this sounds like a joke, I wish I were kidding here, but no, this is seriously the recommendation - implement your own parser for the ClientHello, extract the ALPN field yourself, then construct an OpenSSL connection object based on what you find there, and hand it the bytes you just parsed. Given the mechanics of the challenge message (i.e.: it's going to be in the initial clienthello) you could probably cheese it with a regex just to get something working. I hope this was helpful. Good luck, and please keep us posted on your efforts! -g
![](https://secure.gravatar.com/avatar/d7ff36e4d7c8060fadaa7c20f4a5649e.jpg?s=120&d=mm&r=g)
Wow! Such broken. I was starting to get suspicious of openssl myself. Poor documentation about the rules on context switching and whether doing things in a certain order should trigger callbacks. At least you can get a cert when the ALPN / ACME certificate (and DEFAULT?) is the only one provided by twisted. If the several attempts they make came from the same IP address that might be one way to hack it. If it gets that bad I'll put the ClientHello regex next to the regex-based pkcs parser from my rsalette library :) Fixing the http-01 challenge is a very rational suggestion. Thanks! Daniel
![](https://secure.gravatar.com/avatar/d7ff36e4d7c8060fadaa7c20f4a5649e.jpg?s=120&d=mm&r=g)
HOLY REGEX BATMAN class _ConnectionProxy(object): def bio_write(self, buf): if ACME_TLS_1 in buf: self.acme_tls_1 = True self.bio_write = self._obj.bio_write return self._obj.bio_write(buf) Now we can choose the acme certificate store in the sni callback and make letsencrypt happy!
![](https://secure.gravatar.com/avatar/d7ff36e4d7c8060fadaa7c20f4a5649e.jpg?s=120&d=mm&r=g)
All we have to do is have some kind of per connection certificate store or flag. If acme is in the first packet and the special certificate exists, send it. Otherwise send the normal certificate, for a very short window of possible brokenness. Letsencrypt may or may not require correct alpn negotiation. Should be simple. I'm happy running the acme client separately and listing my domain instead of doing it all on demand inside twisted. On Sat, Mar 23, 2019, 23:59 Glyph <glyph@twistedmatrix.com> wrote:
![](https://secure.gravatar.com/avatar/d7ff36e4d7c8060fadaa7c20f4a5649e.jpg?s=120&d=mm&r=g)
Pull request for txsni acme https://github.com/glyph/txsni/pull/28 On Sun, Mar 24, 2019, 16:33 Glyph <glyph@twistedmatrix.com> wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
Thanks! I put some review comments on it. I would encourage others with interest in this area to have a look; I might not get back to this for a couple of weeks, but I'd be happy to give people collaborator permissions on the repo if they'd like to help out. (Frankly it's probably time that this project grew up and moved over to the Twisted org anyway, given that txacme depends on it...) -g
![](https://secure.gravatar.com/avatar/d7ff36e4d7c8060fadaa7c20f4a5649e.jpg?s=120&d=mm&r=g)
Let me know if you're able to try getting a https certificate in this way: Using tls-alpn-01 negotiation with txsni (acme branch) and the dehydrated letsencrypt client: Install txsni (acme branch): pip install git+https://github.com/dholth/txsni@acme#egg=txsni Unpack dehydrated acme client shell script from https://dehydrated.io/ Make and enter config directory: mkdir -p ~/etc/dehydrated cd ~/etc/dyhydrated Use acmesni to listen on port 443: authbind twist web --listen acmesni:~/etc/dehydrated:tcp6:443 & Make config file: echo CHALLENGETYPE="tls-alpn-01" > config Create letsencrypt account: dehydrated --register --accept-terms Request certificate (replace example.com with your fqnd): dehydrated -c -d example.com Letsencrypt should request a special certificate from the twisted web server to prove domain control, and then dehydrated installs the new certificate for 'example.com' in ~/etc/dehydrated/certs/... https://example.com will be ready to go. Authbind, to listen on privileged ports in linux without root: (install authbind); touch /etc/authbind/byport/{80,443}; chown <username> /etc/authbind/byport/*; chmod u+x /etc/authbind/byport/* On Sun, Mar 24, 2019 at 9:17 PM Daniel Holth <dholth@gmail.com> wrote:
![](https://secure.gravatar.com/avatar/e1554622707bedd9202884900430b838.jpg?s=120&d=mm&r=g)
In fairness, they do realize that this is a bit of a mess, and eventually one hopes there will be something better: https://github.com/openssl/openssl/issues/6109 <https://github.com/openssl/openssl/issues/6109>
What IP addresses does Let’s Encrypt use to validate my web server? We don’t publish a list of IP addresses we use to validate, because they may change at any time. In the future we may validate from multiple IP addresses at once. Source: https://letsencrypt.org/docs/faq/#what-ip-addresses-does-let-s-encrypt-use-t... <https://letsencrypt.org/docs/faq/#what-ip-addresses-does-let-s-encrypt-use-t...>
If it gets that bad I'll put the ClientHello regex next to the regex-based pkcs parser from my rsalette library :)
Oh no :-(. Don't do RSA in pure python, that's an invitation to timing attacks.
Fixing the http-01 challenge is a very rational suggestion.
Thanks! If you could get Warner's patch over the finish line, that would probably be the best, most practical step forward.
participants (2)
-
Daniel Holth
-
Glyph