[Twisted-Python] twisted.internet.ssl
Hi, I am using a self-signed CA to issue server and client(s) certificates. My server is using the standard Python ssl module. One client, that is using twisted.internet.ssl, consistently fails to connect with: On the Server: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:661), On the Client: [WARNING] [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')] This is my code: path = getModule(__name__).filePath.sibling(u'data') txt = path.child(u'ca.crt').getContent() cacert = ssl.Certificate.loadPEM(txt) root = ssl.trustRootFromCertificates([cacert]) txt = path.child(u'client.pem').getContent() mycert = ssl.PrivateCertificate.loadPEM(txt) ctx = ssl.optionsForClientTLS(hostName, trustRoot=root, clientCertificate=mycert) reactor.connectSSL(hostName, portNumber, factory, ctx) I am using the latest git trunk code. With a regular ssl client I don't have an issue. A known bug? Thanks, Enoch.
On Wed, Oct 25, 2017 at 08:07:26PM +0000, Enoch W. wrote:
A known bug? Hello Enoch, A question, does the server have an intermediate cert that it is using? If so, then the problem might be that your trust root needs both the CA and intermediate.
I have some https test code I hacked together a while back (neither reviewed nor heavily tested, but same general idea). https://github.com/jlitzingerdev/twisted-benchmarks/blob/https-benchmark/web... If I omit the intermediate cert I get the same error. Cheers, -Jason
Thanks Jason, With a more specific error message raised by github latest code [thanks!] the problem became clear. I suggest to change the documentation of 'optionsForClientTLS' - from: ctx = ssl.optionsForClientTLS(hostName, trustRoot=root, clientCertificate=mycert) to: ctx = ssl.optionsForClientTLS(commonName, trustRoot=root, clientCertificate=mycert) That is, what this ctx (the contextFactory) really expects is the server's certificate commonName which often is, but not in my case, the server's hostName. Interestingly, Python's standard ssl package does not verify this field. Regards, Enoch. On 10/26/2017 01:13 AM, Jason Litzinger wrote: On Wed, Oct 25, 2017 at 08:07:26PM +0000, Enoch W. wrote: A known bug? Hello Enoch, A question, does the server have an intermediate cert that it is using? If so, then the problem might be that your trust root needs both the CA and intermediate. I have some https test code I hacked together a while back (neither reviewed nor heavily tested, but same general idea). https://github.com/jlitzingerdev/twisted-benchmarks/blob/https-benchmark/web... If I omit the intermediate cert I get the same error. Cheers, -Jason _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com<mailto:Twisted-Python@twistedmatrix.com> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On Oct 26, 2017, at 6:35 AM, Enoch W. <ixew@hotmail.com> wrote:
Thanks Jason,
With a more specific error message raised by github latest code [thanks!] the problem became clear. I suggest to change the documentation of 'optionsForClientTLS' -
from: ctx = ssl.optionsForClientTLS(hostName, trustRoot=root, clientCertificate=mycert)
to: ctx = ssl.optionsForClientTLS(commonName, trustRoot=root, clientCertificate=mycert)
That is, what this ctx (the contextFactory) really expects is the server's certificate commonName which often is, but not in my case, the server's hostName.
Interestingly, Python's standard ssl package does not verify this field.
It sounds like you somehow installed Twisted without the "service_identity" module; it only verifies commonName if it has no other choice. When you install Twisted, be sure to install the `tls` extra: pip install twisted[tls]. -glyph
On 2017-10-25, at 21:07, Enoch W. <ixew@hotmail.com> wrote:
Hi,
I am using a self-signed CA to issue server and client(s) certificates.
My server is using the standard Python ssl module. One client, that is using twisted.internet.ssl, consistently fails to connect with: On the Server: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:661), On the Client: [WARNING] [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')]
This is my code:
path = getModule(__name__).filePath.sibling(u'data')
txt = path.child(u'ca.crt').getContent() cacert = ssl.Certificate.loadPEM(txt) root = ssl.trustRootFromCertificates([cacert])
txt = path.child(u'client.pem').getContent() mycert = ssl.PrivateCertificate.loadPEM(txt)
ctx = ssl.optionsForClientTLS(hostName, trustRoot=root, clientCertificate=mycert)
reactor.connectSSL(hostName, portNumber, factory, ctx)
I am using the latest git trunk code. With a regular ssl client I don't have an issue.
A known bug?
I would review a few things before suspecting a bug. Your code is using client side certificates (nice, not often seen) so both server and client need to validate each other's certificates on connection establishment. It seems they are both failing, but we're only looking at your client code though. Here are a few ideas: 1. Double check your certificates: Issuers, Subjects, Dates, SANs, etc. 2. Check that the hostName in optionsForClientTLS matches the name in the server certificate. 3. Use the latest Twisted release instead of trunk, if possible. Do the same for pyOpenSSL and other dependencies. 4. On the client side try using SSL4ClientEndpoint instead of connectSSL. I'm almost sure they behave differently regarding validation (could not quickly find docs on that, though: I've had your problem before and I think I addressed it this way). Instead of reactor.connectSSL(...) go for something like: ep = endpoints.SSL4ClientEndpoint( reactor, host=hostName, port=portNumber, sslContextFactory=ctx, ) client = yield ep.connect(factory) For completeness, even though my client skeleton is mostly your client code with this diff, a quick test shows that my client also establishes the TLS connection if I do the reverse: replace SSL4ClientEndpoint with connectSSL in my code. This may not be the culprit, but I would try and see if anything changes. 5. On the server side I have the following Twisted code skeleton: dhFile = filepath.FilePath(...) dhParams = ssl.DiffieHellmanParameters.fromFile(dhFile) caCert = ssl.Certificate.loadPEM(...) privateCert = ssl.PrivateCertificate.loadPEM(...) cf = ssl.CertificateOptions( privateKey=privateCert.privateKey.original, certificate=privateCert.original, trustRoot=caCert, dhParameters=dhParams, ) ep = endpoints.SSL4ServerEndpoint( reactor, port=..., sslContextFactory=cf, ) f = protocol.Factory.forProtocol(...) ep.listen(f) reactor.run() Your code is obviously different, if based on the standard library's ssl module. If 1 to 4 don't produce results, this is a good candidate for needing some work. Can you share a minimal version of that? I'll be glad to take it for a spin. 6. Double check your certificates, again. Triple-check them if managed manually. For completeness and reference: - See http://twistedmatrix.com/documents/current/core/howto/ssl.html as a starting point Twisted TLS, containing useful information and working examples. - My working test environment runs Linux with OpenSSL 1.0.1t. Cheers, -- exvito
Thanks exvito for your detailed response. Re 2 & 3: You're right on the nail :-) See my previous email to Jason. Re 4: I can't use the high level Transport mechanism as I am using Twisted (most of the time) through another library layer (pymodbus). Re 5: My server is still using Python's bloated ThreadingTCPServer model. Re 6: Right! Bottom line, I suggested a change of argument name, from "ssl.optionsForClientTLS(hostName, ..." to "ssl.optionsForClientTLS(commonName, ..." Regards, Enoch. On 10/26/2017 05:20 AM, ex vito wrote: On 2017-10-25, at 21:07, Enoch W. <ixew@hotmail.com><mailto:ixew@hotmail.com> wrote: Hi, I am using a self-signed CA to issue server and client(s) certificates. My server is using the standard Python ssl module. One client, that is using twisted.internet.ssl, consistently fails to connect with: On the Server: [SSL: TLSV1_ALERT_UNKNOWN_CA] tlsv1 alert unknown ca (_ssl.c:661), On the Client: [WARNING] [('SSL routines', 'tls_process_server_certificate', 'certificate verify failed')] This is my code: path = getModule(__name__).filePath.sibling(u'data') txt = path.child(u'ca.crt').getContent() cacert = ssl.Certificate.loadPEM(txt) root = ssl.trustRootFromCertificates([cacert]) txt = path.child(u'client.pem').getContent() mycert = ssl.PrivateCertificate.loadPEM(txt) ctx = ssl.optionsForClientTLS(hostName, trustRoot=root, clientCertificate=mycert) reactor.connectSSL(hostName, portNumber, factory, ctx) I am using the latest git trunk code. With a regular ssl client I don't have an issue. A known bug? I would review a few things before suspecting a bug. Your code is using client side certificates (nice, not often seen) so both server and client need to validate each other's certificates on connection establishment. It seems they are both failing, but we're only looking at your client code though. Here are a few ideas: 1. Double check your certificates: Issuers, Subjects, Dates, SANs, etc. 2. Check that the hostName in optionsForClientTLS matches the name in the server certificate. 3. Use the latest Twisted release instead of trunk, if possible. Do the same for pyOpenSSL and other dependencies. 4. On the client side try using SSL4ClientEndpoint instead of connectSSL. I'm almost sure they behave differently regarding validation (could not quickly find docs on that, though: I've had your problem before and I think I addressed it this way). Instead of reactor.connectSSL(...) go for something like: ep = endpoints.SSL4ClientEndpoint( reactor, host=hostName, port=portNumber, sslContextFactory=ctx, ) client = yield ep.connect(factory) For completeness, even though my client skeleton is mostly your client code with this diff, a quick test shows that my client also establishes the TLS connection if I do the reverse: replace SSL4ClientEndpoint with connectSSL in my code. This may not be the culprit, but I would try and see if anything changes. 5. On the server side I have the following Twisted code skeleton: dhFile = filepath.FilePath(...) dhParams = ssl.DiffieHellmanParameters.fromFile(dhFile) caCert = ssl.Certificate.loadPEM(...) privateCert = ssl.PrivateCertificate.loadPEM(...) cf = ssl.CertificateOptions( privateKey=privateCert.privateKey.original, certificate=privateCert.original, trustRoot=caCert, dhParameters=dhParams, ) ep = endpoints.SSL4ServerEndpoint( reactor, port=..., sslContextFactory=cf, ) f = protocol.Factory.forProtocol(...) ep.listen(f) reactor.run() Your code is obviously different, if based on the standard library's ssl module. If 1 to 4 don't produce results, this is a good candidate for needing some work. Can you share a minimal version of that? I'll be glad to take it for a spin. 6. Double check your certificates, again. Triple-check them if managed manually. For completeness and reference: - See http://twistedmatrix.com/documents/current/core/howto/ssl.html as a starting point Twisted TLS, containing useful information and working examples. - My working test environment runs Linux with OpenSSL 1.0.1t. Cheers, -- exvito _______________________________________________ Twisted-Python mailing list Twisted-Python@twistedmatrix.com<mailto:Twisted-Python@twistedmatrix.com> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
On 2017-10-26, at 14:59, Enoch W. <ixew@hotmail.com> wrote:
Thanks exvito for your detailed response. Re 2 & 3: You're right on the nail :-) See my previous email to Jason. Re 4: I can't use the high level Transport mechanism as I am using Twisted (most of the time) through another library layer (pymodbus). Re 5: My server is still using Python's bloated ThreadingTCPServer model. Re 6: Right!
You are welcome.
Bottom line, I suggested a change of argument name, from "ssl.optionsForClientTLS(hostName, ..." to "ssl.optionsForClientTLS(commonName, ..."
That would be misleading: Keep in mind that the name validation will be performed against the server certificate's SANs (subject alternative name) entries, if they exist. More importantly, in the context of your suggestion: as far as I can tell, when SANs are present in the server certificate, not only will the name verification be performed against those, but the certificate's subject commonName will be ignored. IIRC, this is in line with an existing RFC recommendation (maybe 6125?). Thus, hostName is a pretty good name and commonName wouldn't be as good. :) Cheers, -- exvito
participants (4)
-
Enoch W. -
ex vito -
Glyph -
Jason Litzinger