RFC: expose ssl internals for use by ctypes/cffi

Hi all, for quite some time I've been working on a Python implementation of a protocol called NTS which requires access to an API in OpenSSL which is not provided by the Python ssl module. I added a patch for that which unfortunately for me the maintainer did not want to accept. Some comments were made of a possible future way to give more generic access to the openssl library via ctypes/cffi but I have been unable to find more information about that. I was home sick last week and decided to take a shot at it and have built something that I feel is a bit ugly but does seem to work. I'd like to some feedback on this approach. My patches can be found on github, based on the Python 3.11 tag: https://github.com/python/cpython/compare/3.11...wingel:cpython:main Here's a short description of each patch on this branch: "bpo-37952: SSL: add support for export_keying_material" is my old patch which adds the method I need to the ssl library just for reference. The other commits add the necessary infrastructure with some example code. These commits are not ready for submission but hopefully they show what I have in mind. "Add CRYPTO_DLL_PATH and SSL_DLL_PATH to the _ssl module. " This commit adds two constants to the "_ssl" C module with the paths to libcrypto and libssl respectively. On Linux dladdr and on Windows GetModuleHandle/GetModuleFilename are used on a symbol in each library to find the path to the corresponding DLL. I've verified that this works Debian Bulleye and on Windows 10 with Visual Studio 2017. I don't own a Mac so I haven't been able to test this on macOS, but I believe dladdr is available on modern macOS so it might work out of the box. With the paths it's possible to use ctypes or cffi get a handle to these libraries. "Add API to get the address of the SSL structure" then adds an API to an SSLSocket which returns the address of the corresponding "SSL" C structure. This address can be used by ctypes/cffi. One would probably want to expose SSL_CTX, SSL_SESSION and BIO too but I started with just SSL since that's what my code needs right now. "Add a small test program" is a small test program that uses the infrastructure from the two above commits to call C functions in libssl/libcrypto using both ctypes and cffi. It's a bit ugly but hopefully it's not too hard to understand. "Example of how to extend the ssl library using ctypes" is an example of how a Python module that extends the SSL library using ctypes could look. First get a handle to libssl using ctypes, set up ctypes with the correct API for the export_keying_material function, wrap it in a more Pythonic function and then extend SSLSocket with the new function. A simplified version looks like this: import ssl, ctypes ssl_lib = ctypes.CDLL(ssl._ssl.SSL_DLL_PATH) ssl_lib.SSL_export_keying_material.argtypes = ( ctypes.c_void_p, # SSL pointer ctypes.c_void_p, ctypes.c_size_t, # out pointer, out length ctypes.c_void_p, ctypes.c_size_t, # label buffer, label length ctypes.c_void_p, ctypes.c_size_t, # context, context length ctypes.c_int) # use context flag ssl_lib.SSL_export_keying_material.restype = ctypes.c_int def SSL_export_keying_material(self, label, key_len, context = None): c_key = ctypes.create_string_buffer(key_len) c_label = ctypes.create_string_buffer(len(label)) c_context = ctypes.create_string_buffer(context, len(context)) if ssl_lib.SSL_export_keying_material( self._sslobj.get_internal_addr(), c_key, key_len, c_label, len(label), c_context, len(context), 1); return bytes(c_key) ssl.SSLSocket.export_keying_material = SSL_export_keying_material There's a final commit "Expose more OPENSSL_ variables" which add some more constants to the ssl module which expose the cflags and build information from OpenSSL. This patch is not really necessary, but it might be a good idea to compare these constants with the corresponding constants retrieved using ctypes/cffi to make sure that exactly the same version of the openssl library is used. Does this seem like a good idea? As I said, I feel that it is a bit ugly, but it does mean that if someone wants to use some SSL_really_obscure_function in libcrypto or libssl they can do that without having to rebuild all of CPython themselves. Or if they want to integrate with some other C library that wants a raw pointer to a SSL socket. Hopefully this would reduce the burden on the ssl module maintainers a bit. Anyway, if you think this is a good approach I could clean up my patches, add support for SSL_CTX/SSL_SESSION/BIO, document all of this and make it into a proper pull request. /Christer

On 11/30/2022 4:52 PM, christer@weinigel.se wrote:
Does this seem like a good idea? As I said, I feel that it is a bit ugly, but it does mean that if someone wants to use some SSL_really_obscure_function in libcrypto or libssl they can do that without having to rebuild all of CPython themselves.
Broadly, no, I don't think it's a good idea. We don't like encouraging users to do things that make it hard to support them in the future. Nonetheless, it's one that I've had to do, and so realistically I think it's okay to *enable* the hack without endorsing it. This is one of the reasons I switched the Windows builds to dynamically linked OpenSSL builds (they used to be statically linked, which meant there was no way to get at the unused exports). So now you can use `import _ssl; ctypes.CDLL("libssl-1_1")` to get at other exports from the module if you need them, and there's a similar trick to get the raw handle that I don't recall off the top of my head. But the only reason I'd ever want to document this is to tell people not to rely on it. If you control your environment well enough that you can guarantee it'll work for you, that's great. Nobody else should ever think they're doing the "right thing". Cheers, Steve

On Wed, Nov 30, 2022 at 12:47 PM Steve Dower <steve.dower@python.org> wrote:
+1 ... and in general if you want access to other OpenSSL APIs not already in the ssl module, getting them via non-stdlib packages on PyPI would be a better idea. https://pypi.org/project/cryptography/ is very well supported. https://pypi.org/project/oscrypto/ exists and is quite interesting. the old https://pypi.org/project/M2Crypto/ package still exists and seems to be maintained (wow). More context: We don't like the ssl module in the standard library - it is already too tightly tied to OpenSSL: https://discuss.python.org/t/our-future-with-openssl/21486 So if you want specific OpenSSL APIs that are not exposed, seeking to see them added to the standard library where they would then become features that need to be supported for a very long time, is going to be the most difficult approach as there'd need to be a very good reason to have them in the stdlib. Third party libraries that can provide what you need, or rolling your own libssl API wrappings however you choose to implement them, are better bets. -Greg

On Wed, 2022-11-30 at 14:14 -0800, Gregory P. Smith wrote:
Does not support TLS at all as far as I can see.
https://pypi.org/project/oscrypto/ exists and is quite interesting.
Does not support ALPN nor export keying material. It would probably be possible to add support though.
the old https://pypi.org/project/M2Crypto/ package still exists and seems to be maintained (wow).
Does not support ALPN nor export keying material. And considering your surprise at it still being maintained it doesn't feel like something that one should use as a base for new code.
What I have tried to add support for the last two years is not an API specific for OpenSSL, it is part of TLS. It was introduced in TLSv1.2 in 2010 by RFC5705 "Keying Material Exporters for Transport Layer Security (TLS)". OpenSSL added support for it in 2011 and the API has basically been unchanged since (the parameter names in the prototype in openssl/tls1.h has changed but is functionally identical). Support for RFC5705 is available in GnuTLS, Microsoft Schannel, Mbed-TLS, Botan and even good old Mozilla NSS. So basically what you are telling me is that there is no interest in adding support for a 12 year old part of TLS to the standard ssl library in Python, nor any interest in adding hooks to make it possible to extend the ssl library. Because the latter is basically what I was told to do when my patch to add support for RFC5705 was rejected: https://bugs.python.org/issue43902
I have not found any third party library that supports RFC5705, none of the ones you mentioned above has support for it and are also missing other functionality such as ALPN that is needed by RFC8915 "Network Time Security for the Network Time Protocol" which is what I actually want to implement. The standard ssl library in Python is so very close to what I need, it has support for everything else needed by RFC8951, so I would very much prefer to use it if possible. It just feels so silly somehow to have to have my own fork of all of CPython or to use a completely different TLS library just to be able to use a functino that has been a part of TLS and OpenSSL since basically forever. :) /Christer

On 11/30/2022 4:52 PM, christer@weinigel.se wrote:
Does this seem like a good idea? As I said, I feel that it is a bit ugly, but it does mean that if someone wants to use some SSL_really_obscure_function in libcrypto or libssl they can do that without having to rebuild all of CPython themselves.
Broadly, no, I don't think it's a good idea. We don't like encouraging users to do things that make it hard to support them in the future. Nonetheless, it's one that I've had to do, and so realistically I think it's okay to *enable* the hack without endorsing it. This is one of the reasons I switched the Windows builds to dynamically linked OpenSSL builds (they used to be statically linked, which meant there was no way to get at the unused exports). So now you can use `import _ssl; ctypes.CDLL("libssl-1_1")` to get at other exports from the module if you need them, and there's a similar trick to get the raw handle that I don't recall off the top of my head. But the only reason I'd ever want to document this is to tell people not to rely on it. If you control your environment well enough that you can guarantee it'll work for you, that's great. Nobody else should ever think they're doing the "right thing". Cheers, Steve

On Wed, Nov 30, 2022 at 12:47 PM Steve Dower <steve.dower@python.org> wrote:
+1 ... and in general if you want access to other OpenSSL APIs not already in the ssl module, getting them via non-stdlib packages on PyPI would be a better idea. https://pypi.org/project/cryptography/ is very well supported. https://pypi.org/project/oscrypto/ exists and is quite interesting. the old https://pypi.org/project/M2Crypto/ package still exists and seems to be maintained (wow). More context: We don't like the ssl module in the standard library - it is already too tightly tied to OpenSSL: https://discuss.python.org/t/our-future-with-openssl/21486 So if you want specific OpenSSL APIs that are not exposed, seeking to see them added to the standard library where they would then become features that need to be supported for a very long time, is going to be the most difficult approach as there'd need to be a very good reason to have them in the stdlib. Third party libraries that can provide what you need, or rolling your own libssl API wrappings however you choose to implement them, are better bets. -Greg

On Wed, 2022-11-30 at 14:14 -0800, Gregory P. Smith wrote:
Does not support TLS at all as far as I can see.
https://pypi.org/project/oscrypto/ exists and is quite interesting.
Does not support ALPN nor export keying material. It would probably be possible to add support though.
the old https://pypi.org/project/M2Crypto/ package still exists and seems to be maintained (wow).
Does not support ALPN nor export keying material. And considering your surprise at it still being maintained it doesn't feel like something that one should use as a base for new code.
What I have tried to add support for the last two years is not an API specific for OpenSSL, it is part of TLS. It was introduced in TLSv1.2 in 2010 by RFC5705 "Keying Material Exporters for Transport Layer Security (TLS)". OpenSSL added support for it in 2011 and the API has basically been unchanged since (the parameter names in the prototype in openssl/tls1.h has changed but is functionally identical). Support for RFC5705 is available in GnuTLS, Microsoft Schannel, Mbed-TLS, Botan and even good old Mozilla NSS. So basically what you are telling me is that there is no interest in adding support for a 12 year old part of TLS to the standard ssl library in Python, nor any interest in adding hooks to make it possible to extend the ssl library. Because the latter is basically what I was told to do when my patch to add support for RFC5705 was rejected: https://bugs.python.org/issue43902
I have not found any third party library that supports RFC5705, none of the ones you mentioned above has support for it and are also missing other functionality such as ALPN that is needed by RFC8915 "Network Time Security for the Network Time Protocol" which is what I actually want to implement. The standard ssl library in Python is so very close to what I need, it has support for everything else needed by RFC8951, so I would very much prefer to use it if possible. It just feels so silly somehow to have to have my own fork of all of CPython or to use a completely different TLS library just to be able to use a functino that has been a part of TLS and OpenSSL since basically forever. :) /Christer
participants (5)
-
Barry
-
Christer Weinigel
-
christer@weinigel.se
-
Gregory P. Smith
-
Steve Dower