OpenSSL thread safety when reading files?
I'm seeing a number of malloc (actully, free) errors, now that I'm pounding on the OpenSSL server/client setup with lots of server threads and client threads. They all look like either (gdb) bt #0 0x9010b807 in malloc_printf () #1 0x900058ad in szone_free () #2 0x90005588 in free () #3 0x9194e508 in CRYPTO_free () #4 0x91993e77 in ERR_clear_error () #5 0x919b1884 in PEM_X509_INFO_read_bio () #6 0x9197a692 in X509_load_cert_crl_file () #7 0x9197a80e in by_file_ctrl () #8 0x919d6e2e in X509_STORE_load_locations () [...] or (much more frequently) (gdb) bt #0 0x9010b807 in malloc_printf () #1 0x900058ad in szone_free () #2 0x90005588 in free () #3 0x9194e508 in CRYPTO_free () #4 0x91993e77 in ERR_clear_error () #5 0x949fcf11 in SSL_CTX_use_certificate_chain_file () [...] Always in ERR_clear_error(), always from some frame that's reading a certificate file for some purpose. If I disable Py_BEGIN_ALLOW_THREADS/Py_END_ALLOW_THREADS around the places where the C code reads the certificate files, all these free errors go away. ERR_clear_error() is supposed to be thread-safe; it operates on a per-thread error state structure (which I make sure is initialized in my C code). But it sure looks like the client and server threads are both working with the same error state. Bill
The issue seems to be that we assume OpenSSL is thread-safe (that is, we call Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS), but the _ssl.c code never did what was necessary to support that assumption. See http://www.openssl.org/docs/crypto/threads.html#DESCRIPTION. My analysis is that we need to add lock and unlock functions to the OpenSSL initialization code we currently use, which looks like this: /* Init OpenSSL */ SSL_load_error_strings(); SSLeay_add_ssl_algorithms(); Or, just not allow threads, which seems wrong. Bill
My analysis is that we need to add lock and unlock functions to the OpenSSL initialization code we currently use
Yep, this seems to fix the problem. I'm now able to re-enable Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS, and still get a clean run: (gdb) run Starting program: /local/python/trunk/src/python.exe ./Lib/test/regrtest.py -R :4: -u all test_ssl test_ssl /local/python/trunk/src/Lib/test/test_ssl.py:247: DeprecationWarning: socket.ssl() is deprecated. Use ssl.sslsocket() instead. ssl_sock = socket.ssl(s) beginning 9 repetitions 123456789 ......... 1 test OK. [30009 refs] Program exited normally. (gdb) Bill
The issue seems to be that we assume OpenSSL is thread-safe (that is, we call Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS), but the _ssl.c code never did what was necessary to support that assumption. See http://www.openssl.org/docs/crypto/threads.html#DESCRIPTION. My analysis is that we need to add lock and unlock functions to the OpenSSL initialization code we currently use, which looks like this: /* Init OpenSSL */ SSL_load_error_strings(); SSLeay_add_ssl_algorithms(); Or, just not allow threads, which seems wrong. Bill
My analysis is that we need to add lock and unlock functions to the OpenSSL initialization code we currently use
Yep, this seems to fix the problem. I'm now able to re-enable Py_BEGIN_ALLOW_THREADS / Py_END_ALLOW_THREADS, and still get a clean run: (gdb) run Starting program: /local/python/trunk/src/python.exe ./Lib/test/regrtest.py -R :4: -u all test_ssl test_ssl /local/python/trunk/src/Lib/test/test_ssl.py:247: DeprecationWarning: socket.ssl() is deprecated. Use ssl.sslsocket() instead. ssl_sock = socket.ssl(s) beginning 9 repetitions 123456789 ......... 1 test OK. [30009 refs] Program exited normally. (gdb) Bill
participants (1)
-
Bill Janssen