working with Python threads from C extension module?
Reading through the C API documentation, I find: ``This is done so that dynamically loaded extensions compiled with thread support enabled can be loaded by an interpreter that was compiled with disabled thread support.'' I've currently got the set-up-SSL-threading code in _ssl.c surrounded by a "#ifdef HAVE_THREAD" bracket. It sounds like that might not be sufficient. It sounds like I need a runtime test for thread availability, instead, like this: #ifdef HAVE_THREAD if (PyEval_ThreadsInitialized()) _setup_ssl_threads(); #endif Seem right? So what happens when someone loads the _ssl module, initializes the threads, and tries to use SSL? It's going to start failing again. I think I need my own version of Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS, don't I? Which also checks to see if the SSL threading support has been initialized, in addition to the Python threading support. Something like #define SSL_ALLOW_THREADS {if (_ssl_locks != NULL) { Py_BEGIN_ALLOW_THREADS }} #define SSL_DISALLOW_THREADS {if (_ssl_locks != NULL) { Py_BEGIN_ALLOW_THREADS }} Any comments? Bill
#define SSL_ALLOW_THREADS {if (_ssl_locks != NULL) { Py_BEGIN_ALLOW_THREADS }} #define SSL_DISALLOW_THREADS {if (_ssl_locks != NULL) { Py_BEGIN_ALLOW_THREADS }}
I'd forgotten how convoluted Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS were. Anyone have any other suggestions about how to do this? Raise an error if loaded in a non-threaded environment, then used in a threaded environment? Dynamic initialization of threading? Bill
On Fri, 2007-09-07 at 16:20 -0700, Bill Janssen wrote:
#define SSL_ALLOW_THREADS {if (_ssl_locks != NULL) { Py_BEGIN_ALLOW_THREADS }} #define SSL_DISALLOW_THREADS {if (_ssl_locks != NULL) { Py_BEGIN_ALLOW_THREADS }}
I'd forgotten how convoluted Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS were. Anyone have any other suggestions about how to do this?
Be convoluted yourself and do this: #define PySSL_BEGIN_ALLOW_THREADS { if (_ssl_locks) { Py_BEGIN_ALLOW_THREADS #define PySSL_END_ALLOW_THREADS Py_END_ALLOW_THREADS } } (Untested, but I think it should work.)
Be convoluted yourself and do this:
#define PySSL_BEGIN_ALLOW_THREADS { if (_ssl_locks) { Py_BEGIN_ALLOW_THREADS #define PySSL_END_ALLOW_THREADS Py_END_ALLOW_THREADS } }
(Untested, but I think it should work.)
Yes, that had occurred to me. We want the code inside the braces still to run if the locks aren't held, so something more like #define PySSL_BEGIN_ALLOW_THREADS { \ PyThreadState *_save; \ if (_ssl_locks_count>0) {_save = PyEval_SaveThread();} #define PySSL_BLOCK_THREADS if (_ssl_locks_count>0){PyEval_RestoreThread(_save)}; #define PySSL_UNBLOCK_THREADS if (_ssl_locks_count>0){_save = PyEval_SaveThread()}; #define PySSL_END_ALLOW_THREADS if (_ssl_locks_count>0){PyEval_RestoreThread(_save);} \ } would do the trick. Unfortunately, this doesn't deal with the macro behaviour. The user has "turned on" threading; they expect reads and writes to yield the GIL so that other threads can make progress. But the fact that threading has been "turned on" after the SSL module has been initialized, means that threads don't work inside the SSL code. So the user's understanding of the system will be broken. No, I don't see any good way to fix this except to add a callback chain inside PyThread_init_thread, which is run down when threads are initialized. Any module which needs to set up threads registers itself on that chain, and gets called as part of PyThread_init_thread. But I'm far from the smartest person on this list :-), so perhaps someone else will see a good solution. This has got to be a problem with other extension modules linked to libraries which have their own threading abstractions. Bill
This has got to be a problem with other extension modules linked to libraries which have their own threading abstractions.
Sure enough, sqlite3 simply assumes threads (won't build without them), and turns them on if it's used (by calling PyThread_get_thread_ident(), which in turn calls PyThread_init_thread()). Bill
So what happens when someone loads the _ssl module, initializes the threads, and tries to use SSL? It's going to start failing again. I
Which turns out to be exactly what test_ssl.py does. I'm tempted to have the _ssl module call PyEval_InitThreads(). Would that be kosher? Bill
participants (2)
-
Bill Janssen
-
Hrvoje Nikšić