C-extension not threadable?

[Dave Cole]

>>>>> "Martin" == Martin v Loewis <martin at v.loewis.de> writes:

Martin> Dave Cole <djc at object-craft.com.au> writes: Yes, the Global
Martin> Interpreter Lock (GIL). When C code runs that was invoked from
Martin> Python code, it holds the GIL, and only one such C code can
Martin> run at any time. So you need to release the lock, using
Martin> Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS.

>>  If the API which is being wrapped by the extension module utilises
>> callbacks, and you want to make those callbacks available to Python
>> code, and you want to do this in a threading environment where the
>> GIL is released when calling the API, then the above will not work.

Martin> You must not call *any* Python API while not holding the GIL,
Martin> yes.

Martin> That doesn't mean that my approach won't work - it is just not
Martin> the full story.

>> This seems to be working fine so far (fingers crossed).

Martin> Sounds good; if possible, you should indeed pass the thread
Martin> state forward to the callback. If you have a lock around the
Martin> library, passing that thread state in a global variable works
Martin> fine (see _tkinter for an example).

That would be another way to do it.  The problem in the database case
is that some API's will take a long time to execute at the server so
it is not a good idea to block all access to the API across all
connections.  It would defeat one of the major benefits of threading.

Martin> If callbacks can come out of arbitrary threads, or if it is
Martin> not possible to pass the thread-state through, you can always
Martin> create a new thread state when entering the callback. In that
Martin> case, you still need to obtain the interpreter somehow.

If you create a new thread state for the callback, what will happen to
Python exceptions raised in the callback?

- Dave

-- 
http://www.object-craft.com.au