[Python-Dev] Extension modules, Threading, and the GIL

[David Abrahams]

"Martin v. L=F6wis" <martin@v.loewis.de> writes:

> Mark Hammond wrote:
>> It can be done, yes.  I am not looking for a change in semantics, just a
>> simple way to do it (and maybe even a fast way to do it, but that is
>> secondary).  If such a way already exists, please enlighten us.  If not,=
 but
>> it is sufficiently simple to describe, then please describe it.  Otherwi=
se,
>> I do not understand your point.
>
> There is a very simple strategy to support multiple threads in an
> extension module.
>
> 1. In all callbacks, create a thread state and acquire the current
>     thread (this requires a singleton interpreter state).
>
> 2. In all API calls that may invoke callbacks, use
>     Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS around the API call.
>
> If this strategy is followed, every code always has all Python
> resources, and no deadlocks result.

IIUC, that strategy doesn't get Mark what he wants in this case:

    2.2) Extensions that *may* take a little time, but more to the
    point, may directly and synchronously trigger callbacks.  That is,
    it is not expected that much time will be spent outside of Python,
    but rather that Python will be re-entered.

Which is to be able to avoid releasing the GIL in the case where the
extension isn't going to do much other than invoke the callback
function which re-acquires it.

--=20
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution