[Python-Dev] Extension modules, Threading, and the GIL

[David Abrahams]

martin@v.loewis.de (Martin v. L=F6wis) writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> > What matters is whether the callback is invoked in the context of
>> > the thread that installed it (i.e. as a result of calling a function
>> > in B).
>>=20
>> I still don't see how the thread that installed it has any bearing.
>
> If the callbacks would only occur while Q is being called from B,
> and in the context of the thread which also hosts that call, then B
> should not release the lock. Then the callback would not need to
> reacquire it.

AFAICT, that still has nothing to do with which thread installs the
callback.  It has everything to do with which threads may invoke the
callback, and whether they already hold the GIL.

>> There are several ways, IIUC.  It may be invoked in response to
>> direct calls into Q's API (which may be made from a Python
>> extension module).  It may also be invoked by some thread that Q
>> has launched.
>
> I find this surprising. Is this *any* API of Q that can trigger the
> callbacks, or a dispatch_one_event call that can do so?

I don't know; I'll have to defer to Phil for this answer.

>> Yes, we understand that problem.  I had this exact discussion with
>> the designer of B.  He explained that the problem is that Q might
>> also invoke the virtual function on a thread that is not holding
>> the GIL.
>
> I would more deeply question such a statement. It sounds like this
> library uses the callback for event processing of some kind, in
> which case the callbacks are only invoked if events are
> processed. It should be possible to describe more precisely under
> what specific conditions event processing occurs.

Probably.  I can't do it, though.

>> > Then I'm still curious as to what triggers invocation of that virtual
>> > function.
>>=20
>> Q, either directly via its API, or in some thread it started.
>
> Then calling this API must release the GIL.

Yes, that is clearly the case under present conditions.  The point is
that this is merely by virtue of the fact that some other extension
module may install a GIL-acquiring callback in Q.

--=20
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution