[Python-Dev] Extension modules, Threading, and the GIL

[David Abrahams]

martin@v.loewis.de (Martin v. L=F6wis) writes:

> David Abrahams <dave@boost-consulting.com> writes:
>
>> > Wrong. If the code in B that calls Q does not allow threads, the
>> > callbacks don't need to reacquire the GIL.
>>=20
>> I think you must be misunderstanding me.  These callbacks might be
>> invoked on threads that are not the Python main thread.=20=20
>
> Which thread is the main thread is not relevant at all.

Sorry, I got "main" confused with "current".  My point is that the
callback may be invoked by threads that don't hold the GIL.

> What matters is whether the callback is invoked in the context of
> the thread that installed it (i.e. as a result of calling a function
> in B).

I still don't see how the thread that installed it has any bearing.
Imagine there's a global function pointer variable somewhere.  Some
thread comes along and makes it point to some function f ("installs
the callback").  Now there are various ways that callback can be
called.  Some other thread may pick up the global variable at any time
and invoke the function it points to.  Why does it matter which thread
set the variable?

>> > Can you please explain what a callback is?=20
>>=20
>> I'm trying to leave out irrelevant details, but a callback happens to
>> be a virtual function in a C++ class instance in this case.  These
>> callbacks implement behaviors of base classes supplied by the library,
>> Qt.=20=20
>
> Ok, now I know how a callback is installed (by redefining a virtual
> method).=20

Technically, redefining a virtual function by itself doesn't do
anything.  You have to make an instance of the class which redefines
that function available to the library somehow.  But you knew that.

> The other question is: How is it invoked? I.e. who invokes
> it, and why?
>
> I suppose the immediate answer is "the library Q". However, that
> library does not invoke it with out a trigger: What is that trigger?

There are several ways, IIUC.  It may be invoked in response to direct
calls into Q's API (which may be made from a Python extension module).
It may also be invoked by some thread that Q has launched.

>> What matters, AFAICT, is that the callback might be invoked on a
>> thread that's not Python's main thread, thus must acquire the GIL,
>> so if Python's main thread wants to call something in Q that might
>> invoke one of these callbacks, it must release the GIL.
>
> Which thread is the main thread is completely irrelevant. If you have
> something like
>
> class MyB(B.Base):
>   def overridden(self):
>     print "This overrides a virtual function"
>
> def doit():
>   b =3D MyB()
>   B.do_the_real_work(b) # will call Q, which will call the callback,
>                         # which will call overridden
>
> then there is no need for the C++ code in B to release the GIL, nor
> is there a need for B to reacquire the GIL in the callback.  This is
> independent of whether doit is called in the main thread, or in the
> context of any other thread: at the point where do_the_real_work is
> called, the current thread holds the GIL.

Yes, we understand that problem.  I had this exact discussion with the
designer of B.  He explained that the problem is that Q might also
invoke the virtual function on a thread that is not holding the GIL.

> If it keeps the GIL, then the GIL will be held at the point when the
> callback occurs, in the context of the thread where the callback
> occurs.

Yes.

>> Maybe we have different ideas of what "callback" means.  In your
>> terms, it is not a "true callback".
>
> Then I'm still curious as to what triggers invocation of that virtual
> function.

Q, either directly via its API, or in some thread it started.

--=20
                       David Abrahams
   dave@boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution