[Python-Dev] Extension modules, Threading, and the GIL

[Martin v. Löwis]

David Abrahams <dave@boost-consulting.com> writes:

> > So Q creates new threads which perform callbacks? But Q also performs
> > the callbacks when invoked from A? Sounds like a bug in Q to me...
> 
> Why do you say that?  Q doesn't know anything about Python or its
> constraints.  Why should it be prohibited from invoking these
> callbacks in whatever way it deems appropriate for its problem domain?

It is moderately evil for a library to create threads "under the
hoods", IMO; in some domains, that might be a reasonable thing to do,
provided there is a way for the application author to manage the
threads on a higher level (e.g. by limiting the total number of
threads that the library can create simultaneously).

If a library is creating new threads and invokes application code in
these threads, threading should follow a threading model. That
threading model has to be described, so that every application author
can rely on certain features. The threading model is part of the
library interface, just like the API.

It appears that Q has no threading model. That is truly evil.

In some cases, combining libraries with different threading models
just won't work. For example, I recently found that Tcl's appartment
threading model isn't really compatible with Python's GIL. It is
possible to achieve interworking to some respect, but there are
limitations that just can't be overcome (e.g. you just cannot invoke
event dispatching in a thread that hasn't originally create the Tcl
interpreter).

If the threading model of Q is unknown or undefined, you cannot expect
any kind of interworking.

> I think rather that this is a library design which doesn't
> interoperate well with Python's constraints on GIL manipulation.

It seems to me that there is no design in the library, and this is the
cause for the interoperability problem (or, perhaps, you just haven't
presented the design).

> In this case, as Python is intended to be good for general
> interoperability, it seems like Python ought to budge if possible.

As Tim explains, this might not be possible.

Regards,
Martin