Extension modules, Threading, and the GIL

[David Abrahams]

"Donn Cave" <donn at u.washington.edu> writes:

> Quoth David Abrahams <dave at boost-consulting.com>:
> ...
> | This is the continuation of a thread originated on python-dev; I was
> | asked to re-raise it here on python-list until the issue is better
> | understood.
> |
> | The original posting was here:
> |
> |   http://aspn.activestate.com/ASPN/Mail/Message/python-dev/1482879
> |
> | The he essence of the problem is that there's no way to write code
> | that uses the Python 'C' API and which has no knowledge of whether it
> | is running on Python's main thread when it is entered.
> |
> | The two respondents were left with some questions; you can read those,
> | and my responses, in the thread at the bottom of the page referenced
> | above.
>
> I'll agree with Martin von Loewis that it doesn't sound like you
> really should care precisely whether it's the ``main'' thread or
> not, 

You're right; I don't.  It's a question of whether the thread posesses
the GIL or not.

> and it would be interesting to read more about the application
> context here with respect to the callbacks and threads, because it
> isn't that clear why this problem actually comes up.

Nor to me.  Q == Qt, and apparently it may invoke some of the
callbacks in response to calls to its API, and also in response to
asynchronous events (e.g. on some other thread than the one Python is
executing in).

> But while I'm not opposed to some brilliant fix to the thread
> bootstrapping problems in Python, my simplistic answer would be that
> external function calls should always release the interpreter lock,
> whether they need to in anticipation of callbacks or not.  

Interesting; that's what the author of B said as well.  However, most
of my users have absolutely no need to release the GIL, and I'd hate
to make them pay for it just because a few strange applications do.

> The only exception would be the kind of thing that is really
> transparently free of any kind of I/O, recursion 

What does recursion have to do with anything?

> or nontrivial computation.  Not only because of the present issue,
> but to make the most of whatever concurrency you have in Python,
> which is of course limited to these moments.  

I understand the sentiment.  However, lots of Python applications
aren't using threading at all, so it seems like a bit of a waste to
manipulate the GIL at all in those cases.

Hm. I think I take it back.  If the computation is really nontrivial,
we can afford to waste a little GIL manipulation ;-)

> If implementation A fails to do this and it causes problems for B,
> then it's time to fix A.

Clearly, that's the only available solution at present.

> Then the callback's problem is to find out which thread it belongs
> to - at least, in my C++ wrappers I do undertake to always acquire
> the same thread context for the same thread, and I think that's best
> if you want exceptions and so forth to work correctly.  It's easy in
> my application context, but I can see how it would be a problem in a
> more generic situation, and a thread:threadstate table might be a
> handy addition to the core.

How do you approach the problem of figuring out which thread context
to acquire?

-- 
                       David Abrahams
   dave at boost-consulting.com * http://www.boost-consulting.com
Boost support, enhancements, training, and commercial distribution