[Python-Dev] GIL removal question

[Guido van Rossum]

On Sat, Aug 13, 2011 at 8:43 AM, Antoine Pitrou <solipsis at pitrou.net> wrote:
> On Sat, 13 Aug 2011 09:08:16 -0400
> Guido van Rossum <guido at python.org> wrote:
>>
>> And, though mostly off-topic, the worst problem with C code, calling
>> back into Python, and the GIL that I have seen (several times):
>> Suppose you are calling some complex C library that creates threads
>> itself, where those threads may also call back into Python. Here you
>> have to put a block around each Python callback that acquires the GIL
>> before and releases it after, since the new threads (created by C
>> code) start without the GIL acquired. I remember a truly nasty
>> incident where the latter was done, but the main thread did not
>> release the GIL since it was returning directly to Python (which would
>> of course release the GIL every so many opcodes so the callbacks would
>> run). But under certain conditions the block with the
>> acquire-release-GIL code around a Python callback was invoked in the
>> main thread (when a validation problem was detected early), and since
>> the main thread didn't release the GIL around the call into the C
>> code, it hung in a nasty spot. Add many layers of software, and a
>> hard-to-reproduce error condition that triggers this, and you have a
>> problem that's very hard to debug...
>
> These days we have PyGILState_Ensure():
> http://docs.python.org/dev/c-api/init.html#PyGILState_Ensure
>
> and even dedicated documentation:
> http://docs.python.org/dev/c-api/init.html#non-python-created-threads
>
> ;)

That is awesome!

-- 
--Guido van Rossum (python.org/~guido)