[Python-Dev] GIL removal question

[Guido van Rossum]

On Sat, Aug 13, 2011 at 2:12 AM, Stefan Behnel <stefan_ml at behnel.de> wrote:
> Guido van Rossum, 12.08.2011 23:38:
>>
>> On Fri, Aug 12, 2011 at 12:57 PM, Rene Nejsum wrote:
>>>
>>> I think I understand the background and need for GIL. Without it Python
>>> programs would have been cluttered with lock/synchronized statements and
>>> C-extensions would be harder to write.
>>
>> No, sorry, the first half of this is incorrect: with or without the
>> GIL *Python* code would need the same amount of fine-grained locking.
>> (The part about C extensions is correct.) I am butting in because this
>> is a common misunderstanding that really needs to be squashed whenever
>> it is aired -- the GIL does *not* help Python code to synchronize. A
>> thread-switch can occur between any two bytecode opcodes. Without the
>> GIL, atomic operations (e.g. dict lookups that doesn't require
>> evaluation of __eq__ or __hash__ implemented in Python) are still
>> supposed to be atomic.
>
> And in this context, it's worth mentioning that even C code can be bitten by
> the GIL being temporarily released when calling back into the interpreter.
> Only plain C code sequences safely keep the GIL, including many (but not
> all) calls to the C-API.

And, though mostly off-topic, the worst problem with C code, calling
back into Python, and the GIL that I have seen (several times):
Suppose you are calling some complex C library that creates threads
itself, where those threads may also call back into Python. Here you
have to put a block around each Python callback that acquires the GIL
before and releases it after, since the new threads (created by C
code) start without the GIL acquired. I remember a truly nasty
incident where the latter was done, but the main thread did not
release the GIL since it was returning directly to Python (which would
of course release the GIL every so many opcodes so the callbacks would
run). But under certain conditions the block with the
acquire-release-GIL code around a Python callback was invoked in the
main thread (when a validation problem was detected early), and since
the main thread didn't release the GIL around the call into the C
code, it hung in a nasty spot. Add many layers of software, and a
hard-to-reproduce error condition that triggers this, and you have a
problem that's very hard to debug...

-- 
--Guido van Rossum (python.org/~guido)