Python threading (was: Re: global interpreter lock not working as it should)

[Jonathan Hogg]

On 5/8/2002 10:12, in article
ddc19db7.0208050112.56a26827 at posting.google.com, "Armin Steinhoff"
<a-steinhoff at web.de> wrote:

> martin at v.loewis.de (Martin v. Loewis) wrote in message
> news:<m3u1maavgh.fsf at mira.informatik.hu-berlin.de>...
[...]
>> CHECK_STATUS("pthread_mutex_lock[1]");
>> success = thelock->locked == 0;
>> if (success) thelock->locked = 1;
> 
> 'success' will always be 0 ... so why 'if(success)' ??????
> We are in a critical section !!!

Aha. You've slightly misunderstood the way the pthreads GIL works. The mutex
is used only to protect access to 'thelock->locked'. This variable
represents the actual "Global Interpreter Lock".

Grabbing 'thelock->mut' doesn't necessarily mean you have the GIL, it just
means you can safely test and set - which is what the above part does.

>> status = pthread_mutex_unlock( &thelock->mut );
>> CHECK_STATUS("pthread_mutex_unlock[1]");
>> 
>> if ( !success && waitflag ) {
> 
> The variable 'success' is meaningless ... who sets the waitflag ??

'waitflag' is an argument to the function, but it doesn't matter
particularly. 'success' will be true if the lock was free and we took it.

>> /* continue trying until we get the lock */
>                  ????????

This will become clear further down.

>> 
>> /* mut must be locked by me -- part of the condition
>> * protocol */
>> status = pthread_mutex_lock( &thelock->mut );
>> CHECK_STATUS("pthread_mutex_lock[2]");
> 
> There is nothing to check ... you own now simply the mutex here.
> 
> The while loop is plain nonsens and dangerous because 'thelock->mut'
> isn't acquired again!!

The mutex has been released before we enter this loop (the second quoted
section above). In order to wait on a condition we have to hold the mutex
again, so we grab it.

>> while ( thelock->locked ) {
>> status = pthread_cond_wait(&thelock->lock_released,
>>   &thelock->mut);
>> CHECK_STATUS("pthread_cond_wait");
> 
> After calling pthread_cond_wait() the mutex 'thelock->mut' will be
> released!

No. That's not the way pthread_cond_wait works. We must hold the mutex
before we call this. The implementation atomically releases the mutex and
puts us to sleep on the condition variable. When the thread is later woken
because of the condition being signalled, the implementation atomically
reacquires the mutex and returns.

A condition wait always occurs inside a critical section. The reason for
this is that we are usually waiting on a condition being signalled to notify
us that something is available or has changed (in this case, the GIL). That
something must be protected by a mutex otherwise we could end up with a race
condition where the thread is awoken to say something is ready, but the
something has been changed by someone else before we get to it.

> The mutex 'thelock->mut' is only used to build a critical section
> around the condition variable 'thelock->lock_released'
> 
> Who does pthread_cond_signal() and when ???

The pthread_cond_signal is done by the code that releases the GIL. This code
(which isn't quoted) grabs 'thelock->mutex', unsets 'thelock->locked',
releases the mutex and then signals 'thelock->lock_released' allowing
another thread to be unblocked (and possibly switched to depending on the
scheduler).

>> }
> 
> 'thelock->mut is now locked again ...
> 
>> thelock->locked = 1;
> 
> That's really corious ...

We own the mutex, and the 'while ( thelock->locked )' loop has exited so we
know the lock is now free, so we can safely grab it.

>> status = pthread_mutex_unlock( &thelock->mut );
>> CHECK_STATUS("pthread_mutex_unlock[2]");
> 
> 
>> success = 1;
>> }
>> 
>> Nobody is blocking on the mutex;
> 
> Sorry ... that's not true! Please read the semantic of
> pthread_mutex_lock.

OK, this is slightly misleading. It is possible for a thread to block on the
mutex, but highly unlikely. The mutex is used only to protect the few
instructions that access 'thelock->locked'. It is not used by threads to
wait for the GIL. Threads waiting for the GIL block on the condition
variable 'thelock->lock_released' using pthread_cond_wait, as above.

> This code looks like a MEDIUM chaos for me ...
> 
> It should  be possible to implement the GIL by a single condition
> variable!

No, one must have a mutex in order to use conditions and returning from the
condition does not guarantee that the GIL is free as the thread may have
been unblocked, but not scheduled yet. In the time before it is scheduled,
the original thread may have reacquired the GIL.

This is the problem I described before with SCHED_RR threads. The thread
switch will not occur until the timer interrupt, and that is very unlikely
to occur while the lock is released.

The pthreads-GIL actually makes a lot of sense, but you have to think hard
to understand it. I can probably explain it more clearly if you like.

Jonathan