Python threading (was: Re: global interpreter lock not working as it should)

[Martin v. Loewis]

bokr at oz.net (Bengt Richter) writes:

> Hm. I suppose the pthread_lock is supposed to be general purpose,
> not just for the GIL, 

Yes, that is for threading.lock.acquire.

> but ISTM (;-) you could easily add serial handover functionality if
> you wanted to use that for the GIL.

Not easily. If you want to add another threading primitive, you need
to add it for all 11 threading libraries.

Since nobody has simultaneous access to all 11 platforms, such a
change is nearly impossible to implement.

> It might be useful for other things too.

Things that are useful are implemented in threading.py.

> Signaling the condition variable seems to use kill(th->p_pid,
> PTHREAD_SIG_RESTART) to start a waiting thread. Does that affect
> scheduling order between it and the releasing thread?

Usage of this mechanism to signal the condition variable, and the
system reaction to a signal, are both highly system dependent. You'd
have to pick a specific OS, OS release, and threading library release
to answer this question.

> My impression was that it requires the mutex to be locked when called,
> and unlocks it itself so as to wait unlocked, and then re-locks it for
> the waiter. I.e., inside pthread_cond_wait:
>   ...
>   pthread_mutex_unlock(mutex);
>   suspend_with_cancellation(self);
>   pthread_mutex_lock(mutex);
>   ...
> 
> I guess the re-lock involves trying, though -- is that a tiny crack
> for Murphy to sneak through?
> 
> Ah, I guess that's why there's that 'while' in PyThread_acquire_lock:

POSIX specifies that pthread_cond_wait can return spontaneously, which
alone is the reason for the while loop. However, I do think that the
woken-up thread and the thread that just signalled the condition do
compete to lock the mutex (when ceval tries to lock the GIL again).

This is the race condition that I was referring to earlier, and it is
another reason for the while loop.

> Thanks for your pointers. BTW, the source for the linuxthreads (or
> whatever pthread package is actually used) is (AFAICS) not included
> in the win32 python distribution. Might this be a good idea for
> cross-platform documentation purposes?

You want the source of linuxthreads in the Python distribution???
Or do you mean the source of thread_pthread.h of Python? The latter
is available only with the source code of Python (as all other 10
thread_*.h files).

I do believe that linuxthreads is quite specific to linux, and that
you find that condition variables are implemented quite different on
other systems.

Furthermore, Python 2.3 will use POSIX semaphores for threading.lock
where available - which will add another dimension to this issue.

This all comes back to the start of the thread:

Other threads can run when the GIL is released; that does not mean
they will run.

Regards,
Martin